robots.txt Parsing

VisibilityIQ parses robots.txt according to RFC 9309, the formal Robots Exclusion Protocol specification. The parser handles all user-agents simultaneously in a single pass.

What’s parsed

For each robots.txt file, the parser records:

interface RobotsTxtAnalysis {
  url: string;                  // e.g. "https://example.com/robots.txt"
  statusCode: number;
  parseErrors: string[];        // any parse errors found
  groups: RobotsGroup[];        // one entry per User-agent: group
  sitemapUrls: string[];        // all Sitemap: directives
  crawlDelays: Record<string, number>; // per-agent crawl delays
}

interface RobotsGroup {
  userAgents: string[];         // e.g. ["GPTBot", "CCBot"]
  allow: string[];
  disallow: string[];
}

Per-agent access check

To check whether a specific URL is accessible to a specific user-agent:

1. Find all groups that include the agent name (case-insensitive) or *
2. Collect all Allow and Disallow rules from matching groups
3. Find the most specific (longest path match) rule for the URL
4. If Allow and Disallow tie on length, Allow wins (per RFC 9309)
5. If no rule matches, the URL is allowed by default

This is the same algorithm Google uses. VisibilityIQ implements it faithfully so findings reflect what Google would actually do.

Common parsing pitfalls

Group separation

A blank line ends a group. If your robots.txt has:

User-agent: Googlebot
Disallow: /private

User-agent: GPTBot
Disallow: /

These are two separate groups. GPTBot gets Disallow: /. Googlebot gets Disallow: /private.

If the blank line is missing:

User-agent: Googlebot
User-agent: GPTBot
Disallow: /

Both agents are in the same group and both get Disallow: /. VisibilityIQ detects and flags this when it appears unintentional.

Wildcard user-agent

User-agent: * applies to all agents not explicitly named. If you have both User-agent: * and User-agent: Googlebot groups, Googlebot follows the Googlebot group only, not the * group. AI bots not explicitly named fall into the * group.

Allow vs Disallow precedence

If the same URL matches both an Allow and Disallow rule of equal length:

User-agent: GPTBot
Allow: /blog
Disallow: /blog

Allow wins. This is RFC 9309 behavior. VisibilityIQ documents the effective access as “Allowed” and notes the conflict.

Crawl-delay handling

Crawl-delay: N is a non-standard directive (not in RFC 9309) but widely used and respected by most crawlers. VisibilityIQ records crawl-delay values per agent:

User-agent: GPTBot
Crawl-delay: 10

This doesn’t affect the audit crawl speed, but a very long crawl-delay (> 60 seconds) for AI bots is flagged as a finding because it significantly slows AI crawler indexing of your content.

robots.txt not found

If robots.txt returns 404, that’s treated as no restrictions — all agents are allowed to crawl everything. VisibilityIQ notes this in the crawl metadata but doesn’t generate a finding (an absent robots.txt is valid).

If robots.txt returns 5xx, crawlers should treat it as disallowing access. VisibilityIQ generates a Critical finding when your robots.txt endpoint returns a server error, because this effectively blocks all bots from crawling your site.