Multi-Bot User-Agent Simulation

VisibilityIQ audits your site from the perspective of multiple user-agents simultaneously, because different bots have different access levels, different rendering capabilities, and different directives to follow.

Simulated user-agents

User-agent	Represents	Key characteristic
`Googlebot`	Google web crawler	Renders JavaScript; used for rankings
`Bingbot`	Microsoft Bing crawler	Some JS rendering; used for Bing rankings
`GPTBot`	OpenAI training crawler	No JS rendering
`ClaudeBot`	Anthropic training crawler	No JS rendering
`PerplexityBot`	Perplexity AI crawler	No JS rendering
`Google-Extended`	Google AI/Gemini training	No JS rendering; separate from Googlebot
`OAI-SearchBot`	ChatGPT search results	Limited JS rendering
`Bytespider`	ByteDance / TikTok crawler	No JS rendering

How bot access is checked

For each URL in the crawl, the engine checks your robots.txt for every simulated user-agent:

Is this URL disallowed for Googlebot?     → yes/no + which directive
Is this URL disallowed for GPTBot?        → yes/no + which directive
Is this URL disallowed for ClaudeBot?     → yes/no + which directive
... (for all configured user-agents)

A URL can be allowed for Googlebot but blocked for GPTBot — these generate separate findings with separate severity ratings.

Content visibility by agent

For non-blocked URLs, the engine checks what each agent type can actually read:

Full-render bots (Googlebot, Bingbot) — compare raw HTML with rendered DOM; flag content visible only after JS execution
No-render bots (GPTBot, ClaudeBot, etc.) — check whether primary content and metadata are present in raw HTML; flag content that requires JavaScript

This is the AI-surface readiness audit: if your important content only appears after JavaScript runs, AI bots can’t read it.

robots.txt parsing per user-agent

The engine parses your robots.txt according to the Robots Exclusion Protocol, handling:

Named User-agent: groups (e.g., User-agent: GPTBot)
Wildcard User-agent: * (applies to all agents not explicitly named)
Allow: and Disallow: directives
Crawl-delay: (recorded per agent but not enforced — the audit records the value, not the behavior)

If your robots.txt has a parsing error (malformed directive, invalid syntax), the engine generates a finding noting the issue and documents what it could and couldn’t parse.

Configuring which bots to simulate

By default, all 8 user-agents above are simulated. You can reduce this under Project → Settings → Crawl → Bot simulation — deselecting bots you don’t care about reduces crawl time slightly. Googlebot cannot be deselected (it’s required for the baseline audit).

Finding examples

GPTBot blocked, Googlebot allowed:

Severity: High
Check: ai-crawler-access.robots-block
Evidence: {
  "userAgent": "GPTBot",
  "directive": "Disallow: /",
  "googlebot": "Allowed",
  "affectedPageCount": 1842
}

Content invisible to AI bots (JS-only):

Severity: High
Check: ai-crawler-access.js-only-content
Evidence: {
  "rawContent": null,
  "renderedContent": "Our product pricing starts at $29/month...",
  "elementType": "pricing-block",
  "pageUrl": "https://example.com/pricing"
}