AI crawler access checks

AI search engines — ChatGPT, Claude, Perplexity, and Google’s AI Overview — crawl the web to build their knowledge and generate citations. If these crawlers can’t access your content, you won’t appear in AI-generated answers.

VisibilityIQ audits AI crawler access as a first-class concern alongside traditional search bot access. A page can be perfectly visible to Google and still invisible to an AI crawler — there are three places that access quietly breaks down:

flowchart TB
  S([AI bot requests a page]) --> RT{Blocked in<br/>robots.txt?}
  RT -->|yes| BLK[Not crawled —<br/>invisible to AI]
  RT -->|no| WAF{CDN / WAF returns<br/>403 or 429?}
  WAF -->|yes| BLK
  WAF -->|no| JS{Content needs<br/>JavaScript to appear?}
  JS -->|yes| UNREAD[Crawled but unreadable —<br/>AI sees an empty page]
  JS -->|no| OK([Readable — eligible<br/>for AI citations])

Bots checked

Bot	Product
GPTBot	ChatGPT / OpenAI
OAI-SearchBot	ChatGPT search
ClaudeBot	Claude (Anthropic)
PerplexityBot	Perplexity AI
Google-Extended	Google’s AI training
Bytespider	ByteDance / TikTok

What’s checked

robots.txt access — VisibilityIQ parses your robots.txt for each bot listed above and identifies which pages or directories are blocked per bot. It flags cases where a bot is blocked for AI crawlers but allowed for Googlebot (selective blocking).

AI bot response codes — Some sites return 403 or 429 to AI user-agents at the CDN or WAF level, without a robots.txt directive. VisibilityIQ detects this by comparing response codes across user-agents.

JavaScript-only content — AI crawlers generally don’t render JavaScript. Pages where the primary content, product descriptions, or key facts only exist in the rendered DOM are flagged as “AI-unreadable content.”

Crawl-delay directives — A Crawl-delay set to 60+ seconds for AI bots effectively blocks them in practice.

llms.txt

llms.txt is an emerging standard (similar to robots.txt) that helps AI assistants understand your site structure. VisibilityIQ checks:

• Whether llms.txt exists at your site root
• Whether it follows the proposed spec (sections for key pages, products, documentation)
• Whether the URLs listed in llms.txt are accessible and return 200

If you don’t have an llms.txt, VisibilityIQ will generate one based on your site structure. See the finding guidance for a generated file you can deploy directly.

How to allow AI crawlers

If you want AI crawlers to access your site (most publishers do — it’s how you get cited in AI answers):

1. Review your robots.txt. Remove any Disallow rules targeting GPTBot, ClaudeBot, or PerplexityBot that you didn’t intentionally add.
2. Check your CDN/WAF rules. Some security presets block “unknown bots” and catch AI crawlers.
3. Add an llms.txt at your site root.
4. Ensure critical content is in the raw HTML — not client-side-only.

If you want to block AI training crawlers (separate from AI search), use Google-Extended and specific Disallow rules. VisibilityIQ will note this as intentional, not a finding.