Crawlability & Discovery

Crawlability checks answer whether search engines can reach your pages in the first place. A page can have perfect on-page SEO and still be invisible to search engines if it’s not reachable by the crawler.

robots.txt analysis

VisibilityIQ parses your robots.txt for every user-agent it simulates: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, and Bytespider.

For each page in the crawl, it checks whether any of these agents are blocked from accessing it. Findings include:

• Critical pages blocked — a page with significant internal links or traffic data is blocked to Googlebot
• AI bots blocked — GPTBot or ClaudeBot are blocked while Googlebot is allowed (separate finding, see AI crawler access)
• Crawl-delay directives — pages with crawl-delay set for specific user-agents, which can slow AI bot indexation
• Overly broad allow/disallow rules — a disallow rule that accidentally blocks important URL patterns

When VisibilityIQ detects a problematic robots.txt rule, it generates the corrected robots.txt file for you — the exact text you can deploy to fix the issue.

Orphan pages

An orphan page is a page that no other page on your site links to. It can only be found via sitemap or direct URL entry — crawlers may never discover it through normal link-following. Orphan pages with valuable content are a common and easy-to-fix SEO problem.

flowchart LR
  H([Homepage]) --> A[Page A]
  H --> B[Page B]
  A --> C[Page C]
  B --> C
  SM[(Sitemap)] -.->|listed, never linked| O[Orphan page]

VisibilityIQ identifies orphans by building the full internal link graph from the crawl and finding pages with zero inbound internal links. It cross-references the sitemap to distinguish between:

• Sitemap-only orphans — in the sitemap but not linked from any page (medium priority)
• Invisible orphans — not in the sitemap and not linked (high priority)

Internal links

Internal link findings cover:

• Broken links (4xx/5xx) — links pointing to pages that return errors. These waste crawl budget and create poor user experience.
• Redirecting links — links pointing to URLs that redirect. The redirect works, but fixing the link saves a hop and passes link equity more cleanly.
• Weak anchor text — links where the anchor text is generic (“click here”, “read more”) rather than descriptive.
• Dead-end pages — pages with no outbound internal links. These break the crawl graph and trap link equity.
• Pages with only one inbound link — potentially underlinked important pages.

Crawl depth

Important pages that are buried too deep in the site hierarchy (many clicks from the homepage) receive less crawl attention and pass less link equity. VisibilityIQ flags pages beyond a configurable depth threshold, especially when those pages have significant backlinks or traffic.

Crawl traps

Crawl traps are URL patterns that can generate infinite or near-infinite URLs — for example, calendar pages that generate URLs for every date, or filters that create arbitrary parameter combinations. VisibilityIQ detects:

• Parameter expansion — URLs differing only by query parameter values that appear to generate duplicate content
• Infinite pagination — paginated sequences with no clear endpoint
• Session ID URLs — URLs containing session identifiers that generate unique URLs for the same content

Crawl trap findings include the URL pattern detected and a recommendation for how to handle it (e.g., add noindex, use robots.txt disallow, or implement canonical tags).

Sitemap consistency

VisibilityIQ checks your sitemap against crawl results:

• URLs in the sitemap that return 4xx or 5xx
• URLs in the sitemap that redirect (sitemaps should contain the final URL)
• Pages found by crawl that are missing from the sitemap
• lastmod values in the sitemap that don’t match the actual content modification date

WAF and CDN blocking

Sometimes Cloudflare, Akamai, or another WAF returns a challenge page or 403 to crawlers. VisibilityIQ detects challenge pages by looking for Cloudflare’s “Just a moment” interstitial pattern and other common WAF response signatures. If your WAF is blocking the audit crawl, you’ll see a finding with instructions for temporarily allowlisting our crawler IP range.