How the crawler works

VisibilityIQ crawls your site in two passes every audit run. The first pass sees your page the way a fast, JavaScript-blind bot does; the second sees it the way a browser does. Comparing the two is where the hard-to-find SEO bugs surface.

flowchart TB
  U([A page on your site]) --> P1[Pass 1 · Raw HTML<br/>HTTP GET, no JavaScript]
  U --> P2[Pass 2 · Rendered DOM<br/>headless browser, JS runs]
  P1 --> F1[Title · canonical · robots<br/>links · headers · schema]
  P2 --> F2[Same fields,<br/>after hydration]
  F1 --> D{Field-by-field diff}
  F2 --> D
  D -->|fields match| OK([Parity OK])
  D -->|fields differ| FLAG([Render-parity finding])

Pass 1 — Raw HTML

The first pass fetches each page as a standard HTTP GET without JavaScript execution. This simulates what a search bot without a rendering engine sees: the HTML delivered by your server or CDN before any client-side code runs.

What’s captured: status code, redirect chain, response headers (including X-Robots-Tag, Link: rel=canonical), raw HTML, title, meta description, canonical tag, meta robots, structured data blocks, internal and external links, hreflang attributes, security headers, and HTTP protocol version.

Pass 2 — Rendered DOM

The second pass loads the same URL in a headless browser (Chromium via Cloudflare Browser Rendering) and captures the DOM after JavaScript has executed and the page has settled. This simulates what a rendering-capable crawler like Googlebot sees.

What’s captured: all of the above, plus any values that differ from the raw pass — canonicals injected by JS, meta robots tags added post-hydration, content only present after client-side rendering, structured data written by JavaScript.

Render-parity diff

The two snapshots are diffed field by field. Any value that exists in one pass but not the other, or that differs between the two, is flagged as a render-parity issue. These are often the hardest SEO bugs to find because raw-HTML-only crawlers simply don’t see them.

See Render-parity diff explained →

Discovery

The crawler starts from your sitemap (if present) and your domain’s root URL, then follows internal links found on every page it visits. Pages found only in the sitemap but not linked internally are flagged as sitemap-only pages. Pages linked internally but not in the sitemap are noted.

By default the crawler stays on your domain. Cross-domain crawling is off by default.

Robots.txt

The crawler respects your robots.txt for all user-agents it simulates: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. A page blocked by robots.txt is noted but not crawled — it still appears in findings as “blocked by robots.txt.”

Crawl budget

Standard crawls cover up to 2,500 pages per project. If your site has more, the crawler prioritises pages found in the sitemap and linked from the homepage. Contact support if you need a higher limit.