Crawler cross-check
The crawler cross-check runs an independent second crawl of your site and compares its findings against your main audit. The value is agreement: when two separate crawl engines both flag a cluster as duplicated or a page as non-indexable, the finding is solid and you can act on it with confidence. A single crawl pass can also simply miss things — a redirect chain it didn’t follow, a duplicate it didn’t cluster — and a second engine with a different traversal catches those. Cross-check surfaces duplicate-content clusters, duplicate title and description tags, non-indexable pages, redirect chains, and the internal-link edge graph. The crawl runs on VisibilityIQ.
What you’ll need
Section titled “What you’ll need”- A project with a completed audit to cross-check against.
- Available monthly data budget — a cross-check crawl consumes from the project’s included budget.
How to use
Section titled “How to use”Open the project, go to Site Audit → Crawler Cross-Check, and click Run cross-check. VisibilityIQ launches the independent crawl; status moves from running to done when it completes. When it finishes, the view reports how many rows each category produced — duplicate content, duplicate tags, non-indexable pages, redirect chains, and link edges — so you can see at a glance where the second engine found something.
How to read the results
Section titled “How to read the results”The headline is the row counts per category. They tell you where to look:
| Category | What a row means |
|---|---|
| Duplicate content | Two pages whose body content is substantially the same |
| Duplicate tags | Two pages sharing the same title or meta description |
| Non-indexable | A page the crawler determined search engines cannot index |
| Redirect chains | A URL that redirects through one or more hops before resolving |
| Link edges | A directed internal link from one page to another, forming the link graph |
Open a duplicate cluster to see the detail behind it. Each cluster shows the anchor URL (the page treated as canonical for the cluster), the duplicate URL paired with it, a similarity score quantifying how close the two are, the duplicating tag value when the match is a shared title or description, and the HTTP status code of both pages so you can tell a live duplicate from one that already redirects or 404s.
Use the result as a confidence check, not a replacement for your main audit. Where the cross-check and your audit agree, the issue is real — prioritize it. Where only the cross-check flags something, you have caught a gap your first pass missed; verify it before acting. Where only your main audit flags something, the second engine may simply traverse the site differently — neither is automatically wrong.
Refreshing the data
Section titled “Refreshing the data”Click Run cross-check again to launch a fresh independent crawl; status returns to running and then done. A cross-check consumes from the project’s monthly budget, so run it after a significant audit or after you have applied fixes you want a second engine to confirm, rather than on every audit.