Skip to content
VisibilityIQ365 VisibilityIQ365 VisibilityIQ Help

Crawled Pages

Crawled Pages is the ground truth of your audit: a table of every URL the crawler actually fetched in the latest run, with what each page told the crawler about itself. Before you drill into Issues, this is where you confirm a page was discovered, spot an unexpected status code, and eyeball canonical and robots directives at a glance. Everything else in Crawl Data is a different lens on this same inventory.

What you’ll need

Section titled “What you’ll need”
  • A project with at least one completed audit. Crawled Pages reads from the most recent run, so it is empty until your first audit finishes.

What you’ll see

Section titled “What you’ll see”

Each row is one URL the crawler fetched, with four columns that tell you how the page responded and what it declares.

ColumnWhat it tells you
URLThe exact address the crawler fetched, after normalization.
StatusThe HTTP status code the server returned (200, 301, 404, 500, and so on).
CanonicalThe canonical URL the page declares for itself. A self-canonical points back to its own URL.
RobotsThe indexing directive the page sends (index, noindex, and related values from the meta robots tag or X-Robots-Tag header).

How to read it

Section titled “How to read it”

A page that returned 200 with a self-canonical and an index directive is behaving as expected. Anything else is worth a look. A 404 or 500 on a page you meant to keep means a broken or misconfigured URL the crawler still reached. A 3xx means the page redirects, and you can follow that thread in Redirects. A canonical pointing somewhere other than the page’s own URL is intentional sometimes and a mistake often. A noindex on a page you want indexed is the kind of quiet error this table is built to surface.

If a page you expected is missing from the table entirely, the crawler never reached it, usually because nothing links to it or robots.txt blocks it. That is a discovery problem, not a status problem.

How to use it

Section titled “How to use it”

Scan the Status column first for codes outside the 200 range, then scan Canonical and Robots for values that contradict your intent. This view is for confirmation and triage at a glance; when a row looks wrong, the matching finding in Issues carries the full evidence, remediation, and a generated fix where one applies.

Next steps

Section titled “Next steps”