Crawl not completing

Symptoms

• A crawl has been in Running for more than 30 minutes.
• A crawl finishes almost instantly with 0 (or very few) pages.
• You trigger a crawl and nothing happens for several minutes.
• The competitor crawl panel returns “Crawl returned an unexpected response”.

Most crawls complete in 5–15 minutes for sites under 500 pages. Crawls checkpoint and resume — an interrupted crawl picks up where it left off rather than restarting from scratch — and failed URLs retry with exponential backoff up to a maximum of three attempts before being recorded as failed. That resilience is also why a crawl against a flaky server can run longer than you expect: each retry adds delay before the URL is finally abandoned.

Likely causes

• robots.txt is blocking the crawler. VisibilityIQ respects robots.txt for every user-agent — Googlebot, Bingbot, and AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) alike. A broad Disallow rule finishes the crawl quickly with few pages while it finalises.
• A crawl trap. Parameterised URLs (?page=1&color=red&size=large) generate effectively infinite URL variations. A rapidly growing page count in the crawl log is the tell.
• Server instability under load. 5xx responses or timeouts trigger the retry/backoff logic, slowing the whole crawl.
• Queue backlog. Crawls run through a job queue. During peak periods a new crawl may wait before a worker picks it up.
• A genuinely large site. Tens of thousands of pages, or browser rendering enabled across many templates, simply takes longer.

How to fix it

1. Check the crawl log first. Go to Audit → Crawl log to see how many pages have been found versus crawled. This single view distinguishes “blocked” (few pages found) from “trapped” (page count climbing without end) from “slow but healthy” (steady progress).
2. Resolve a block. Open yourdomain.com/robots.txt and look for a Disallow: / or a rule matching the VisibilityIQ user-agent. Remove or narrow it, or temporarily add User-agent: VisibilityIQ / Allow: /. If a WAF or CDN is challenging the crawler, allowlist our IP range — contact support for the current range.
3. Defuse a crawl trap. Add Disallow rules for the offending parameter patterns in robots.txt, or configure parameter handling under Settings → Crawl. Re-run the crawl; it will resume rather than start over.
4. Stabilise the origin. Check your server error logs for 5xx spikes during the crawl window. Lowering crawl concurrency in Settings → Crawl reduces load on a fragile origin.
5. Wait out a backlog. If a crawl hasn’t started within 5 minutes, refresh the project page — status updates every 30 seconds. Give it up to 15 minutes before escalating.

”Crawl returned an unexpected response”

In the competitor crawl panel, this means the target returned a non-standard response — a WAF challenge, a bot-detection redirect, or a CAPTCHA interstitial. VisibilityIQ respects these and won’t attempt to bypass them. Confirm the domain loads in a browser, wait 24 hours and retry (some WAF block windows are short), or mark the competitor as manually entered rather than crawled.

Still stuck?

If a crawl shows steady progress but never completes after 60+ minutes, or hasn’t started after 15 minutes, email support@visibilityiq365.com with your project domain. We’ll inspect the crawl log and queue state. See also Audit shows no issues or empty results when a crawl completes but findings are missing, and Missing rank data for rankings-specific gaps.