Crawler Rate Limits & Ethics

VisibilityIQ crawls responsibly. It’s designed to audit your own site — not to scrape third parties aggressively. These rules apply to all crawl types.

Concurrency

The crawler enforces per-domain concurrency via Durable Objects. Default: 3 concurrent requests per domain.

This means at most 3 pages are being fetched simultaneously from a single origin. This matches Googlebot’s typical behavior for most sites and avoids overloading servers with limited connection capacity.

You can lower this for sensitive origins under Project → Settings → Crawl → Concurrency.

Crawl-delay respect

The crawler reads Crawl-delay: directives from robots.txt and honors them per user-agent. If your robots.txt says:

User-agent: *
Crawl-delay: 2

The crawler waits at least 2 seconds between requests to the domain.

For competitor crawls, the minimum delay is always 1 second, regardless of robots.txt — this is stricter than the spec requires but appropriate for crawling third-party sites.

robots.txt compliance

The crawler uses User-agent: VisibilityIQ for site audits. If your robots.txt disallows VisibilityIQ, the crawler respects that and doesn’t fetch blocked URLs. The finding will note that certain pages were blocked from the audit crawl.

For competitor crawls, the crawler uses User-agent: VisibilityIQ/1.0 (+https://visibilityiq.io/bot) and fully respects competitor robots.txt.

Request headers

Every crawl request sends:

User-Agent: VisibilityIQ/1.0 (+https://visibilityiq365.com/bot)
Accept: text/html,application/xhtml+xml
Accept-Encoding: gzip, deflate, br

The +https://visibilityiq365.com/bot operator URL allows server operators to identify and contact us if needed.

Timeout

Individual page fetches time out after 25 seconds. Pages that don’t respond within 25 seconds are marked with status timeout and generate a finding.

Retry policy

Failed pages (network error, 5xx) are retried with exponential backoff:

Attempt	Delay
1	Immediate
2	250ms
3	750ms
4 (final)	2,000ms

After 4 attempts, the URL is marked error and a finding is generated.

HTTP 4xx responses are not retried — they’re recorded as-is. A 429 response from the target server is treated as a rate_limited status and recorded; VisibilityIQ notes that the server actively throttled the crawler.

Crawl depth limit

Default: unlimited. Set a maximum crawl depth under Project → Settings → Crawl → Max depth to limit crawls on very deep sites. Pages beyond the depth limit are not crawled but are counted in the “discovered but not crawled” metric.

Domain scope

By default, the crawler only follows links to the same hostname. Cross-domain crawling (following links to subdomains or other domains) requires explicit opt-in under Project → Settings → Crawl → Cross-domain crawling.

Competitor crawls always stay on-domain and are subject to a hard 20-page maximum.