Crawler Architecture

The VisibilityIQ crawl engine is a Cloudflare Worker (workers/crawler/) that processes URLs from a Cloudflare Queue (CRAWL_QUEUE). It runs entirely at the edge, with no central server.

Architecture overview

Dashboard / API
     │ POST /api/audits (starts crawl)
     ▼
 Scheduler / Producer
     │ enqueues seed URLs → CRAWL_QUEUE
     ▼
 Crawler Worker (consumer)
     │ for each message:
     │   1. Fetch raw HTML
     │   2. Parse, extract links, metadata
     │   3. If render needed: Cloudflare Browser Rendering
     │   4. Diff raw vs rendered
     │   5. Run audit checks
     │   6. Enqueue discovered URLs
     │   7. Write results to D1 + KV progress
     ▼
 Audit findings in D1

Queue and concurrency

The crawler uses Cloudflare Queues with at-least-once delivery. This means the same URL may be delivered more than once — the crawler is idempotent by design. Duplicate URL processing is handled by a content hash check: if the same URL with the same content hash was processed in the current audit run, the duplicate message is acknowledged and discarded without re-processing.

Per-domain concurrency is enforced via Durable Objects. Each domain gets a Durable Object that tracks how many concurrent fetches are in flight, enforcing the configured maximum (default: 3 concurrent requests per domain). This prevents the crawler from hammering a site’s server.

Checkpoint and resume

Crawl state is checkpointed to KV (CRAWL_PROGRESS). If a crawl worker instance is interrupted (Worker timeout, edge node restart), the next worker instance picks up from the last checkpoint rather than restarting the entire crawl.

The checkpoint stores:

• URLs discovered but not yet processed (the frontier)
• URLs already processed in this crawl (the seen set, as a hash)
• Crawl statistics (pages fetched, errors, start time)

Important: KV progress writes are batched — not one write per page. Writing to KV on every page fetch caused KV 429 rate limit errors at scale. The crawler batches checkpoint writes every 50 pages or 30 seconds, whichever comes first.

Retry policy

Failed URL fetches (network error, 5xx response, timeout) are retried with exponential backoff up to 3 times. After 3 failures, the URL is marked with status error and a finding is generated noting the failure. HTTP 4xx responses are not retried — they’re recorded as-is.

Crawl modes

Mode	Trigger	What it does
html	Default first crawl	Raw HTTP fetch only, no browser rendering
rendered	Configured or auto-detected	Raw fetch + browser rendering via Cloudflare Browser Rendering
validation	Targeted re-crawl	Re-fetches specific URLs to verify fixes
delta	Change-based	Re-crawls only URLs where content hash changed since last crawl

Browser rendering sessions are reused across pages via Durable Objects to avoid the overhead of creating a new session per page.

Audit check execution

After a URL is fetched and parsed, all applicable audit checks run in the same Worker invocation. This is efficient because the page data is already in memory — checks just transform it into findings. Check execution is synchronous and in-process (no additional queue hops per check).

See Audit Engine Overview for how checks are structured.