Audit Engine Overview

The audit engine transforms raw crawl data into structured findings. It runs inside the crawler worker — after a page is fetched and parsed, all applicable checks run in the same Worker invocation against the in-memory page data.

Pipeline stages

Page fetched (raw HTML + rendered DOM)
         │
         ▼
    Data extraction
    ─────────────────────────────
    • Status code, headers
    • Canonical, meta robots
    • Title, description
    • Internal links, external links
    • Structured data blocks
    • hreflang attributes
    • Security headers
    • Content hash, word count
         │
         ▼
    Audit checks run
    ─────────────────────────────
    Each check receives:
    • PageData (raw + rendered)
    • Site context (robots.txt, sitemap)
    • Project config (thresholds, exclusions)

    Each check produces:
    • AuditFinding[] (may be empty if no issue)
         │
         ▼
    Template aggregation
    ─────────────────────────────
    Findings with same checkId across
    URLs sharing a template pattern
    are aggregated into template-scope findings
         │
         ▼
    Findings written to D1

Check categories

Checks are organized into categories. See Check Registry for the full list. Categories:

• Indexation & canonicalization — noindex conflicts, canonical chains, protocol normalization, soft 404s
• Discovery & crawlability — robots.txt, orphan pages, broken links, crawl depth, crawl traps
• AI crawler access — bot-specific blocking, llms.txt, JS-only content, crawl-delay
• Render & JavaScript SEO — raw vs rendered diffs, client-side canonical changes, hydration mismatches
• Structured data — JSON-LD syntax, required properties, visible-text mismatch, deprecated types
• Performance & CWV — LCP, INP, CLS, TTFB, bfcache, Speculation Rules, image optimization
• Security & trust — HTTPS enforcement, HSTS, CSP, X-Content-Type-Options, certificate validity
• International SEO — hreflang reciprocity, invalid codes, x-default coverage
• Content quality — duplicate clusters, thin content, content freshness
• E-E-A-T signals — author schema, organization schema, topical authority
• AI-surface readiness — passage ranking structure, citation-worthy structure, content freshness for AI
• IndexNow — endpoint detection, sitemap freshness, lastmod accuracy

Check interface

All checks implement the same interface:

interface AuditCheck {
  checkId: string;
  name: string;
  description: string;
  scope: FindingScope;
  defaultSeverity: Severity;
  defaultConfidence: Confidence;

  run(context: CheckContext): Promise<AuditFinding[]>;
}

interface CheckContext {
  page: PageData;           // raw + rendered page data
  site: SiteContext;        // robots.txt, sitemap, project settings
  connectors: ConnectorData; // GSC/GA4/CrUX data if connected
}

Checks are pure functions — they receive input and return findings. No side effects, no network calls, no database writes. This makes them easy to test in isolation.

Template aggregation

After all checks run on all pages, the engine groups findings by checkId + templateId. If more than 3 pages share a template (detected by URL pattern similarity) and the same finding, the individual page findings are collapsed into a single template-scope finding.

This prevents 10,000 identical findings for a template bug. The template finding records the affected URL count and a sample of affected URLs.

Test harness

Checks can be tested in isolation using the Vitest test harness in workers/crawler/src/__tests__/. Each check should have:

• A positive test (issue correctly detected on a fixture with the defect)
• A negative test (no false positive on a clean fixture)
• An edge case test (e.g., canonical loop, redirect chain, empty robots.txt)

Every high-severity check ships with all three test types before it is considered complete.