PageObservation

PageObservation is the primary output of the crawler for each fetched URL. It captures everything the crawl found about a page — raw HTML data, HTTP metadata, rendered DOM data if rendering was triggered, and links.

TypeScript interface

interface PageObservation {
  // Identity
  url: string;                    // normalized final URL (after redirects)
  crawlId: string;
  projectId: string;
  observedAt: string;             // ISO 8601 timestamp

  // HTTP metadata
  statusCode: number;
  redirectChain: RedirectHop[];   // empty if no redirects
  responseTimeMs: number;

  // Raw HTML pass
  rawTitle: string | null;
  rawMetaDescription: string | null;
  rawCanonical: string | null;
  rawMetaRobots: string | null;
  rawXRobotsTag: string | null;   // from HTTP header
  rawWordCount: number;
  rawLinkCount: number;           // total <a href> count
  contentHash: string;            // SHA-256 of raw body

  // Rendered DOM pass (null if not rendered)
  renderedTitle: string | null;
  renderedMetaDescription: string | null;
  renderedCanonical: string | null;
  renderedMetaRobots: string | null;
  renderedWordCount: number | null;
  renderedLinkCount: number | null;
  renderTimeMs: number | null;
  renderErrors: string[];

  // Links extracted
  internalLinks: Link[];
  externalLinks: Link[];

  // Structured data
  structuredDataBlocks: StructuredDataBlock[];

  // hreflang
  hreflangAttributes: HreflangAttribute[];

  // Security headers
  securityHeaders: SecurityHeaders;

  // AI access
  robotsTxtAccessByAgent: Record<string, 'allowed' | 'blocked'>;
  llmsTxtPresent: boolean | null;  // null if not checked

  // Open Graph / social
  ogTitle: string | null;
  ogDescription: string | null;
  ogImage: string | null;

  // HTTP response headers (SEO-relevant)
  linkHeader: string | null;       // Link: rel=canonical from HTTP header
  varyHeader: string | null;
  cacheControl: string | null;
  contentType: string | null;
  httpVersion: string;             // "HTTP/1.1", "HTTP/2", "HTTP/3"
}

Supporting types

interface RedirectHop {
  url: string;
  statusCode: number;
}

interface Link {
  href: string;
  anchorText: string;
  rel: string[];      // ["nofollow", "sponsored"], etc.
  isJsOnly: boolean;  // only present in rendered DOM
}

interface StructuredDataBlock {
  type: string;           // "@type" value, e.g. "Product", "Article"
  raw: Record<string, unknown>; // full parsed JSON-LD
  parseError: string | null;
}

interface HreflangAttribute {
  lang: string;   // e.g. "en-US", "x-default"
  href: string;
}

interface SecurityHeaders {
  strictTransportSecurity: string | null;
  contentSecurityPolicy: string | null;
  xContentTypeOptions: string | null;
  xFrameOptions: string | null;
  permissionsPolicy: string | null;
}

Storage

PageObservation is normalized into several D1 tables: crawl_pages (core fields), page_links (links), page_structured_data (JSON-LD blocks), page_hreflang (hreflang attributes). The raw HTML body is stored in R2 (not D1) with a retention policy.

Parity calculation

Fields with both raw* and rendered* variants are compared field-by-field during audit check execution. Any difference is a potential render-parity finding. The comparison happens at check-run time, not at crawl time — the observation stores both values and the check decides what constitutes a meaningful difference.