Adding competitors

Competitor intelligence tracks which keywords your competitors rank for that you don’t, where you overlap, and where new content opportunities sit. Before any of that analysis exists, you have to tell VisibilityIQ who your competitors are — and understand how it crawls them, because the crawl is deliberately polite and rate-limited.

What you’ll need

A project with the rankings connector configured, so competitor keywords can be cross-referenced against organic data.
The domains of the competitors you want to track (e.g. competitor.com).
Awareness of the 20-competitor cap so you spend those slots on your closest rivals.

Add a competitor manually

In your project, go to Competitor Intel.
Click Add competitor.
Enter the competitor’s domain.
Click Add.

VisibilityIQ crawls the competitor and cross-references the extracted keywords with organic data for that domain. Manually added competitors are crawled right away.

Auto-discovered competitors

VisibilityIQ also suggests competitors it finds through keyword overlap. Each suggestion carries a source that records how it was found — manual, serp_overlap, gsc_import, organic_api, or industry_match — and a discovery_score from 0 to 1 representing organic overlap with your domain. A score of 0.8 means roughly 80% of your tracked keywords also appear in their detected keyword set; higher means a more direct competitor.

Auto-discovered competitors are not crawled until you confirm them. This is intentional: it keeps you in control of which domains consume a competitor slot and a crawl job, and prevents the system from fanning out across loosely related sites. Review the suggestions, confirm the ones that are genuinely competing for your terms, and dismiss the rest.

The 20-competitor cap

Each project supports up to 20 active competitors. The cap is a focus mechanism, not a billing limit — beyond about twenty domains, gap analysis becomes noise. Spend the slots on the rivals ranking for the same commercial and informational terms you target. If you hit the cap and want to add a new competitor, remove a weaker one first.

Crawl ethics

The competitor crawler is built to be a good citizen, and these limits are non-negotiable in the engine:

Identifies itself as VisibilityIQ/1.0 (+https://visibilityiq.io/bot) so site owners can see who’s visiting in their logs.
Respects robots.txt — it checks before fetching any competitor page and honours disallow rules.
Caps the crawl at 20 pages per job. VisibilityIQ does not deep-crawl competitors; it samples enough to extract a keyword signal and stops.
Throttles with a minimum one-second delay between page fetches and a 25-second job timeout, after which the job ends gracefully and is marked stale rather than hammering the server.
Limits recrawls to at most once per 7 days per domain, so a competitor is never hit repeatedly. To force a fresh crawl sooner you must trigger it explicitly.

What VisibilityIQ extracts

From each crawl it pulls keywords from page titles, H1–H3 tags, meta descriptions, and URL paths, distinguishing primary signals (title and H1 extractions) from secondary ones, and estimates organic keyword overlap with your domain. Full page HTML is discarded after extraction — only the keyword signal is stored.