llms.txt not detected

Symptoms

• Your audit reports llms.txt as missing even though you published one.
• An llms.txt finding doesn’t clear after you uploaded the file.
• The file loads in your browser but VisibilityIQ still doesn’t register it.

VisibilityIQ checks for llms.txt at the site root during the crawl and records its presence and contents. The check looks in one place only — https://yourdomain.com/llms.txt — so the usual cause is a file that isn’t actually served from the root, is served with the wrong content type, or is blocked from the crawler. The detection is also tied to the crawl: until a fresh crawl runs, the result reflects the last crawl, not your latest upload.

Likely causes

• Wrong location. The file sits at a subpath (/.well-known/llms.txt, /docs/llms.txt) instead of the exact root path /llms.txt.
• Wrong content type. It’s served as text/html, as a download (application/octet-stream), or behind a redirect, so it isn’t read as the plain-text resource expected.
• Blocked by robots.txt. A Disallow rule covering /llms.txt stops the crawler from fetching it. VisibilityIQ respects robots.txt for every user-agent and won’t bypass the rule.
• Not yet recrawled. You published the file after the last crawl, so the stale result still shows “missing”.
• Returns a non-200. The path 404s, redirects, or returns a soft error — there’s no file to record.

How to fix it

1. Confirm the exact URL. Open https://yourdomain.com/llms.txt directly. It must resolve at the root with a 200 status and no redirect. If it only works at a subpath, move or alias it to the root.
2. Serve it as plain text. Set the response Content-Type to text/plain (text/plain; charset=utf-8 is fine). If your host forces a download or returns HTML, fix the server/CDN rule for that path.
3. Unblock the crawler. Check yourdomain.com/robots.txt for any Disallow matching /llms.txt. Remove it so the file is fetchable. Confirm no WAF or CDN rule challenges requests for the path.
4. Verify the contents. The check records the file’s contents and assesses completeness, so make sure it actually describes your key sections rather than being empty or a placeholder.
5. Re-run the crawl. Detection only updates on a fresh crawl. Trigger a new audit; the crawl resumes and re-checks the root. Once the file is fetched, the finding clears and the recorded contents update.

Verify it worked

After the recrawl completes, the llms.txt finding should resolve and the audit should show the file as present. If you want to confirm what the crawler saw, the file’s recorded contents appear in the finding evidence — compare them against what you published to rule out a stale cache.

Still stuck?

If the file loads at the root as plain text, isn’t blocked, and a fresh crawl still reports it missing, email support@visibilityiq365.com with the project domain so we can inspect what the crawler fetched. See also Crawl not completing if the recrawl itself stalls and Audit shows no issues or empty results when a crawl returns fewer findings than expected.