--- name: ai-crawler-accessibility-audit description: Use when auditing whether a site can be crawled, read, and cited by AI answer engines, especially for JavaScript-rendered sites, CDN bot controls, robots.txt policy, llms.txt, or unexplained absence from AI citations. license: MIT source: adapted from https://github.com/citeworthyio/seo-agent/blob/b78bb487f2819c3c7cc2ba6f3f5626ea0b841840/src/aeo.ts --- # AI Crawler Accessibility Audit Audit three separate layers: policy, fetch delivery, and readable content. A healthy `` does not make a client-rendered page citable because AI crawlers and user-fetchers generally do not execute the site's JavaScript. ## 1. Separate answer-engine bots from training bots Read the live `/robots.txt` and evaluate its actual groups using longest-match robots rules; an exact user-agent group takes precedence over `*`, and `Allow` wins equal-length ties. Treat blocking answer/search fetchers as a discoverability defect. Check at least OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Meta-WebIndexer, DuckAssistBot, MistralAI-User, Amazonbot, and Applebot. Googlebot and Bingbot also feed AI answer surfaces. Treat training-only crawlers separately: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, and Bytespider. Allowing or denying these is a content-policy choice; do not claim that blocking them alone removes the site from citations. Explicit groups are preferable to an accidental allow-all because they document intent and survive managed `robots.txt` defaults. ## 2. Validate agent-facing text resources by content Fetch `/llms.txt` and, if advertised, `/llms-full.txt`. A `404` is an honest absence. A `200` containing an HTML application shell or `text/html` is a soft 404 and worse: an agent asked for a text map and received misleading content. Serve a real plain-text/Markdown resource or a clean absence. Treat these files as cheap insurance, not as a proven ranking lever. ## 3. Compare ordinary and AI-user-agent delivery From successful content pages, rotate a small daily sample rather than checking the same first URLs forever. Fetch each sampled URL once normally and once with the relevant AI user agent. - If the normal request is `200` but the AI request is `403`, `429`, or `451`, investigate CDN/WAF/bot-management policy. The controlled user-agent difference is stronger evidence than either status alone. - If the AI response succeeds, strip scripts, styles, and markup and inspect the visible body. Fewer than roughly 200 characters, without full content in `Article.articleBody`, is a useful empty-shell tripwire; inspect borderline pages rather than treating the number as a universal quality score. - Record transient fetch errors as unknown/check errors. Do not turn a failed measurement into a claim that the site is blocked or healthy. ## 4. Fix the serving layer, then verify it Prefer server rendering or static rendering. If that is not viable, serve a real, fetchable HTML or Markdown content lane from the same source data, with correct content negotiation and `Vary` headers. `articleBody` can be a fallback, but visible server-delivered content is stronger. Re-run both the ordinary and AI-user-agent requests against the public URL. Only suppress an empty-body finding when there is positive evidence that an alternate readable twin is live for that exact path. A configured twin that cannot actually be fetched must not hide a true failure.