MalinandClaude Sonnet 5
ae61c7ce7b
Evaluated citeworthyio/seo-agent for reusable process knowledge. Relevant to this fleet's WordPress content sites (granja travel news, content-agent pipeline): a crawl-diagnose-approve-recrawl-measure loop for SEO metadata changes, a three-layer AI-crawler accessibility audit (robots policy, bot delivery, readable content), and a fail-closed method for generating and safely injecting Article JSON-LD. Keyword research and citation-probe pieces of the source repo were skipped as underdeveloped or too infra-coupled to be reusable here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
69 lines
3.5 KiB
Markdown
69 lines
3.5 KiB
Markdown
---
|
|
name: ai-crawler-accessibility-audit
|
|
description: Use when auditing whether a site can be crawled, read, and cited by AI answer engines, especially for JavaScript-rendered sites, CDN bot controls, robots.txt policy, llms.txt, or unexplained absence from AI citations.
|
|
license: MIT
|
|
source: adapted from https://github.com/citeworthyio/seo-agent/blob/b78bb487f2819c3c7cc2ba6f3f5626ea0b841840/src/aeo.ts
|
|
---
|
|
|
|
# AI Crawler Accessibility Audit
|
|
|
|
Audit three separate layers: policy, fetch delivery, and readable content. A
|
|
healthy `<head>` does not make a client-rendered page citable because AI crawlers
|
|
and user-fetchers generally do not execute the site's JavaScript.
|
|
|
|
## 1. Separate answer-engine bots from training bots
|
|
|
|
Read the live `/robots.txt` and evaluate its actual groups using longest-match
|
|
robots rules; an exact user-agent group takes precedence over `*`, and `Allow`
|
|
wins equal-length ties.
|
|
|
|
Treat blocking answer/search fetchers as a discoverability defect. Check at
|
|
least OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot,
|
|
Perplexity-User, Googlebot, Bingbot, Meta-WebIndexer, DuckAssistBot,
|
|
MistralAI-User, Amazonbot, and Applebot. Googlebot and Bingbot also feed AI
|
|
answer surfaces.
|
|
|
|
Treat training-only crawlers separately: GPTBot, ClaudeBot, CCBot,
|
|
Google-Extended, Applebot-Extended, meta-externalagent, and Bytespider. Allowing
|
|
or denying these is a content-policy choice; do not claim that blocking them
|
|
alone removes the site from citations. Explicit groups are preferable to an
|
|
accidental allow-all because they document intent and survive managed
|
|
`robots.txt` defaults.
|
|
|
|
## 2. Validate agent-facing text resources by content
|
|
|
|
Fetch `/llms.txt` and, if advertised, `/llms-full.txt`. A `404` is an honest
|
|
absence. A `200` containing an HTML application shell or `text/html` is a soft
|
|
404 and worse: an agent asked for a text map and received misleading content.
|
|
Serve a real plain-text/Markdown resource or a clean absence. Treat these files
|
|
as cheap insurance, not as a proven ranking lever.
|
|
|
|
## 3. Compare ordinary and AI-user-agent delivery
|
|
|
|
From successful content pages, rotate a small daily sample rather than checking
|
|
the same first URLs forever. Fetch each sampled URL once normally and once with
|
|
the relevant AI user agent.
|
|
|
|
- If the normal request is `200` but the AI request is `403`, `429`, or `451`,
|
|
investigate CDN/WAF/bot-management policy. The controlled user-agent
|
|
difference is stronger evidence than either status alone.
|
|
- If the AI response succeeds, strip scripts, styles, and markup and inspect the
|
|
visible body. Fewer than roughly 200 characters, without full content in
|
|
`Article.articleBody`, is a useful empty-shell tripwire; inspect borderline
|
|
pages rather than treating the number as a universal quality score.
|
|
- Record transient fetch errors as unknown/check errors. Do not turn a failed
|
|
measurement into a claim that the site is blocked or healthy.
|
|
|
|
## 4. Fix the serving layer, then verify it
|
|
|
|
Prefer server rendering or static rendering. If that is not viable, serve a
|
|
real, fetchable HTML or Markdown content lane from the same source data, with
|
|
correct content negotiation and `Vary` headers. `articleBody` can be a fallback,
|
|
but visible server-delivered content is stronger.
|
|
|
|
Re-run both the ordinary and AI-user-agent requests against the public URL.
|
|
Only suppress an empty-body finding when there is positive evidence that an
|
|
alternate readable twin is live for that exact path. A configured twin that
|
|
cannot actually be fetched must not hide a true failure.
|
|
|