MalinandClaude Sonnet 5
ae61c7ce7b
Evaluated citeworthyio/seo-agent for reusable process knowledge. Relevant to this fleet's WordPress content sites (granja travel news, content-agent pipeline): a crawl-diagnose-approve-recrawl-measure loop for SEO metadata changes, a three-layer AI-crawler accessibility audit (robots policy, bot delivery, readable content), and a fail-closed method for generating and safely injecting Article JSON-LD. Keyword research and citation-probe pieces of the source repo were skipped as underdeveloped or too infra-coupled to be reusable here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3.5 KiB
name, description, license, source
| name | description | license | source |
|---|---|---|---|
| ai-crawler-accessibility-audit | Use when auditing whether a site can be crawled, read, and cited by AI answer engines, especially for JavaScript-rendered sites, CDN bot controls, robots.txt policy, llms.txt, or unexplained absence from AI citations. | MIT | adapted from https://github.com/citeworthyio/seo-agent/blob/b78bb487f2819c3c7cc2ba6f3f5626ea0b841840/src/aeo.ts |
AI Crawler Accessibility Audit
Audit three separate layers: policy, fetch delivery, and readable content. A
healthy <head> does not make a client-rendered page citable because AI crawlers
and user-fetchers generally do not execute the site's JavaScript.
1. Separate answer-engine bots from training bots
Read the live /robots.txt and evaluate its actual groups using longest-match
robots rules; an exact user-agent group takes precedence over *, and Allow
wins equal-length ties.
Treat blocking answer/search fetchers as a discoverability defect. Check at least OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Meta-WebIndexer, DuckAssistBot, MistralAI-User, Amazonbot, and Applebot. Googlebot and Bingbot also feed AI answer surfaces.
Treat training-only crawlers separately: GPTBot, ClaudeBot, CCBot,
Google-Extended, Applebot-Extended, meta-externalagent, and Bytespider. Allowing
or denying these is a content-policy choice; do not claim that blocking them
alone removes the site from citations. Explicit groups are preferable to an
accidental allow-all because they document intent and survive managed
robots.txt defaults.
2. Validate agent-facing text resources by content
Fetch /llms.txt and, if advertised, /llms-full.txt. A 404 is an honest
absence. A 200 containing an HTML application shell or text/html is a soft
404 and worse: an agent asked for a text map and received misleading content.
Serve a real plain-text/Markdown resource or a clean absence. Treat these files
as cheap insurance, not as a proven ranking lever.
3. Compare ordinary and AI-user-agent delivery
From successful content pages, rotate a small daily sample rather than checking the same first URLs forever. Fetch each sampled URL once normally and once with the relevant AI user agent.
- If the normal request is
200but the AI request is403,429, or451, investigate CDN/WAF/bot-management policy. The controlled user-agent difference is stronger evidence than either status alone. - If the AI response succeeds, strip scripts, styles, and markup and inspect the
visible body. Fewer than roughly 200 characters, without full content in
Article.articleBody, is a useful empty-shell tripwire; inspect borderline pages rather than treating the number as a universal quality score. - Record transient fetch errors as unknown/check errors. Do not turn a failed measurement into a claim that the site is blocked or healthy.
4. Fix the serving layer, then verify it
Prefer server rendering or static rendering. If that is not viable, serve a
real, fetchable HTML or Markdown content lane from the same source data, with
correct content negotiation and Vary headers. articleBody can be a fallback,
but visible server-delivered content is stronger.
Re-run both the ordinary and AI-user-agent requests against the public URL. Only suppress an empty-body finding when there is positive evidence that an alternate readable twin is live for that exact path. A configured twin that cannot actually be fetched must not hide a true failure.