skills: add SEO/AEO audit and grounded-JSON-LD skills from citeworthyio/seo-agent

Evaluated citeworthyio/seo-agent for reusable process knowledge. Relevant to this fleet's WordPress content sites (granja travel news, content-agent pipeline): a crawl-diagnose-approve-recrawl-measure loop for SEO metadata changes, a three-layer AI-crawler accessibility audit (robots policy, bot delivery, readable content), and a fail-closed method for generating and safely injecting Article JSON-LD. Keyword research and citation-probe pieces of the source repo were skipped as underdeveloped or too infra-coupled to be reusable here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-11 13:28:29 +02:00
parent c7ce105e00
commit ae61c7ce7b
4 changed files with 225 additions and 0 deletions
@@ -0,0 +1,68 @@
+---
+name: ai-crawler-accessibility-audit
+description: Use when auditing whether a site can be crawled, read, and cited by AI answer engines, especially for JavaScript-rendered sites, CDN bot controls, robots.txt policy, llms.txt, or unexplained absence from AI citations.
+license: MIT
+source: adapted from https://github.com/citeworthyio/seo-agent/blob/b78bb487f2819c3c7cc2ba6f3f5626ea0b841840/src/aeo.ts
+---
+
+# AI Crawler Accessibility Audit
+
+Audit three separate layers: policy, fetch delivery, and readable content. A
+healthy `<head>` does not make a client-rendered page citable because AI crawlers
+and user-fetchers generally do not execute the site's JavaScript.
+
+## 1. Separate answer-engine bots from training bots
+
+Read the live `/robots.txt` and evaluate its actual groups using longest-match
+robots rules; an exact user-agent group takes precedence over `*`, and `Allow`
+wins equal-length ties.
+
+Treat blocking answer/search fetchers as a discoverability defect. Check at
+least OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot,
+Perplexity-User, Googlebot, Bingbot, Meta-WebIndexer, DuckAssistBot,
+MistralAI-User, Amazonbot, and Applebot. Googlebot and Bingbot also feed AI
+answer surfaces.
+
+Treat training-only crawlers separately: GPTBot, ClaudeBot, CCBot,
+Google-Extended, Applebot-Extended, meta-externalagent, and Bytespider. Allowing
+or denying these is a content-policy choice; do not claim that blocking them
+alone removes the site from citations. Explicit groups are preferable to an
+accidental allow-all because they document intent and survive managed
+`robots.txt` defaults.
+
+## 2. Validate agent-facing text resources by content
+
+Fetch `/llms.txt` and, if advertised, `/llms-full.txt`. A `404` is an honest
+absence. A `200` containing an HTML application shell or `text/html` is a soft
+404 and worse: an agent asked for a text map and received misleading content.
+Serve a real plain-text/Markdown resource or a clean absence. Treat these files
+as cheap insurance, not as a proven ranking lever.
+
+## 3. Compare ordinary and AI-user-agent delivery
+
+From successful content pages, rotate a small daily sample rather than checking
+the same first URLs forever. Fetch each sampled URL once normally and once with
+the relevant AI user agent.
+
+- If the normal request is `200` but the AI request is `403`, `429`, or `451`,
+  investigate CDN/WAF/bot-management policy. The controlled user-agent
+  difference is stronger evidence than either status alone.
+- If the AI response succeeds, strip scripts, styles, and markup and inspect the
+  visible body. Fewer than roughly 200 characters, without full content in
+  `Article.articleBody`, is a useful empty-shell tripwire; inspect borderline
+  pages rather than treating the number as a universal quality score.
+- Record transient fetch errors as unknown/check errors. Do not turn a failed
+  measurement into a claim that the site is blocked or healthy.
+
+## 4. Fix the serving layer, then verify it
+
+Prefer server rendering or static rendering. If that is not viable, serve a
+real, fetchable HTML or Markdown content lane from the same source data, with
+correct content negotiation and `Vary` headers. `articleBody` can be a fallback,
+but visible server-delivered content is stronger.
+
+Re-run both the ordinary and AI-user-agent requests against the public URL.
+Only suppress an empty-body finding when there is positive evidence that an
+alternate readable twin is live for that exact path. A configured twin that
+cannot actually be fetched must not hide a true failure.
+