diff --git a/README.md b/README.md index 7b10c28..a7eaa9e 100644 --- a/README.md +++ b/README.md @@ -100,6 +100,15 @@ than assuming the delegate can fetch it itself. - `finishing-development-branches` -- verify the exact branch tip, confirm its base, and require an explicit integration choice before merge, Gitea push/PR, or cleanup. +- `seo-change-lifecycle` -- audit the metadata crawlers actually receive, move + fixes through approval and recrawl verification, then evaluate delayed + page-level Search Console signals without claiming causation. +- `ai-crawler-accessibility-audit` -- distinguish answer-engine bots from + training crawlers, detect robots/CDN blocks and misleading llms.txt soft + 404s, and verify that non-JS fetchers receive readable page content. +- `grounded-article-jsonld` -- generate Article structured data only from + observed page facts, validate it at every boundary, and serialize it safely + for additive script injection. ## Provenance @@ -115,6 +124,7 @@ each skill's frontmatter: - [Leonxlnx/taste-skill](https://github.com/Leonxlnx/taste-skill) (MIT) - [JuliusBrussee/caveman](https://github.com/JuliusBrussee/caveman) (MIT; evaluated, no skill retained) +- [citeworthyio/seo-agent](https://github.com/citeworthyio/seo-agent) (MIT) ## Vetting external skills diff --git a/skills/ai-crawler-accessibility-audit/SKILL.md b/skills/ai-crawler-accessibility-audit/SKILL.md new file mode 100644 index 0000000..a50a5ca --- /dev/null +++ b/skills/ai-crawler-accessibility-audit/SKILL.md @@ -0,0 +1,68 @@ +--- +name: ai-crawler-accessibility-audit +description: Use when auditing whether a site can be crawled, read, and cited by AI answer engines, especially for JavaScript-rendered sites, CDN bot controls, robots.txt policy, llms.txt, or unexplained absence from AI citations. +license: MIT +source: adapted from https://github.com/citeworthyio/seo-agent/blob/b78bb487f2819c3c7cc2ba6f3f5626ea0b841840/src/aeo.ts +--- + +# AI Crawler Accessibility Audit + +Audit three separate layers: policy, fetch delivery, and readable content. A +healthy `` does not make a client-rendered page citable because AI crawlers +and user-fetchers generally do not execute the site's JavaScript. + +## 1. Separate answer-engine bots from training bots + +Read the live `/robots.txt` and evaluate its actual groups using longest-match +robots rules; an exact user-agent group takes precedence over `*`, and `Allow` +wins equal-length ties. + +Treat blocking answer/search fetchers as a discoverability defect. Check at +least OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, +Perplexity-User, Googlebot, Bingbot, Meta-WebIndexer, DuckAssistBot, +MistralAI-User, Amazonbot, and Applebot. Googlebot and Bingbot also feed AI +answer surfaces. + +Treat training-only crawlers separately: GPTBot, ClaudeBot, CCBot, +Google-Extended, Applebot-Extended, meta-externalagent, and Bytespider. Allowing +or denying these is a content-policy choice; do not claim that blocking them +alone removes the site from citations. Explicit groups are preferable to an +accidental allow-all because they document intent and survive managed +`robots.txt` defaults. + +## 2. Validate agent-facing text resources by content + +Fetch `/llms.txt` and, if advertised, `/llms-full.txt`. A `404` is an honest +absence. A `200` containing an HTML application shell or `text/html` is a soft +404 and worse: an agent asked for a text map and received misleading content. +Serve a real plain-text/Markdown resource or a clean absence. Treat these files +as cheap insurance, not as a proven ranking lever. + +## 3. Compare ordinary and AI-user-agent delivery + +From successful content pages, rotate a small daily sample rather than checking +the same first URLs forever. Fetch each sampled URL once normally and once with +the relevant AI user agent. + +- If the normal request is `200` but the AI request is `403`, `429`, or `451`, + investigate CDN/WAF/bot-management policy. The controlled user-agent + difference is stronger evidence than either status alone. +- If the AI response succeeds, strip scripts, styles, and markup and inspect the + visible body. Fewer than roughly 200 characters, without full content in + `Article.articleBody`, is a useful empty-shell tripwire; inspect borderline + pages rather than treating the number as a universal quality score. +- Record transient fetch errors as unknown/check errors. Do not turn a failed + measurement into a claim that the site is blocked or healthy. + +## 4. Fix the serving layer, then verify it + +Prefer server rendering or static rendering. If that is not viable, serve a +real, fetchable HTML or Markdown content lane from the same source data, with +correct content negotiation and `Vary` headers. `articleBody` can be a fallback, +but visible server-delivered content is stronger. + +Re-run both the ordinary and AI-user-agent requests against the public URL. +Only suppress an empty-body finding when there is positive evidence that an +alternate readable twin is live for that exact path. A configured twin that +cannot actually be fetched must not hide a true failure. + diff --git a/skills/grounded-article-jsonld/SKILL.md b/skills/grounded-article-jsonld/SKILL.md new file mode 100644 index 0000000..216f38c --- /dev/null +++ b/skills/grounded-article-jsonld/SKILL.md @@ -0,0 +1,61 @@ +--- +name: grounded-article-jsonld +description: Use when generating, reviewing, storing, or injecting Article JSON-LD from page content. Prevent fabricated machine-readable claims and script-context injection while preserving existing structured data. +license: MIT +source: adapted from https://github.com/citeworthyio/seo-agent/blob/b78bb487f2819c3c7cc2ba6f3f5626ea0b841840/README.md#structured-data-json-ld and src/overrides.ts +--- + +# Grounded Article JSON-LD + +Structured data is a factual publication, not decorative copy. An invented +author or date becomes a machine-readable claim under the site's name and may +be consumed without a reader ever seeing and correcting it. + +## Generate only from observed page facts + +Give the generator only the page content actually fetched: title, description, +canonical, visible article content, and explicit author/date/image fields. Tell +it to omit anything it cannot point to in that material. Never infer an author, +publisher, organization, date, image, rating, price, or other plausible default. + +For an article page missing structured data, a minimal correct `Article` object +with `@context`, `@type`, and a grounded headline is better than a rich-looking +object containing guesses. Keep automated generation narrow to page-level types +such as `Article`; organization, product, rating, and similar business claims +need their own authoritative data source and review process. + +## Validate at every boundary + +Use the same fail-closed validator for generated drafts, manual proposals, and +the final publishing write: + +1. Parse as JSON. +2. Accept one object or a non-empty array of objects, never primitives or null. +3. Require every node to have an `@context` referencing `schema.org` and a + non-empty `@type`. +4. Enforce a bounded canonical serialization (the source implementation uses + 8,192 characters). +5. Reject the value rather than storing it if any check fails. + +Using one validator matters: otherwise a draft can pass review and fail at +publish time, or a manual path can bypass the protections applied to AI output. + +## Canonicalize for safe script embedding + +Store JSON text without a `