The /spec and /plan commands write spec, plan, and todo artifacts to paths that /build and the spec/plan skills read back. When a producer moves an artifact without updating the consumers, the pipeline breaks and nothing in CI catches it: the command-parity check only compares descriptions, not paths. PR #93 hit exactly this, pointing /spec and /plan at docs/features/[name]/ while /build still required SPEC.md and tasks/plan.md.
validate-artifact-paths.js enforces one canonical set of artifact paths across every file in the pipeline (the spec/plan/build commands, the spec-driven-development and planning-and-task-breakdown skills, and the getting-started and adoption guides). Any spec/plan/todo artifact path outside the allowlist fails CI, so changing the convention has to touch the allowlist and every guarded file in the same change.
Scope is deliberately narrow: only spec/plan/todo artifacts, only the pipeline files; it is not a general markdown path linter. Wired into the validate-commands CI job alongside its test.
actions/checkout@v4 and actions/setup-node@v4 target the deprecated Node 20
runner runtime, so every run logs forced-Node-24 deprecation annotations.
Harmonize checkout on the v6 major already used by the other three jobs,
bump setup-node to v6, and move node-version from Node 20 (EOL 2026-04-30)
to Node 24 LTS.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gives contributors a short intake path when a skill assumes the wrong
ecosystem or outdated steps, so gaps like #404 reach maintainers via
the repo instead of external channels. Closes the intake-only scope
from #412.
Co-authored-by: Cursor <cursoragent@cursor.com>
There was no way to measure whether skills trigger correctly, stay
distinct, or change agent behavior. This adds evals, aligned with what
the community has converged on, with a deterministic CI tier on top:
- evals/cases/<skill>.json for all 24 skills. The evals[] block uses
Anthropic skill-creator's evals.json schema verbatim (id, prompt,
expected_output, expectations[]) so its runner, benchmarks, and eval
viewer work against our files unmodified. A trigger block (this
repo's extension) adds positive/negative routing prompts per skill.
- scripts/run-evals.js, zero-dependency runner:
Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill
descriptions (positive prompts must rank top-k, negative prompts
must not rank first), catalog collision detection between skill
descriptions, schema and coverage checks.
Tier 3 (opt-in): --behavioral <skill> executes each eval through
headless claude -p and grades the transcript against expectations[]
(superpowers-style); --dry-run previews without spending tokens.
- CI: run the deterministic tier in the validate-skills job.
- Docs: evals/README.md defines the framework and prior art;
CONTRIBUTING requires an eval file for new skills (warning-level in
the runner until in-flight skill PRs clear); CLAUDE.md pointers.
Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72
positive prompts, zero catalog collisions.
The validate-skills job pinned actions/checkout@v4 while the other two
jobs in the same workflow (test-fresh-install, test-plugin-structure)
use @v6. Align all three on @v6 for consistency.
- scripts/validate-skills.js: zero-dependency Node.js validator that
checks every skill for valid frontmatter, description length (≤1024),
required sections (Overview, When to Use, Common Rationalizations,
Red Flags, Verification), and dead cross-skill references
- Skills with type:meta or exempt:sections in frontmatter skip section
checks; applied to using-agent-skills (meta) and idea-refine (legacy
structure predating the anatomy spec)
- CI: validate-skills job runs before plugin-manifest validation and
blocks merge on any error; uses Node 20, no npm install required
Runs on every push and PR:
1. Validates marketplace and plugin manifests
2. Tests marketplace add + plugin install end-to-end
No ANTHROPIC_API_KEY required — these are CLI/filesystem
operations, not LLM calls.
