mirror of
https://github.com/addyosmani/agent-skills.git
synced 2026-08-12 18:07:26 +02:00
Addy Osmani
45e1449138
There was no way to measure whether skills trigger correctly, stay distinct, or change agent behavior. This adds evals, aligned with what the community has converged on, with a deterministic CI tier on top: - evals/cases/<skill>.json for all 24 skills. The evals[] block uses Anthropic skill-creator's evals.json schema verbatim (id, prompt, expected_output, expectations[]) so its runner, benchmarks, and eval viewer work against our files unmodified. A trigger block (this repo's extension) adds positive/negative routing prompts per skill. - scripts/run-evals.js, zero-dependency runner: Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill descriptions (positive prompts must rank top-k, negative prompts must not rank first), catalog collision detection between skill descriptions, schema and coverage checks. Tier 3 (opt-in): --behavioral <skill> executes each eval through headless claude -p and grades the transcript against expectations[] (superpowers-style); --dry-run previews without spending tokens. - CI: run the deterministic tier in the validate-skills job. - Docs: evals/README.md defines the framework and prior art; CONTRIBUTING requires an eval file for new skills (warning-level in the runner until in-flight skill PRs clear); CLAUDE.md pointers. Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72 positive prompts, zero catalog collisions.