Follow-up to #419, addressing the remaining normative npm-family
commands federicobartoli's grep on #404 identified:
- incremental-implementation: the four increment-checklist exit criteria
and the example prompt now use the repository's own test/build/
typecheck/lint commands, pointing at the TDD skill's Discover the
Stack First section
- planning-and-task-breakdown: task-template verification lines use the
template's placeholder style instead of hardcoded npm commands
- shipping-and-launch: the security checkbox names the ecosystem's
dependency audit rather than npm audit alone
- debugging-and-error-recovery: the diagnosis/bisect/verify command
blocks are labeled as npm examples with substitution notes
- references/security-checklist.md: OWASP row 6 generalizes npm audit
to the native dependency audit
- security-and-hardening needed no change: its SKILL.md was already
neutralized (detected-package-manager wording)
Ride-along: pins the below-zero debit behavior (ValueError) in TDD eval
case 3, per nucliweb's non-blocking review note on #419.
- SKILL.md gains a 'Discover the Stack First' step: identify the
language, build system, checked-in wrappers, test framework, and
documented commands before choosing any test command, and run the
repository's own focused/full-suite commands throughout the loop
- verification checklist no longer hardcodes 'npm test'; red flag added
for reaching for a default test command without checking the repo
- references/testing-patterns.md relabeled in place as JavaScript/
TypeScript examples illustrating the universal principles (no
relocation, per the #361/#236 sequencing agreed in the issue)
- new behavioral eval: a Python/unittest fixture verifying the skill
adapts to non-npm repository tooling
Two latent Tier-3 bugs caught in review before the path was ever exercised:
- The grader prompt embeds the full stream-json trace (up to megabytes) and
was passed as an argv entry, which would fail with E2BIG on any real run.
It now goes to `claude -p` over stdin; the executor prompt moves to stdin
for the same reason.
- The executor ran headless with no permission mode, so file edits and
command runs could be denied, forcing the narrate-instead-of-perform
failure mode that trace grading exists to catch. It now runs with
--permission-mode acceptEdits and a pre-approved tool list
(Read,Glob,Grep,Edit,Write,Bash), documented in the README.
Wire the framework docs to the tracking issues: #351 (description
vocabulary gaps) and #352 (Tier 3 graduation + deterministic ratchets),
so warning-level checks have an explicit promotion path instead of
becoming permanent.
Per review from @federicobartoli and @nucliweb on #342:
Tier 3 (behavioral):
- Grade the execution trace, not the final output: executor runs with
--output-format stream-json --verbose so the grader judges tool calls
and file edits rather than the model's self-reporting.
- Run each eval in a throwaway workspace; files[] fixtures materialize
from evals/fixtures/ so evals can operate on real code.
- Node-level timeouts on executor and grader calls; grader output parsed
and shape-validated before writing (raw saved on failure); the trace is
fenced as untrusted data in the grader prompt.
- All 24 behavioral evals flagged trust_level: "provisional" until they
gain fixtures; the runner surfaces this and exits nonzero on failed
expectations.
Tier 2 (deterministic):
- Negative triggers accept an "owner" skill that must outrank this one,
turning them into pairwise routing tests that cannot pass vacuously;
37 of 48 negatives now declare owners (the rest are tracked in #351).
- Warn when a case file is below the documented minimums (3 positive /
2 negative / 1 behavioral); promotion to error tracked in #352.
- Stemmer: cluster trailing y/i ("simplify"/"simplifies").
Baseline holds: 120 checks, 0 errors, 85% trigger rank-1 rate.
There was no way to measure whether skills trigger correctly, stay
distinct, or change agent behavior. This adds evals, aligned with what
the community has converged on, with a deterministic CI tier on top:
- evals/cases/<skill>.json for all 24 skills. The evals[] block uses
Anthropic skill-creator's evals.json schema verbatim (id, prompt,
expected_output, expectations[]) so its runner, benchmarks, and eval
viewer work against our files unmodified. A trigger block (this
repo's extension) adds positive/negative routing prompts per skill.
- scripts/run-evals.js, zero-dependency runner:
Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill
descriptions (positive prompts must rank top-k, negative prompts
must not rank first), catalog collision detection between skill
descriptions, schema and coverage checks.
Tier 3 (opt-in): --behavioral <skill> executes each eval through
headless claude -p and grades the transcript against expectations[]
(superpowers-style); --dry-run previews without spending tokens.
- CI: run the deterministic tier in the validate-skills job.
- Docs: evals/README.md defines the framework and prior art;
CONTRIBUTING requires an eval file for new skills (warning-level in
the runner until in-flight skill PRs clear); CLAUDE.md pointers.
Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72
positive prompts, zero catalog collisions.
ayobamiseun