mirror of
https://github.com/addyosmani/agent-skills.git
synced 2026-08-12 18:07:26 +02:00
Addy Osmani
194b2099c6
Per review from @federicobartoli and @nucliweb on #342: Tier 3 (behavioral): - Grade the execution trace, not the final output: executor runs with --output-format stream-json --verbose so the grader judges tool calls and file edits rather than the model's self-reporting. - Run each eval in a throwaway workspace; files[] fixtures materialize from evals/fixtures/ so evals can operate on real code. - Node-level timeouts on executor and grader calls; grader output parsed and shape-validated before writing (raw saved on failure); the trace is fenced as untrusted data in the grader prompt. - All 24 behavioral evals flagged trust_level: "provisional" until they gain fixtures; the runner surfaces this and exits nonzero on failed expectations. Tier 2 (deterministic): - Negative triggers accept an "owner" skill that must outrank this one, turning them into pairwise routing tests that cannot pass vacuously; 37 of 48 negatives now declare owners (the rest are tracked in #351). - Warn when a case file is below the documented minimums (3 positive / 2 negative / 1 behavioral); promotion to error tracked in #352. - Stemmer: cluster trailing y/i ("simplify"/"simplifies"). Baseline holds: 120 checks, 0 errors, 85% trigger rank-1 rate.
43 lines
1.4 KiB
JSON
43 lines
1.4 KiB
JSON
{
|
|
"skill_name": "browser-testing-with-devtools",
|
|
"trigger": {
|
|
"positive": [
|
|
{
|
|
"prompt": "Use Chrome DevTools to figure out why the checkout button click does nothing",
|
|
"top_k": 3
|
|
},
|
|
{
|
|
"prompt": "Inspect the network requests the dashboard makes when it first loads",
|
|
"top_k": 3
|
|
},
|
|
{
|
|
"prompt": "Verify in a real browser that the form submits and the data persists in localStorage",
|
|
"top_k": 3
|
|
}
|
|
],
|
|
"negative": [
|
|
{
|
|
"prompt": "Write a spec for the new billing feature before we start",
|
|
"owner": "spec-driven-development"
|
|
},
|
|
{
|
|
"prompt": "Break this PRD into ordered implementation tasks",
|
|
"owner": "planning-and-task-breakdown"
|
|
}
|
|
]
|
|
},
|
|
"evals": [
|
|
{
|
|
"id": 1,
|
|
"prompt": "The signup form renders but submitting it appears to do nothing. Verify the real behavior in the browser and report findings.",
|
|
"expected_output": "Runtime evidence from the browser: console errors, network activity, DOM state, and a diagnosis",
|
|
"expectations": [
|
|
"Findings are grounded in observed runtime data (console, network, DOM), not static code reading alone",
|
|
"The report distinguishes what was observed from what is inferred",
|
|
"A concrete next step or fix hypothesis is provided"
|
|
],
|
|
"trust_level": "provisional"
|
|
}
|
|
]
|
|
}
|