5 Commits
Author SHA1 Message Date
Violin e0b3cea379 fix(benchmark): curl-form endpoint matching + decisive discriminators
- _method_route_in_text now accepts '-X POST <url>' reproduction commands
  (quoted URLs no longer break the match) — recovers referral-abuse
- coupon-100-discount: decisive patterns (INTERNAL100 / 'discounted to $0')
  kill the false positive from a neg-qty finding whose checkout sent
  coupon_code:null
- weak-admin-creds: decisive patterns (DefaultCredentials / admin:password)
  stop mass-assign role-granted text from confirming it
- vuln-research playbook + hypothesis template: mandate 1:N Linked findings
  (an unlinked FIND file scores 0 even with perfect evidence)

Calibration still 20/20 + 0/0. Latest run re-scored: 6/20, now all true
positives (false coupon + false creds removed, referral recovered).
2026-08-11 13:03:47 +01:00
Violin 3807d22eaa fix(benchmark): evidence-gated scorer contract, false-positive discriminators, runner validity
- Confirm Validated hypotheses via canonical FIND-NNN.md matched in live-app
  terms (METHOD + endpoint route) alongside execution bundles
- require/require_any discriminators for shared-endpoint challenges;
  absence-type scoring for no-rate-limiting
- Runner: closeout timeout is a soft warning, not run invalidation
- ai_judge: whitelist legitimate closeout artifacts in schema-drift audit
- Align version surfaces to 3.1.0; move pyyaml to runtime deps

Calibration: known-good 20/20, known-bad 0/0. Latest run re-scored 4 -> 9.
2026-08-10 20:52:02 +01:00
Violin 48119a64cf feat(benchmark): implement proof verifier, AI judge refinements, and duck-store calibration evidence 2026-08-10 08:54:35 +01:00
3b21e0a92a feat: Automated Benchmark Runner and Evaluation Framework (#68)
* fix: bump distribution version to 3.0.1

* feat(benchmark): introduce benchmark runner engine, Docker containerization, and calibration datasets

* Fix URL/path scope exclusions and update duck-store template

- Modified targets.py to ensure that excluding a specific URL or path does not result in the entire host being blocked. Implemented a strict command payload check for excluded endpoints.

- Updated duck-store benchmark scope template to explicitly block the /vulnerabilities endpoint (which leaks intentional challenges) while allowing access to endpoint documentation.

* fix: correct scope.yaml indentation and refine guard blocking messages

* fix: resolve ModuleNotFoundError when score.py is executed directly

---------

Co-authored-by: Dan <dan@strategicautomation.local>
2026-08-08 13:29:11 +01:00
Violin 397ba907d7 feat: add Duck Store benchmark harness
4-file suite evaluating Violin against escape.tech's Duck Store:
- benchmark/score.py — evidence-gated scorer (8 fixes from first run)
- benchmark/targets/duck-store/challenges.json — 20 article-parity vulns
- benchmark/targets/duck-store/scope.yaml — pre-baked scope
- benchmark/targets/duck-store/engage.md + report.md — prompts

Also adds explicit model section to config.yaml.
2026-07-19 16:55:00 +01:00