VIOLIN_BENCHMARK_RECEIPT_KEY -> VIOLIN_RECEIPT_KEY. The all-caps
'BENCHMARK' slipped past the original case-sensitive de-cheat grep in
plugins/violin_guard. No consumers hardcode the literal; all use the
RECEIPT_KEY_ENV constant.
Run 20260812_184313 scored 0/20 formalized despite 7 FIND files and 8
Validated hypotheses: the agent canonized findings carrying 'Linked
Hypothesis: H-00N' but never wrote the forward 'Linked findings' board
field (it closed PT-102 by editing state/ptt.md directly, bypassing the
record_ptt handler gates). The scorer now:
- parse_findings extracts reverse 'Linked Hypothesis' references
- validated_challenge_ids falls back to reverse links / shared evidence
when a Validated hypothesis has no forward links
- paper trail is unchanged: Validated hypothesis + substantive finding
over real evidence, in either direction
Re-scoring 184313: formalized 0% -> 30% (6 confirmed). Regression test
test_validated_hypothesis_confirms_via_find_reverse_link added.
- _method_route_in_text now accepts '-X POST <url>' reproduction commands
(quoted URLs no longer break the match) — recovers referral-abuse
- coupon-100-discount: decisive patterns (INTERNAL100 / 'discounted to $0')
kill the false positive from a neg-qty finding whose checkout sent
coupon_code:null
- weak-admin-creds: decisive patterns (DefaultCredentials / admin:password)
stop mass-assign role-granted text from confirming it
- vuln-research playbook + hypothesis template: mandate 1:N Linked findings
(an unlinked FIND file scores 0 even with perfect evidence)
Calibration still 20/20 + 0/0. Latest run re-scored: 6/20, now all true
positives (false coupon + false creds removed, referral recovered).
* fix: bump distribution version to 3.0.1
* feat(benchmark): introduce benchmark runner engine, Docker containerization, and calibration datasets
* Fix URL/path scope exclusions and update duck-store template
- Modified targets.py to ensure that excluding a specific URL or path does not result in the entire host being blocked. Implemented a strict command payload check for excluded endpoints.
- Updated duck-store benchmark scope template to explicitly block the /vulnerabilities endpoint (which leaks intentional challenges) while allowing access to endpoint documentation.
* fix: correct scope.yaml indentation and refine guard blocking messages
* fix: resolve ModuleNotFoundError when score.py is executed directly
---------
Co-authored-by: Dan <dan@strategicautomation.local>
Violin