VIOLIN_BENCHMARK_RECEIPT_KEY -> VIOLIN_RECEIPT_KEY. The all-caps
'BENCHMARK' slipped past the original case-sensitive de-cheat grep in
plugins/violin_guard. No consumers hardcode the literal; all use the
RECEIPT_KEY_ENV constant.
Run 184313's agent canonized findings but never wrote the forward
'Linked findings' board field, so the scorer lost the confirmations.
The closeout goal now permits bookkeeping-only updates: link each
Validated hypothesis to its FIND-*.md via violin_record_hypothesis.
No new testing, findings, or evidence re-verification allowed.
Run 20260812_184313 scored 0/20 formalized despite 7 FIND files and 8
Validated hypotheses: the agent canonized findings carrying 'Linked
Hypothesis: H-00N' but never wrote the forward 'Linked findings' board
field (it closed PT-102 by editing state/ptt.md directly, bypassing the
record_ptt handler gates). The scorer now:
- parse_findings extracts reverse 'Linked Hypothesis' references
- validated_challenge_ids falls back to reverse links / shared evidence
when a Validated hypothesis has no forward links
- paper trail is unchanged: Validated hypothesis + substantive finding
over real evidence, in either direction
Re-scoring 184313: formalized 0% -> 30% (6 confirmed). Regression test
test_validated_hypothesis_confirms_via_find_reverse_link added.
- Remove challenge_ids seeding (vulnerability-name leak) from scope;
seed engagement.coverage_obligations (client-style in-scope endpoints)
+ generic engagement.audit_mode flag instead
- plugins/violin_guard now contains zero benchmark/run-id references;
gates are framework-owned and audit-mode-gated
- Cross-engagement guard blocks ANY foreign engagement dir, not just
benchmark-run-*
- Anti-cheat regression test asserts no vuln names in generated scope
- _method_route_in_text now accepts '-X POST <url>' reproduction commands
(quoted URLs no longer break the match) — recovers referral-abuse
- coupon-100-discount: decisive patterns (INTERNAL100 / 'discounted to $0')
kill the false positive from a neg-qty finding whose checkout sent
coupon_code:null
- weak-admin-creds: decisive patterns (DefaultCredentials / admin:password)
stop mass-assign role-granted text from confirming it
- vuln-research playbook + hypothesis template: mandate 1:N Linked findings
(an unlinked FIND file scores 0 even with perfect evidence)
Calibration still 20/20 + 0/0. Latest run re-scored: 6/20, now all true
positives (false coupon + false creds removed, referral recovered).
- _engagement_needs_closeout no longer fires on leftover unchecked PTT rows:
a single '[ ]' task marker triggered a continuation pass that re-ran the
entire assessment (255 msgs / 20m50s) instead of closing out. Closeout is
now gated on the artifacts the scorer consumes (FIND files + report.md +
retrospective.md) only.
- Closeout goal rewritten to be surgical: assessment is COMPLETE, evidence is
final, no new tests/scans/verification, no evidence modification; only
generate missing report/retrospective (prefer generate-closeout) and update
PTT statuses.
- Friction-log instruction moved out of the general pentest SKILL.md (it is a
benchmark-only artifact) into the benchmark goal prompt so the agent
actually sees it.
- New pinned tests: leftover-PTT-row closeout detection, closeout-goal
wording.
* fix: bump distribution version to 3.0.1
* feat(benchmark): introduce benchmark runner engine, Docker containerization, and calibration datasets
* Fix URL/path scope exclusions and update duck-store template
- Modified targets.py to ensure that excluding a specific URL or path does not result in the entire host being blocked. Implemented a strict command payload check for excluded endpoints.
- Updated duck-store benchmark scope template to explicitly block the /vulnerabilities endpoint (which leaks intentional challenges) while allowing access to endpoint documentation.
* fix: correct scope.yaml indentation and refine guard blocking messages
* fix: resolve ModuleNotFoundError when score.py is executed directly
---------
Co-authored-by: Dan <dan@strategicautomation.local>
Violin