The advisory contract rule was not enough: flash-class agents defer
hypothesis/coverage/PTT writes to closeout, then reconstruct results from
conversation memory after compression — the false-positive factory.
check_hypothesis_freshness now hard-blocks further target commands when the
newest execution evidence is newer than the hypothesis board's last update
(15 min grace for burst timing). The agent MUST record the batch result via
violin_record_hypothesis before proceeding. Bursts preflight all commands
before any executes, so no mid-burst false blocks.
Also fixes a silent test-suite gap: tests/__init__.py and the four
subpackage __init__.py files were missing, so 171 integration tests
(incl. test_plugin_guard.py which pins freshness behavior) NEVER ran.
Suite went from 248 passed / 7 collection errors to 419 passed, 0 errors.
Agent was deferring framework feedback and state writes to closeout, then
reconstructing what happened from conversation memory after compression —
the exact mechanism that fabricates false positives.
- Guard now self-logs: _check_command_internal appends a Guard Block/Review
row to state/framework_feedback.md at the moment check_command rejects a
command (only when the file exists, i.e. benchmark engagements; no-op
otherwise). Friction is captured with zero agent bookkeeping.
- SKILL.md Operational Contract: 'Record as you go' — after EVERY
violin_review_batch, immediately update hypothesis board + coverage matrix
+ PTT in the same turn; never batch state writes to closeout, never
reconstruct tests from memory. State files are the only source of truth.
- 5 new tests: block-row append, no-op without file/errors, dedupe, pipe
escaping.
243 tests pass, ruff clean.
- _method_route_in_text now accepts '-X POST <url>' reproduction commands
(quoted URLs no longer break the match) — recovers referral-abuse
- coupon-100-discount: decisive patterns (INTERNAL100 / 'discounted to $0')
kill the false positive from a neg-qty finding whose checkout sent
coupon_code:null
- weak-admin-creds: decisive patterns (DefaultCredentials / admin:password)
stop mass-assign role-granted text from confirming it
- vuln-research playbook + hypothesis template: mandate 1:N Linked findings
(an unlinked FIND file scores 0 even with perfect evidence)
Calibration still 20/20 + 0/0. Latest run re-scored: 6/20, now all true
positives (false coupon + false creds removed, referral recovered).
- _engagement_needs_closeout no longer fires on leftover unchecked PTT rows:
a single '[ ]' task marker triggered a continuation pass that re-ran the
entire assessment (255 msgs / 20m50s) instead of closing out. Closeout is
now gated on the artifacts the scorer consumes (FIND files + report.md +
retrospective.md) only.
- Closeout goal rewritten to be surgical: assessment is COMPLETE, evidence is
final, no new tests/scans/verification, no evidence modification; only
generate missing report/retrospective (prefer generate-closeout) and update
PTT statuses.
- Friction-log instruction moved out of the general pentest SKILL.md (it is a
benchmark-only artifact) into the benchmark goal prompt so the agent
actually sees it.
- New pinned tests: leftover-PTT-row closeout detection, closeout-goal
wording.
* fix: bump distribution version to 3.0.1
* feat(benchmark): introduce benchmark runner engine, Docker containerization, and calibration datasets
* Fix URL/path scope exclusions and update duck-store template
- Modified targets.py to ensure that excluding a specific URL or path does not result in the entire host being blocked. Implemented a strict command payload check for excluded endpoints.
- Updated duck-store benchmark scope template to explicitly block the /vulnerabilities endpoint (which leaks intentional challenges) while allowing access to endpoint documentation.
* fix: correct scope.yaml indentation and refine guard blocking messages
* fix: resolve ModuleNotFoundError when score.py is executed directly
---------
Co-authored-by: Dan <dan@strategicautomation.local>
Violin