Run 20260813_101504 burned ~15 min and 19 close attempts on the
undispositioned-coverage gate: the error listed failures without naming
the fix, so the agent flailed (even reading guard source) and wrote
invalid reasons ('evidence: none else'). The error now spells out the
exact remediation per cell status (tested/not_applicable/blocked/missing).
VIOLIN_BENCHMARK_RECEIPT_KEY -> VIOLIN_RECEIPT_KEY. The all-caps
'BENCHMARK' slipped past the original case-sensitive de-cheat grep in
plugins/violin_guard. No consumers hardcode the literal; all use the
RECEIPT_KEY_ENV constant.
Run 20260812_204014 died at 27m with HTTP 402 prompt-token limit
(141557 > 99922): the agent executed 328 tool calls without ever
compacting, cutting exploitation short (12/20, 8 challenges never
probed). The heartbeat now reminds the agent to compact when the run
has executed many commands.
- check_command now accepts hypothesis_id (schema + handler + args)
- Exploit-phase research gate checks only the named hypothesis when
provided; otherwise all candidates (previous behavior)
- Error message names the remediation tool: violin_record_hypothesis
id=H-00N cve_research=... exploit_research=...
- Fixes silent bypass: normalized id comparison (002 vs 2) so an
unresearched hypothesis is actually blocked
- New test: named-hypothesis passes, unresearched H-002 blocked,
all-candidates mode still enforces full rows
- Remove challenge_ids seeding (vulnerability-name leak) from scope;
seed engagement.coverage_obligations (client-style in-scope endpoints)
+ generic engagement.audit_mode flag instead
- plugins/violin_guard now contains zero benchmark/run-id references;
gates are framework-owned and audit-mode-gated
- Cross-engagement guard blocks ANY foreign engagement dir, not just
benchmark-run-*
- Anti-cheat regression test asserts no vuln names in generated scope
The advisory contract rule was not enough: flash-class agents defer
hypothesis/coverage/PTT writes to closeout, then reconstruct results from
conversation memory after compression — the false-positive factory.
check_hypothesis_freshness now hard-blocks further target commands when the
newest execution evidence is newer than the hypothesis board's last update
(15 min grace for burst timing). The agent MUST record the batch result via
violin_record_hypothesis before proceeding. Bursts preflight all commands
before any executes, so no mid-burst false blocks.
Also fixes a silent test-suite gap: tests/__init__.py and the four
subpackage __init__.py files were missing, so 171 integration tests
(incl. test_plugin_guard.py which pins freshness behavior) NEVER ran.
Suite went from 248 passed / 7 collection errors to 419 passed, 0 errors.
Agent was deferring framework feedback and state writes to closeout, then
reconstructing what happened from conversation memory after compression —
the exact mechanism that fabricates false positives.
- Guard now self-logs: _check_command_internal appends a Guard Block/Review
row to state/framework_feedback.md at the moment check_command rejects a
command (only when the file exists, i.e. benchmark engagements; no-op
otherwise). Friction is captured with zero agent bookkeeping.
- SKILL.md Operational Contract: 'Record as you go' — after EVERY
violin_review_batch, immediately update hypothesis board + coverage matrix
+ PTT in the same turn; never batch state writes to closeout, never
reconstruct tests from memory. State files are the only source of truth.
- 5 new tests: block-row append, no-op without file/errors, dedupe, pipe
escaping.
243 tests pass, ruff clean.
* fix: bump distribution version to 3.0.1
* feat(benchmark): introduce benchmark runner engine, Docker containerization, and calibration datasets
* Fix URL/path scope exclusions and update duck-store template
- Modified targets.py to ensure that excluding a specific URL or path does not result in the entire host being blocked. Implemented a strict command payload check for excluded endpoints.
- Updated duck-store benchmark scope template to explicitly block the /vulnerabilities endpoint (which leaks intentional challenges) while allowing access to endpoint documentation.
* fix: correct scope.yaml indentation and refine guard blocking messages
* fix: resolve ModuleNotFoundError when score.py is executed directly
---------
Co-authored-by: Dan <dan@strategicautomation.local>
Violin