Run 20260813_160102 scored 11/20 while run 152147 scored 16/20 on
identical code; the delta is canonization quality. FIND records that were
one-line pointers (no route, no specific field, no quoted proof) failed the
scorer's decisive-finding contract even when the vuln was real and proven
in evidence/. State the contract: substantive, exact METHOD/route, named
field, observable outcome, quoted proof, Impact+Remediation.
Run 20260813_152147 reached 16/20 but dropped referral-abuse and
no-rate-limiting to 'no decisive proof' — probes ran correctly but saved
only %{http_code} tails and bare status lines. Explicit rule: never save
only a status code; a confirmed vector's proof must include the response
body demonstrating it (credit balance for referral, no-429 summary for
rate-limit absence).
- redirects-unvalidated.md: systematic open-redirect probe loop (17 params,
Location-header capture, external-redirect follow, allowlist-substring bypass)
- auth-bypass.md: when to run default/weak creds (after user-enum finds admin;
hand-testable set is not brute force)
- xss.md: URL-valued fields (avatar/link) as stored/blind XSS vectors
- business-logic.md: referral self-referral registration recipe (register A,
refer B, check both sides for credit)
- exhaustive-coverage.md: mandatory auth/registration-flow coverage step so
the routing triggers no matter which playbook the agent reads first
No benchmark names or session inventory; generic OWASP technique only.
Runs 1-2 of the rebuilt-image set both forced guard-code reads on the
VULN_RESEARCH coverage gate: the flat-key schema and per-status proof
rules existed only in ptt_handlers.py. The reference now documents the
exact schema (flat top-level keys, tested/not_applicable/blocked with
evidence-path requirements, obligation matching), per the agent's own
prevention suggestion in framework_feedback.md.
Agent was deferring framework feedback and state writes to closeout, then
reconstructing what happened from conversation memory after compression —
the exact mechanism that fabricates false positives.
- Guard now self-logs: _check_command_internal appends a Guard Block/Review
row to state/framework_feedback.md at the moment check_command rejects a
command (only when the file exists, i.e. benchmark engagements; no-op
otherwise). Friction is captured with zero agent bookkeeping.
- SKILL.md Operational Contract: 'Record as you go' — after EVERY
violin_review_batch, immediately update hypothesis board + coverage matrix
+ PTT in the same turn; never batch state writes to closeout, never
reconstruct tests from memory. State files are the only source of truth.
- 5 new tests: block-row append, no-op without file/errors, dedupe, pipe
escaping.
243 tests pass, ruff clean.
- _method_route_in_text now accepts '-X POST <url>' reproduction commands
(quoted URLs no longer break the match) — recovers referral-abuse
- coupon-100-discount: decisive patterns (INTERNAL100 / 'discounted to $0')
kill the false positive from a neg-qty finding whose checkout sent
coupon_code:null
- weak-admin-creds: decisive patterns (DefaultCredentials / admin:password)
stop mass-assign role-granted text from confirming it
- vuln-research playbook + hypothesis template: mandate 1:N Linked findings
(an unlinked FIND file scores 0 even with perfect evidence)
Calibration still 20/20 + 0/0. Latest run re-scored: 6/20, now all true
positives (false coupon + false creds removed, referral recovered).
- _engagement_needs_closeout no longer fires on leftover unchecked PTT rows:
a single '[ ]' task marker triggered a continuation pass that re-ran the
entire assessment (255 msgs / 20m50s) instead of closing out. Closeout is
now gated on the artifacts the scorer consumes (FIND files + report.md +
retrospective.md) only.
- Closeout goal rewritten to be surgical: assessment is COMPLETE, evidence is
final, no new tests/scans/verification, no evidence modification; only
generate missing report/retrospective (prefer generate-closeout) and update
PTT statuses.
- Friction-log instruction moved out of the general pentest SKILL.md (it is a
benchmark-only artifact) into the benchmark goal prompt so the agent
actually sees it.
- New pinned tests: leftover-PTT-row closeout detection, closeout-goal
wording.
- Created a comprehensive playbook for SQL Injection (SQLi) detailing types, detection methods, tools, and remediation strategies.
- Added a playbook for Server-Side Request Forgery (SSRF) covering classification, detection payloads, and remediation guidance.
- Introduced a playbook for Cross-Site Scripting (XSS) including types, detection techniques, and safe proof of concept examples.
- Expanded the test suite with new tests for command history deduplication, scope authorization, and structured argument handling.
- Improved existing tests for engagement and execution tracking, ensuring robust coverage of functionality.
- Updated various playbooks to use `$ENG_DIR` for storing evidence files instead of hardcoded paths.
- Enhanced documentation in playbooks to include mandatory CVE and exploit research fields in hypotheses.
- Introduced a new `pty-safe-delivery.md` reference for safe file delivery practices over PTY.
- Added a `shell_ctrl.py` template for PTY shell control with safeguards against long commands and file transfers.
- Improved tests to validate new hypothesis fields and ensure compliance with updated playbook requirements.
- General cleanup and consistency improvements across playbooks and templates.
Violin