The runs archive copied a fixed filename list, index.html, tokens.css,
styles.css, and stampPresent scans that archive. A swept build whose stamp
lives in page.css scored skillLoaded=false with a correct stamp on disk,
which is the same stale assumption already fixed in stampPresent and in the
reference budget, one function lower. The archive now takes every top-level
html and css the build produced.
The three affected run.json files are rescored from the real artifacts:
only the buggy cell changed (inspo false to true); the bare control stays
false, which is the leak check holding. Also archives the second three-arm
run: bare 56 FAIL with the same six numbered eyebrows, both Hallmark arms
clean, and the control now provably offered zero MCP tools.
b1 names no theme, so it must derive, and it did: direction.md was the first
reference read, the stamp carries a real seed and a five-line direction
contract, and the derived world (the IPA vowel quadrilateral, plotted, on a
speech-lab plate) reaches the actual hero rather than stopping at the stamp.
Twelve reference reads, zero waste, in the right order. R.1's rejection is
legible in the genre note: it refused the radial blooms the atmospheric genre
would have allowed, because the reflex check had already spent that move.
b7 passes --fast and must not derive, and did not. It read two theme files and
never opened direction.md. No seed, no contract, no slate, no draw line. The
posture line still emits, which is the color.md repair working: posture is
universal now, not a derived-run field.
Both artifacts lint clean at 0 FAIL, 0 REFLEX, 0 WARN.
Three harness bugs, all of them the flip's doing. stampPresent scanned a fixed
filename list of index.html, styles.css and tokens.css, and a build that named
its stylesheet page.css scored skillLoaded=false with a correct stamp sitting in
it; its regex also demanded Hallmark immediately after the comment open, so
"/* Formant Studio · Hallmark derived system" missed too. It now scans every
emitted css and html and matches the marker anywhere in a leading comment.
The reference-read budget was ten, set when a run read one theme file. A derived
run reads direction.md and theme-axes.md in its place, so the floor moved up by
one, and a clean derived build measures twelve. Fourteen leaves room for two
conditionals and still catches the thing the check is for, which is defensive
pre-loading of a forty-file tree.
The cost cap was three dollars, hardcoded. A derived build on Opus lands near
three and stopped mid Step-7, which scored a complete artifact as
error_max_budget_usd and never exercised the gate sweep at all. It is --budget-usd
now, defaulting to five.
b7 also joins the brief set: the suite had no coverage of the catalog fast path
after that path stopped being the default.
Running the harness from inside a Claude Code session leaks
ANTHROPIC_BASE_URL and the CLAUDE_CODE_* host vars into the spawned
claude, which then reports not-logged-in even in an authenticated
terminal. The child env now drops them alongside the API key. Also
removes the six dead cells that run recorded: headless subscription auth
needs a terminal login this environment cannot perform, so the A/B
results were produced with in-session subagents instead (disclosed in
the PR).
gen-cli.mjs learns --model and --effort (forwarded to the claude call),
a bare no-skill control arm (plain brief prompt, no skill copy), a $3
per-cell budget guard, and --setting-sources project on BOTH arms so the
user-scope hallmark install can neither leak into the control nor
version-skew the skill arm; stampPresent on a bare cell doubles as the
leak detector. The result event's usage lands in run.json (tokens
in/out/cacheRead, turns, stop subtype) with the camelCase modelUsage
fallback, and cells are named arm-effort-cli so effort variants coexist.
Runs 'claude -p "/hallmark <brief>"' with the skill copied into a scratch
project, parses the stream-json transcript for conformance (skill loaded via
stamp, files written, reference-read count + load-order, cost), writes into
runs/<brief>/<arm>-cli/. Skills do not auto-trigger headless (measured), so
it invokes by name: a conformance harness, not an auto-trigger test. Claude
arm uses the subscription (no key); GLM/Kimi arms Anthropic-shaped and
disabled until their auth tokens exist. Plumbing validated against the real
binary here (spawn, stream-json parse both event shapes, run.json); a real
run needs a logged-in terminal.
Youssef