External/caveman
1
0
Fork 0
mirror of https://github.com/JuliusBrussee/caveman.git synced 2026-08-11 13:21:09 +02:00
Code Issues Packages Projects Releases Wiki Activity
177 Commits 12 Branches 17 Tags
v1.8.0
Commit Graph
9 Commits
Author SHA1 Message Date
Julius BrusseeandClaude Opus 4.6 a97af68741 fix: correct inverted sign in fmt_pct eval output 
Positive savings (skill reduces tokens) should display as "+" not "−".
The sign logic was backwards, making all eval results misleading.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-09 01:05:25 +02:00
Julius BrusseeandClaude Opus 4.6 f6ca007f49 Remove generated eval snapshot files from git tracking 
These are build artifacts (HTML report and PNG plot) that should not be
version-controlled. Added evals/snapshots/*.html and *.png patterns to
.gitignore.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-09 01:00:06 +02:00
sebastianbreguel 64c4e2113f Switch eval plot to real boxplot with per-prompt distribution 2026-04-08 18:44:27 -04:00
sebastianbreguel 6588ad1e26 Flip eval plot to vertical orientation 2026-04-08 18:36:03 -04:00
sebastianbreguel 784df686b4 Switch eval plot to dot+whiskers (median, IQR, min/max) 2026-04-08 18:32:06 -04:00
sebastianbreguel 7de87db9e6 Reframe eval plot as 'tokens saved' stacked bars 2026-04-08 18:28:58 -04:00
sebastianbreguel 1254e714e6 Switch eval plot to plotly, more intuitive horizontal layout 2026-04-08 18:26:08 -04:00
sebastianbreguel 6391a702b7 Add bar chart of skill compression with median/mean/range 2026-04-08 18:24:28 -04:00
sebastianbreguel 2b10489888 Add token-compression eval harness with three-arm methodology 
Closes #18 by replacing the unverified "~75% savings" claim with a real,
auditable measurement.

Methodology:
- Three arms per prompt: baseline (no system prompt), terse control
  ("Answer concisely."), and terse+SKILL.md. The honest delta is
  skill vs terse — this isolates the skill's contribution from the
  generic "be terse" effect.
- Real LLM in the loop via `claude -p --system-prompt`. No hand-written
  baselines, no circularity.
- Snapshot of LLM outputs committed to git as the source of truth.
  measure.py runs in CI with no network and no auth.
- Reports median, mean, min, max, stdev across prompts so noise is visible.
- Metadata pinned in the snapshot: model, CLI version, generation timestamp.

Initial run on claude-opus-4-6 (n=10 prompts) shows the real numbers
sit in the −22% to −49% mean range, not the previously claimed ~75%.
The README's headline number should be updated to match.
 2026-04-08 18:20:09 -04:00