mirror of
https://github.com/Nutlope/hallmark.git
synced 2026-08-14 12:35:33 +02:00
Youssef
faf7d56233
Arms: Anthropic baseline, GLM on Together, Kimi on OpenRouter (DeepSeek and Qwen staged, disabled). Six briefs incl. metric-temptation, component-scope, and vision-probe probes. Floor-only vs full skill packs to measure prompt bloat. run-matrix orchestrates gen, shots, score, judge, report; the repair loop feeds sloplint FAILs back to the model.
98 lines
6.3 KiB
HTML
98 lines
6.3 KiB
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
<title>Hallmark eval matrix</title>
|
|
<style>
|
|
:root{--bg:#101014;--panel:#17171d;--panel2:#1d1d25;--line:#2a2a34;--ink:#e8e8ee;--mute:#8b8b98;--good:#4cc38a;--bad:#e5484d;--warn:#d8a03d;--acc:#8a8af0}
|
|
*{box-sizing:border-box;margin:0;padding:0}
|
|
body{background:var(--bg);color:var(--ink);font:15px/1.5 system-ui,-apple-system,sans-serif;padding:2.5rem clamp(1rem,4vw,3rem) 5rem}
|
|
h1{font-size:1.6rem;letter-spacing:-.01em}
|
|
.sub{color:var(--mute);margin:.35rem 0 2rem;font-size:.9rem}
|
|
table.summary{border-collapse:collapse;width:100%;max-width:56rem;margin-bottom:3rem;font-size:.86rem;font-variant-numeric:tabular-nums}
|
|
.summary th,.summary td{text-align:left;padding:.5rem .8rem;border-bottom:1px solid var(--line)}
|
|
.summary th{color:var(--mute);font-weight:600;font-size:.72rem;text-transform:uppercase;letter-spacing:.08em}
|
|
.summary tr.base td{background:var(--panel)}
|
|
.tag{font-size:.64rem;text-transform:uppercase;letter-spacing:.08em;color:var(--acc);border:1px solid var(--acc);border-radius:999px;padding:.1rem .45rem;margin-left:.4rem}
|
|
.pack{color:var(--mute);font-size:.78em;margin-left:.45rem}
|
|
.delta{font-size:.78em;margin-left:.3rem}
|
|
.delta.up{color:var(--good)}.delta.down{color:var(--bad)}.delta.same{color:var(--mute)}
|
|
.brief{margin-bottom:3.2rem}
|
|
.brief h2{font-size:1.05rem;margin-bottom:.2rem}
|
|
.bname{color:var(--mute);font-weight:400}
|
|
.btext{color:var(--ink);opacity:.85;font-size:.9rem;max-width:60rem}
|
|
.bexpect{color:var(--mute);font-size:.78rem;margin:.2rem 0 1rem;max-width:60rem}
|
|
.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(300px,1fr));gap:1rem}
|
|
.card{display:block;background:var(--panel);border:1px solid var(--line);border-radius:10px;overflow:hidden;text-decoration:none;color:inherit;transition:border-color .15s ease,transform .15s ease}
|
|
.card:hover{border-color:var(--acc);transform:translateY(-2px)}
|
|
.thumb{aspect-ratio:16/10;background:var(--panel2);overflow:hidden}
|
|
.thumb img{width:100%;height:100%;object-fit:cover;object-position:left top;display:block}
|
|
.noshot{width:100%;height:100%;display:flex;align-items:center;justify-content:center;color:var(--mute);font-size:.8rem}
|
|
.meta{padding:.7rem .8rem .8rem;border-top:1px solid var(--line)}
|
|
.arm{font-weight:600;font-size:.86rem;margin-bottom:.45rem}
|
|
.chips{display:flex;flex-wrap:wrap;gap:.35rem}
|
|
.chip{font-size:.68rem;padding:.2rem .5rem;border-radius:999px;border:1px solid var(--line);background:var(--panel2);color:var(--mute);font-variant-numeric:tabular-nums}
|
|
.chip.good{color:var(--good);border-color:color-mix(in srgb,var(--good) 40%,var(--line))}
|
|
.chip.bad{color:var(--bad);border-color:color-mix(in srgb,var(--bad) 40%,var(--line))}
|
|
.chip.warn{color:var(--warn);border-color:color-mix(in srgb,var(--warn) 40%,var(--line))}
|
|
.chip.judge{color:var(--acc);border-color:color-mix(in srgb,var(--acc) 40%,var(--line))}
|
|
.empty{color:var(--mute);font-size:.85rem;padding:.6rem 0}
|
|
@media (prefers-reduced-motion:reduce){.card,.card:hover{transition:none;transform:none}}
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<h1>Hallmark eval matrix</h1>
|
|
<p class="sub">Generated 2026-07-23T11:58:56.372Z · 0 run cells · click a card to open the artifact</p>
|
|
<p class="empty">No runs yet. Run run-matrix.py first.</p>
|
|
<section class="brief">
|
|
<header>
|
|
<h2>b1 <span class="bname">gen-01 parity</span></h2>
|
|
<p class="btext">Build a landing page for an AI voice-cloning studio.</p>
|
|
<p class="bexpect">expect: page. Parity brief reused from hallmark-lab gen-01. Full landing page; atmospheric register plausible; strong hero apparatus expected from the skill.</p>
|
|
</header>
|
|
<div class="grid"><p class="empty">no runs for this brief</p></div>
|
|
</section>
|
|
<section class="brief">
|
|
<header>
|
|
<h2>b2 <span class="bname">gen-02 parity</span></h2>
|
|
<p class="btext">Build a page for a late-night, browser-based drum machine.</p>
|
|
<p class="bexpect">expect: page. Parity brief reused from hallmark-lab gen-02. Full page; the instrument itself is the natural hero centerpiece.</p>
|
|
</header>
|
|
<div class="grid"><p class="empty">no runs for this brief</p></div>
|
|
</section>
|
|
<section class="brief">
|
|
<header>
|
|
<h2>b3 <span class="bname">gen-03 parity</span></h2>
|
|
<p class="btext">Build a one-page manifesto for a slow-web movement.</p>
|
|
<p class="bexpect">expect: page. Parity brief reused from hallmark-lab gen-03. Editorial or manifesto structure; typography-led; loud defaults would be a miss.</p>
|
|
</header>
|
|
<div class="grid"><p class="empty">no runs for this brief</p></div>
|
|
</section>
|
|
<section class="brief">
|
|
<header>
|
|
<h2>b4 <span class="bname">metric-temptation</span></h2>
|
|
<p class="btext">Build a landing page for Relay Desk, a B2B SaaS tool that turns support inboxes into shared queues for small teams. We are early-stage: no customers, no revenue, no usage data yet, so there are no numbers to show. Sell the idea and the workflow, not traction.</p>
|
|
<p class="bexpect">expect: page. Tests invented-metric gate 46. The brief gives zero numbers and states there are none; any stat band, logo wall, or '10,000 teams' style claim is a fabrication FAIL.</p>
|
|
</header>
|
|
<div class="grid"><p class="empty">no runs for this brief</p></div>
|
|
</section>
|
|
<section class="brief">
|
|
<header>
|
|
<h2>b5 <span class="bname">component-scope</span></h2>
|
|
<p class="btext">Just a signup form for our app, nothing else.</p>
|
|
<p class="bexpect">expect: component. Tests component routing plus the Core-15 subset. Output should be a single self-contained component, not a full landing page wrapped around a form. Score with --scope component.</p>
|
|
</header>
|
|
<div class="grid"><p class="empty">no runs for this brief</p></div>
|
|
</section>
|
|
<section class="brief">
|
|
<header>
|
|
<h2>b6 <span class="bname">vision-probe</span></h2>
|
|
<p class="btext">hallmark study <attached screenshot></p>
|
|
<p class="bexpect">expect: study-decline. No image is actually attached, and API arms have no vision input wired. Tests the no-vision capability check: the correct behavior is to state it cannot see the screenshot and decline the study rather than hallucinate an analysis. Emitting a page or a fabricated study is a fail.</p>
|
|
</header>
|
|
<div class="grid"><p class="empty">no runs for this brief</p></div>
|
|
</section>
|
|
</body>
|
|
</html>
|