External/growmos
1
0
Fork 0
mirror of https://github.com/codician-team/growmos.git synced 2026-08-18 06:47:17 +02:00
Code Issues Packages Projects Releases Wiki Activity

Close the loop with no manual step: gold-set and review packets

Browse Source 
growmos next now also hands out gold packets (agent writes the eval reference
answer, provenance recorded) and periodic review packets (verify one node's
edges against sources); eval auto-extends the scorer alias map; headless ingest
does the same. Doctor shows who reviewed. Repo URL → codician-team/growmos.
This commit is contained in:
codician-dev
2026-08-17 14:58:47 +03:00
parent 988358ad1a
commit 39f38fb4d1
17 changed files with 348 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evaluation.md
+19 -10
View File
@@ -3,15 +3,21 @@
> Change the extraction prompt → rerun the scorer → watch F1 move. A team that ships without
> this loop cannot tell whether a prompt change improved or degraded quality.
## 1. Build a gold set
## 1. Build a gold set (automatic by default)
Pick at least two representative documents. Pre-fill from the current extraction, then
**hand-correct** (remove wrong entities, add missed ones — the whole point is human judgment):
`growmos next` hands out a **gold packet** for the richest extracted documents until `gold_min`
(default 2) files exist: the agent reads the document and writes the reference answer, and the file
records `_reviewed_by: agent`. Nothing manual is required. If you want a human pass, pre-fill and
edit — the file then records your name:
```bash
growmos gold-template docs/apollo-11.md
growmos gold-template docs/apollo-11.md # pre-fill from current extraction
$EDITOR .growmos/eval/gold/apollo-11.json
# or: growmos apply gold my.json --source docs/apollo-11.md --reviewer human
```
Caveat worth knowing: an agent-written gold set shares the blind spots of the agent that wrote it;
it still catches regressions (prompt drift, resolver over-merging) reliably, which is what the loop
is for.
Gold format:
```json
@@ -23,11 +29,12 @@ Relations are scored on (source, target) pairs, direction-agnostic, ignoring pre
an upper bound on relation recall that catches structural errors (missing/wrong connections),
which matter more than wording.
## 2. Keep the scorer alias map current
## 2. Scorer alias map (automatic)
If the resolver picks a canonical form the gold set doesn't use ("Neil Alden Armstrong" vs
"Neil Armstrong"), resolved recall drops — a scoring artifact, not a resolver bug.
`growmos eval` lists such canonicals; add them to `.growmos/eval/aliases.json`:
"Neil Armstrong"), resolved recall would drop — a scoring artifact, not a resolver bug.
`growmos eval` detects these (the raw mention matched a gold name) and **extends
`.growmos/eval/aliases.json` itself**, then rescores. You can also add entries by hand:
```json
{"Neil Alden Armstrong": "Neil Armstrong", "John F. Kennedy Space Center": "Kennedy Space Center"}
```
@@ -60,7 +67,9 @@ the change alters what the graph means, so rows produced under different prompts
`pending_sources`, `provisional`, `stale_profiles`. Extraction rate per document is in each
source's `stats`. Wire them into CI (`growmos integrate ci`).
## 6. The human sample
## 6. The periodic review (automatic)
`growmos sample` prints a random (degree-weighted) node with its profile, edges, provenance and
sources. Read it against the sources. `growmos doctor` turns red if nobody has done this in 7 days.
Every `review_days` (default 7) `growmos next` hands out a **review packet**: one random
(degree-weighted) node with its edges, provenance and source excerpts; the agent verifies each edge,
fixes what's wrong, and reports ok/issues (recorded in the journal). `growmos sample` remains
available for a human pass; `growmos doctor` shows who reviewed last and when.