mirror of
https://github.com/merlinhu1/truthmark.git
synced 2026-08-25 07:53:25 +02:00
Merlin's CatGitHubCopilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>MerlinHCopilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>* feat: add workflow eval framework Add workflow evaluation scenarios, rubrics, schemas, and runner scripts for installed Truthmark workflows. Move research notes under docs/research and migrate tests from Vitest to node:test. * Potential fix for pull request finding 'CodeQL / Replacement of a substring with itself' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: MerlinH <merlinh221@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
1.2 KiB
1.2 KiB
Improvement Loop
When a workflow eval fails:
- For catalog runs, read
final-report.mdfirst and useaudit.jsonfor scenario verdicts, deterministic failures, changed-file summaries, and judge summaries. For single-scenario debug runs, read the detailedreport.md,result.json, deterministic results, command output, changed files, patches, and judge results. - Record a human review as
accepted,rejected,needs-rerun, ornot-evaluable. - Classify failures with labels from
failure-taxonomy.md. - Decide the smallest fix target:
- skill or prompt text when the agent misunderstood workflow behavior;
- rubric text when the evaluation expectation was underspecified;
- scenario fixture when the setup did not isolate the intended failure;
- deterministic validator when an objective violation was missed;
- judge prompt/schema when semantic scoring was malformed.
- Promote real failures into minimal scenarios instead of checking in large run artifacts.
- Rerun the scenario with one manual command and compare the new report folder.
LLM judge scores are advisory until calibrated by human review. Deterministic gates cannot be overridden by a judge score.