mirror of
https://github.com/merlinhu1/truthmark.git
synced 2026-08-25 07:53:25 +02:00
Merlin's CatGitHubCopilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>MerlinHCopilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>* feat: add workflow eval framework Add workflow evaluation scenarios, rubrics, schemas, and runner scripts for installed Truthmark workflows. Move research notes under docs/research and migrate tests from Vitest to node:test. * Potential fix for pull request finding 'CodeQL / Replacement of a substring with itself' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: MerlinH <merlinh221@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
1.4 KiB
1.4 KiB
status, doc_type, last_reviewed, source_of_truth
| status | doc_type | last_reviewed | source_of_truth | ||||
|---|---|---|---|---|---|---|---|
| draft | research-index | 2026-06-29 |
|
Manual Agent Skill And Prompt Quality Eval Framework
The implemented manual evaluation framework lives under workflow-eval-framwork/.
Use these artifacts:
- Framework README:
../../workflow-eval-framwork/README.md - Catalog:
../../workflow-eval-framwork/catalog.yaml - Manual runner:
../../scripts/workflow-eval-framwork/run-agent-scenario.mjs - Schema and fake-runner tests:
../../tests/evals/workflow-eval-framwork.test.ts
The requested framework folder name is intentionally spelled workflow-eval-framwork.
The key invariant is that Truthmark's generated-surface tests prove injection and freshness, while this framework tests actual agent behavior when workflow skills and prompts are used.
The framework is manual and token-expensive. It must not become a default CI gate, hosted service, daemon, database, hidden memory layer, or required downstream runtime.