truthmark/workflow-eval-framwork

Workflow Eval Framwork

workflow-eval-framwork is Truthmark's contributor-only, manual, token-expensive behavioral eval framework for installed workflow skills and prompts.

It is not part of the published npm package and is not needed by normal Truthmark users.

The folder name is intentionally spelled framwork because that is the requested canonical path.

One-command manual run

Run the whole catalog with one explicit command when checking workflow quality:

node scripts/workflow-eval-framwork/run-agent-scenario.mjs \
  --catalog workflow-eval-framwork/catalog.yaml \
  --host codex \
  --agent-command "codex exec --full-auto" \
  --agent-model "gpt-5.5" \
  --judge-command "your-llm-judge-command" \
  --judge-model "gpt-5.5" \
  --require-usage

Catalog runs create a suite folder with exactly two persistent files:

• final-report.md — one comprehensive human-facing report for the full evaluation run.
• audit.json — compact machine-readable audit data with run metadata, scenario verdicts, deterministic failures, changed-file summaries, and judge summaries.

Per-scenario command output, patches, reports, and judge scratch files are temporary in catalog mode and are discarded after the two suite files are written. Scenario setup patches are still applied after the fixture baseline commit so agents evaluate a real pre-existing diff rather than an empty checkout.

Scenarios should model realistic development situations rather than one-file toy repos. Fixtures include source, tests, product truth, engineering behavior truth, API contract truth, operations truth, and route ownership so agents must choose the correct workflow lane and bounded owner.

A catalog run without --judge-command is intentionally not_evaluable. It can smoke-test fixture materialization and deterministic gates, but it is not a meaningful workflow-quality result. When --agent-model or --judge-model is supplied, the runner records those model names in audit.json, final-report.md, and the TRUTHMARK_EVAL_AGENT_MODEL / TRUTHMARK_EVAL_JUDGE_MODEL environment variables for wrapper scripts.

Token usage tracking

The runner exports usage sidecar paths for wrappers:

• TRUTHMARK_EVAL_AGENT_USAGE — JSON path for the agent command to write token usage.
• TRUTHMARK_EVAL_JUDGE_USAGE — JSON path for the judge command to write token usage.

Each sidecar may contain one usage object or an array of usage objects:

{
  "schemaVersion": 1,
  "source": "codex-jsonl-wrapper",
  "model": "gpt-5.5",
  "inputTokens": 1000,
  "cachedInputTokens": 200,
  "outputTokens": 300,
  "reasoningOutputTokens": 50,
  "totalTokens": 1300
}

Catalog audit.json, debug result.json, and final-report.md record measured agent tokens, judge tokens, and suite totals. The framework intentionally does not enforce token budgets or estimate dollar cost; token counts are a trend signal for comparing prompt and workflow-surface changes over time.

Missing sidecars are recorded as usage.status: "unavailable"; invalid sidecars are recorded as usage.status: "invalid" with warnings.

Use --require-usage for real model-comparison runs. In required-usage mode, missing or invalid usage data makes the run non-passing so usage regressions are not silently untracked.

Run one scenario only when debugging a specific failure, and mark it debug-only:

node scripts/workflow-eval-framwork/run-agent-scenario.mjs \
  --debug-scenario \
  --scenario workflow-eval-framwork/scenarios/truthmark-sync/docs-only-skip/scenario.yaml \
  --host fake \
  --agent-command "node workflow-eval-framwork/fake-agents/pass.mjs" \
  --agent-model "fake-agent" \
  --judge-command "node workflow-eval-framwork/fake-agents/judge-pass.mjs" \
  --judge-model "fake-judge"

By default, catalog artifacts are written under:

workflow-eval-framwork/runs/<timestamp>-all-workflows-<host>/

Single-scenario debug runs still write detailed per-scenario files under the selected debug output directory because they are for harness repair and failure investigation, not normal full-evaluation reporting.

Use failure-taxonomy.md and improvement-loop.md when converting a failed run into a prompt/skill fix or a smaller regression scenario.

Real model runners and LLM judges are manual. Normal truthmark check, package install, downstream repository use, and default CI must not launch real agents or judges.

Deterministic gates catch objective failures first, but workflow quality claims require LLM judge output plus human review. Without semantic judge results, a run proves only harness mechanics and write-boundary behavior, and the runner records it as not_evaluable.