mirror of
https://github.com/merlinhu1/codex-game-studio.git
synced 2026-08-25 07:54:34 +02:00
feat: add performance evaluation framework
This commit is contained in:
MerlinH
@@ -0,0 +1,81 @@
|
||||
# Skill And Prompt Performance Eval Framework
|
||||
|
||||
This is Open Game Studio's maintainer-only framework for evaluating how skills, workflow prompts, and agent-facing surfaces perform in realistic runs.
|
||||
|
||||
It follows the CCGS pattern of catalog → rubric → behavioral scenario, and the Truthmark pattern of manual workflow-quality runs with deterministic boundaries, semantic judging, human review, and token tracking.
|
||||
|
||||
Normal game-project users do not need this folder. It is not a hidden runtime, daemon, hosted service, or downstream requirement. If a downstream game repository only wants to build a game and not maintain Open Game Studio's prompt surfaces, users may delete `eval-framework/` and the related maintainer-only OpenSpec change files from their copy.
|
||||
|
||||
## What this evaluates
|
||||
|
||||
- Whether a skill or workflow is triggered for the right task.
|
||||
- Whether the agent selects bounded context instead of loading the whole repository.
|
||||
- Whether output quality matches the rubric and scenario contract.
|
||||
- Whether write boundaries are respected.
|
||||
- Whether verification is run or explicitly blocked.
|
||||
- Whether the final report gives a human reviewer a useful verdict, risks, changed files, and next owner.
|
||||
- Raw token usage and selected evaluation model for comparing prompt and workflow-surface changes over time.
|
||||
|
||||
## What this does not evaluate
|
||||
|
||||
- A pass is never awarded merely because a skill file is present.
|
||||
- A pass is never awarded merely because a prompt contains a literal phrase.
|
||||
- The framework is not a package-install gate for normal users.
|
||||
- The framework does not enforce token-budget thresholds by default.
|
||||
|
||||
## Files
|
||||
|
||||
```text
|
||||
eval-framework/
|
||||
├── catalog.json # targets, rubrics, scenarios, runner hosts
|
||||
├── rubrics/ # deterministic gates + semantic dimensions
|
||||
├── scenarios/ # realistic task prompts and expected boundaries
|
||||
├── failure-taxonomy.md # stable failure labels
|
||||
├── improvement-loop.md # how to turn failures into small fixes
|
||||
└── runs/ # repository-saved evaluation summaries and compact audit files
|
||||
```
|
||||
|
||||
## First-pass coverage
|
||||
|
||||
This pass covers 31 behavior scenarios:
|
||||
|
||||
| Area | Count | Examples |
|
||||
|---|---:|---|
|
||||
| Workflow prompts | 12 | `vertical-slice`, `bugfix`, `playtest`, `ship-check`, `sprint-plan` |
|
||||
| Skills | 12 | `cgs-gate-check`, `cgs-skill-test`, `cgs-skill-improve`, `cgs-code-review` |
|
||||
| Role prompts | 7 | `producer`, `qa-playtester`, `gameplay-programmer`, `release-manager` |
|
||||
|
||||
This is not CCGS full parity. It is the first practical coverage threshold for the highest-risk workflow, skill-maintenance, QA/review/gate, and role-cluster surfaces.
|
||||
|
||||
## Evaluation model and token estimation
|
||||
|
||||
- Default manual evaluation model: `gpt-5.3-codex-spark`.
|
||||
- Allowed models are listed in `catalog.json` under `modelPolicy.allowedEvaluationModels`.
|
||||
- A manual runner may override the model per scenario or batch with the Codex `--model` value, but the selected model must be recorded in the run report and audit JSON.
|
||||
- Token estimation is part of normal evaluation planning: estimate token usage before each staged run from the selected model, scenario count, expected context, and expected output/reasoning size.
|
||||
- Every run should record raw token usage when the host exposes it: input, cached input, output, reasoning output, and total tokens.
|
||||
- The framework does not estimate or record money cost.
|
||||
|
||||
## Gradual evaluation plan
|
||||
|
||||
You do not need to evaluate every skill and prompt at once. Use the staged plan in `catalog.json`:
|
||||
|
||||
1. `smoke-critical` — run only the smallest critical workflow/skill/role set first.
|
||||
2. `workflow-high-risk` — expand to high-priority workflow prompts.
|
||||
3. `skill-maintenance` — evaluate skill-maintenance, gate, QA, review, and evidence skills.
|
||||
4. `role-boundary` — evaluate role ownership and handoff boundaries.
|
||||
5. `full-regression` — run all scenarios only before broad prompt-surface releases or after large refactors.
|
||||
|
||||
## Manual run shape
|
||||
|
||||
A future runner should follow this contract:
|
||||
|
||||
1. Materialize the scenario fixture or use the current repository when the scenario says so.
|
||||
2. Run one agent attempt with the scenario prompt.
|
||||
3. Collect trace evidence, changed files, final report, verification output, and raw token usage.
|
||||
4. Apply deterministic gates first.
|
||||
5. Run semantic judges only after deterministic boundary checks are available.
|
||||
6. Write a repository-saved run directory under `eval-framework/runs/<timestamp>-<stage-or-scenario>/`.
|
||||
7. Save `summary.md` for human review and `audit.json` for machine-readable scenario results, selected model, raw token usage, trace references, failure labels, changed files, verification evidence, and recommended follow-up fixes.
|
||||
|
||||
Evaluation results are repository artifacts when intentionally preserved. Do not rely on chat transcript as the record; commit the selected `eval-framework/runs/...` summary and audit files when an evaluation result should survive.
|
||||
@@ -0,0 +1,572 @@
|
||||
{
|
||||
"version": 1,
|
||||
"manualOnly": true,
|
||||
"lastReviewed": "2026-07-01",
|
||||
"targets": [
|
||||
{
|
||||
"id": "workflow.vertical-slice",
|
||||
"kind": "workflow",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/vertical-slice.md",
|
||||
".agents/skills/cgs-vertical-slice/SKILL.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-vertical-slice/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.bugfix",
|
||||
"kind": "workflow",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/bugfix.md",
|
||||
".agents/skills/cgs-bugfix/SKILL.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-bugfix/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.playtest",
|
||||
"kind": "workflow",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/playtest.md",
|
||||
".agents/skills/cgs-playtest-report/SKILL.md",
|
||||
"templates/playtest_report_template.md",
|
||||
"templates/test_evidence_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-playtest/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.market-analysis",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/market-analysis.md",
|
||||
".agents/skills/cgs-content-audit/SKILL.md",
|
||||
"templates/market_analysis_template.md",
|
||||
"templates/pitch_document_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-market-analysis/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.release-checklist",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/release-checklist.md",
|
||||
".agents/skills/cgs-release-checklist/SKILL.md",
|
||||
"templates/release_notes_template.md",
|
||||
"templates/risk_register_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-release-checklist/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.ship-check",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/ship-check.md",
|
||||
"templates/ship_check_template.md",
|
||||
"templates/risk_register_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-ship-check/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.prototype",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/prototype.md",
|
||||
".agents/skills/cgs-prototype/SKILL.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-prototype/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.design-spec",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/design-spec.md",
|
||||
".agents/skills/cgs-design-system/SKILL.md",
|
||||
"templates/gdd_template.md",
|
||||
"templates/feature_spec_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-design-spec/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.game-feel-tuning",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/game-feel-tuning.md",
|
||||
".agents/skills/cgs-balance-check/SKILL.md",
|
||||
"templates/game_feel_tuning_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-game-feel-tuning/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.ui-ux-review",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/ui-ux-review.md",
|
||||
".agents/skills/cgs-ui-ux-review/SKILL.md",
|
||||
"templates/ui_ux_review_template.md",
|
||||
"templates/accessibility_requirements_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-ui-ux-review/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.architecture-review",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/architecture-review.md",
|
||||
".agents/skills/cgs-architecture-review/SKILL.md",
|
||||
"templates/technical_design_template.md",
|
||||
"templates/architecture_traceability_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-architecture-review/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "workflow.sprint-plan",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/workflows/sprint-plan.md",
|
||||
".agents/skills/cgs-sprint-plan/SKILL.md",
|
||||
"templates/sprint_plan_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/workflow-sprint-plan/behavior/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-skill-test",
|
||||
"kind": "skill",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-skill-test/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-skill-test/behavioral-spec/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-skill-improve",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-skill-improve/SKILL.md",
|
||||
"eval-framework/failure-taxonomy.md",
|
||||
"eval-framework/improvement-loop.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-skill-improve/failure-loop/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-gate-check",
|
||||
"kind": "skill",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-gate-check/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-gate-check/mode-boundary/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-design-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-design-review/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-design-review/read-only-verdict/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-architecture-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-architecture-review/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-architecture-review/risk-verdict/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-story-readiness",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-story-readiness/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-story-readiness/readiness-verdict/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-story-done",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-story-done/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-story-done/done-verdict/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-qa-plan",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-qa-plan/SKILL.md",
|
||||
"templates/test_plan_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-qa-plan/coverage-plan/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-regression-suite",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-regression-suite/SKILL.md",
|
||||
"templates/test_evidence_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-regression-suite/repeatability/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-test-evidence-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-test-evidence-review/SKILL.md",
|
||||
"templates/test_evidence_template.md"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-test-evidence-review/evidence-review/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-test-flakiness",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-test-flakiness/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-test-flakiness/flakiness-diagnosis/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cgs-code-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".agents/skills/cgs-code-review/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/skill-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/cgs-code-review/review-findings/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "role.producer",
|
||||
"kind": "role",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/agents/producer.toml"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/role-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/role-producer/domain-boundary/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "role.qa-playtester",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/agents/qa-playtester.toml"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/role-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/role-qa-playtester/domain-boundary/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "role.gameplay-programmer",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/agents/gameplay-programmer.toml"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/role-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/role-gameplay-programmer/domain-boundary/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "role.game-designer",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/agents/game-designer.toml"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/role-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/role-game-designer/domain-boundary/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "role.market-analyst",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/agents/market-analyst.toml"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/role-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/role-market-analyst/domain-boundary/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "role.technical-artist",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/agents/technical-artist.toml"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/role-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/role-technical-artist/domain-boundary/scenario.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "role.release-manager",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"surfacePaths": [
|
||||
".codex/agents/release-manager.toml"
|
||||
],
|
||||
"rubric": "eval-framework/rubrics/role-behavior.json",
|
||||
"scenarios": [
|
||||
"eval-framework/scenarios/role-release-manager/domain-boundary/scenario.json"
|
||||
]
|
||||
}
|
||||
],
|
||||
"runners": {
|
||||
"harnessHosts": [
|
||||
"fake"
|
||||
],
|
||||
"manualAgentHosts": [
|
||||
"codex"
|
||||
],
|
||||
"runOutputPolicy": {
|
||||
"repositoryTracked": true,
|
||||
"root": "eval-framework/runs",
|
||||
"perRunDirectory": "<timestamp>-<stage-or-scenario>",
|
||||
"requiredFiles": [
|
||||
"summary.md",
|
||||
"audit.json"
|
||||
],
|
||||
"summaryContract": [
|
||||
"run id and evaluation stage",
|
||||
"selected model and token usage summary",
|
||||
"scenario verdict table",
|
||||
"deterministic failures and semantic findings",
|
||||
"changed files and verification evidence",
|
||||
"recommended follow-up fixes"
|
||||
],
|
||||
"auditContract": [
|
||||
"machine-readable scenario results",
|
||||
"raw token usage fields",
|
||||
"model used per scenario",
|
||||
"trace evidence references",
|
||||
"failure taxonomy labels"
|
||||
],
|
||||
"notes": "Evaluation results are repository artifacts. Commit intentionally selected run summaries and compact audit files when preserving an eval result; do not rely on chat transcript as the record."
|
||||
}
|
||||
},
|
||||
"modelPolicy": {
|
||||
"defaultEvaluationModel": "gpt-5.3-codex-spark",
|
||||
"allowedEvaluationModels": [
|
||||
"gpt-5.3-codex-spark",
|
||||
"gpt-5.5",
|
||||
"gpt-5.4",
|
||||
"gpt-5.4-mini"
|
||||
],
|
||||
"overrideMechanism": "Manual runners may pass an explicit Codex --model value per scenario or batch; record the selected model in the run report and audit JSON.",
|
||||
"tokenEstimation": {
|
||||
"required": true,
|
||||
"fields": [
|
||||
"inputTokens",
|
||||
"cachedInputTokens",
|
||||
"outputTokens",
|
||||
"reasoningOutputTokens",
|
||||
"totalTokens"
|
||||
],
|
||||
"notes": "Estimate token usage as part of each staged evaluation run using the selected model, scenario count, expected context, and expected output size; record actual token usage when the host exposes it."
|
||||
}
|
||||
},
|
||||
"evaluationPlan": [
|
||||
{
|
||||
"stage": "smoke-critical",
|
||||
"purpose": "Run the smallest critical token-estimation set first before touching broad coverage.",
|
||||
"targetPriorities": [
|
||||
"critical"
|
||||
],
|
||||
"scenarioKinds": [
|
||||
"workflow",
|
||||
"skill",
|
||||
"role"
|
||||
],
|
||||
"recommendedMaxScenarios": 5
|
||||
},
|
||||
{
|
||||
"stage": "workflow-high-risk",
|
||||
"purpose": "Evaluate high-priority workflow prompts that drive real game-production behavior.",
|
||||
"targetPriorities": [
|
||||
"critical",
|
||||
"high"
|
||||
],
|
||||
"scenarioKinds": [
|
||||
"workflow"
|
||||
],
|
||||
"recommendedMaxScenarios": 12
|
||||
},
|
||||
{
|
||||
"stage": "skill-maintenance",
|
||||
"purpose": "Evaluate skill-maintenance, gate, QA, review, and evidence skills after workflow smoke passes.",
|
||||
"targetPriorities": [
|
||||
"critical",
|
||||
"high"
|
||||
],
|
||||
"scenarioKinds": [
|
||||
"skill"
|
||||
],
|
||||
"recommendedMaxScenarios": 12
|
||||
},
|
||||
{
|
||||
"stage": "role-boundary",
|
||||
"purpose": "Evaluate role prompts for ownership boundaries and handoff quality.",
|
||||
"targetPriorities": [
|
||||
"critical",
|
||||
"high"
|
||||
],
|
||||
"scenarioKinds": [
|
||||
"role"
|
||||
],
|
||||
"recommendedMaxScenarios": 7
|
||||
},
|
||||
{
|
||||
"stage": "full-regression",
|
||||
"purpose": "Run all scenarios only before large prompt-surface releases or after broad refactors.",
|
||||
"targetPriorities": [
|
||||
"critical",
|
||||
"high",
|
||||
"medium",
|
||||
"low"
|
||||
],
|
||||
"scenarioKinds": [
|
||||
"workflow",
|
||||
"skill",
|
||||
"role",
|
||||
"prompt"
|
||||
],
|
||||
"recommendedMaxScenarios": 31
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
# Failure Taxonomy
|
||||
|
||||
Use stable labels when reviewing skill and prompt performance eval failures.
|
||||
|
||||
- `skill-not-triggered`: the expected skill or workflow prompt was not used.
|
||||
- `wrong-workflow`: the agent selected a different workflow lane.
|
||||
- `over-triggered-workflow`: the agent ran a write workflow for a read-only review task.
|
||||
- `missing-required-context`: the agent skipped required skill, workflow, rubric, template, or project-state context.
|
||||
- `wrong-role-routing`: the prompt routed work to the wrong studio role.
|
||||
- `forbidden-surface-write`: the agent modified skill, workflow, template, or source surfaces during an eval-only task.
|
||||
- `missing-required-artifact`: the agent did not produce the expected report, plan, or evidence artifact.
|
||||
- `verification-skipped-without-rationale`: required checks were neither run nor explained.
|
||||
- `report-invalid`: the report is absent, malformed, or missing required evidence fields.
|
||||
- `weak-human-review`: the output lacks a verdict, risks, next owner, or decision point.
|
||||
- `token-bloat`: the agent loaded or repeated unnecessary context for the scenario.
|
||||
- `judge-not-evaluable`: semantic judge output was missing, malformed, or insufficient.
|
||||
@@ -0,0 +1,18 @@
|
||||
# Improvement Loop
|
||||
|
||||
When a skill or prompt performance eval fails:
|
||||
|
||||
1. Read the run report first, then use machine-readable audit data for scenario verdicts, changed-file summaries, deterministic failures, judge summaries, and token usage.
|
||||
2. Record a human review as `accepted`, `rejected`, `needs-rerun`, or `not-evaluable`.
|
||||
3. Classify failures with labels from `failure-taxonomy.md`.
|
||||
4. Choose the smallest fix target:
|
||||
- skill text when task procedure or handoff language misled the agent;
|
||||
- workflow prompt text when role routing, context selection, or stop conditions were weak;
|
||||
- rubric text when the expectation was underspecified;
|
||||
- scenario fixture when the setup did not isolate the intended behavior;
|
||||
- deterministic validator when an objective boundary violation was missed;
|
||||
- judge prompt/schema when semantic scoring was malformed.
|
||||
5. Promote real failures into minimal scenarios instead of checking in large run artifacts.
|
||||
6. Rerun the scenario and compare reports plus raw token counts.
|
||||
|
||||
LLM judge scores are advisory until calibrated by human review. Deterministic boundary failures remain failures even if a semantic judge likes the final prose.
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"id": "prompt-workflow-behavior",
|
||||
"manualOnly": true,
|
||||
"deterministicGates": [
|
||||
"workflow-triggering",
|
||||
"context-boundary",
|
||||
"write-boundary",
|
||||
"template-selection",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"id": "role-behavior",
|
||||
"manualOnly": true,
|
||||
"deterministicGates": [
|
||||
"required-read",
|
||||
"domain-boundary",
|
||||
"write-boundary",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"id": "skill-behavior",
|
||||
"manualOnly": true,
|
||||
"deterministicGates": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-architecture-review risk-verdict
|
||||
|
||||
Evaluate architecture review risk identification and director-gate boundary.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-architecture-review/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/architecture-review-skill-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-architecture-review.risk-verdict",
|
||||
"target": "cgs-architecture-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-architecture-review/risk-verdict/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-architecture-review/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-architecture-review/risk-verdict/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/architecture-review-skill-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-code-review review-findings
|
||||
|
||||
Evaluate code-review severity, evidence, and no-auto-write behavior.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-code-review/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/code-review-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-code-review.review-findings",
|
||||
"target": "cgs-code-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-code-review/review-findings/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-code-review/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-code-review/review-findings/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/code-review-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-design-review read-only-verdict
|
||||
|
||||
Evaluate read-only structured design review and verdict quality.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-design-review/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/design-review-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-design-review.read-only-verdict",
|
||||
"target": "cgs-design-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-design-review/read-only-verdict/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-design-review/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-design-review/read-only-verdict/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/design-review-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-gate-check mode-boundary
|
||||
|
||||
Evaluate phase gate mode handling and no auto-advance behavior.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-gate-check/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/gate-check-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-gate-check.mode-boundary",
|
||||
"target": "cgs-gate-check",
|
||||
"kind": "skill",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-gate-check/mode-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-gate-check/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-gate-check/mode-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/gate-check-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-qa-plan coverage-plan
|
||||
|
||||
Evaluate QA plan coverage, test levels, and execution evidence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-qa-plan/SKILL.md`
|
||||
- `templates/test_plan_template.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/qa-plan-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-qa-plan.coverage-plan",
|
||||
"target": "cgs-qa-plan",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-qa-plan/coverage-plan/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-qa-plan/SKILL.md",
|
||||
"templates/test_plan_template.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-qa-plan/coverage-plan/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/qa-plan-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-regression-suite repeatability
|
||||
|
||||
Evaluate regression-suite selection, repeatability, and evidence requirements.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-regression-suite/SKILL.md`
|
||||
- `templates/test_evidence_template.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/regression-suite-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-regression-suite.repeatability",
|
||||
"target": "cgs-regression-suite",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-regression-suite/repeatability/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-regression-suite/SKILL.md",
|
||||
"templates/test_evidence_template.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-regression-suite/repeatability/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/regression-suite-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
# Scenario: cgs-skill-improve failure-loop
|
||||
|
||||
Evaluate whether skill-improve converts failures into smallest useful fixes.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-skill-improve/SKILL.md`
|
||||
- `eval-framework/failure-taxonomy.md`
|
||||
- `eval-framework/improvement-loop.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/skill-improvement-plan.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"id": "skill.cgs-skill-improve.failure-loop",
|
||||
"target": "cgs-skill-improve",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-skill-improve/failure-loop/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-skill-improve/SKILL.md",
|
||||
"eval-framework/failure-taxonomy.md",
|
||||
"eval-framework/improvement-loop.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-skill-improve/failure-loop/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/skill-improvement-plan.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-skill-test behavioral-spec
|
||||
|
||||
Evaluate whether skill-test tests behavior specs rather than skill existence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-skill-test/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-skill-test.behavioral-spec",
|
||||
"target": "cgs-skill-test",
|
||||
"kind": "skill",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-skill-test/behavioral-spec/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-skill-test/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-skill-test/behavioral-spec/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-story-done done-verdict
|
||||
|
||||
Evaluate completion evidence, acceptance criteria, and handoff behavior.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-story-done/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/story-done-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-story-done.done-verdict",
|
||||
"target": "cgs-story-done",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-story-done/done-verdict/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-story-done/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-story-done/done-verdict/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/story-done-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-story-readiness readiness-verdict
|
||||
|
||||
Evaluate story readiness blockers, verdict levels, and next-owner clarity.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-story-readiness/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/story-readiness-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-story-readiness.readiness-verdict",
|
||||
"target": "cgs-story-readiness",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-story-readiness/readiness-verdict/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-story-readiness/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-story-readiness/readiness-verdict/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/story-readiness-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-test-evidence-review evidence-review
|
||||
|
||||
Evaluate test evidence review findings, gaps, and verifier discipline.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-test-evidence-review/SKILL.md`
|
||||
- `templates/test_evidence_template.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/test-evidence-review-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-test-evidence-review.evidence-review",
|
||||
"target": "cgs-test-evidence-review",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-test-evidence-review/evidence-review/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-test-evidence-review/SKILL.md",
|
||||
"templates/test_evidence_template.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-test-evidence-review/evidence-review/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/test-evidence-review-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: cgs-test-flakiness flakiness-diagnosis
|
||||
|
||||
Evaluate flaky-test diagnosis, reproduction notes, and isolation boundaries.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.agents/skills/cgs-test-flakiness/SKILL.md`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
- `eval-framework/rubrics/skill-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/test-flakiness-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "skill.cgs-test-flakiness.flakiness-diagnosis",
|
||||
"target": "cgs-test-flakiness",
|
||||
"kind": "skill",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/cgs-test-flakiness/flakiness-diagnosis/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".agents/skills/cgs-test-flakiness/SKILL.md",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/rubrics/skill-behavior.json",
|
||||
"eval-framework/scenarios/cgs-test-flakiness/flakiness-diagnosis/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/test-flakiness-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"task-framing",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"failure-handling",
|
||||
"human-review-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scenario: role.game-designer domain-boundary
|
||||
|
||||
Evaluate game designer acceptance criteria and design-system alignment.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/agents/game-designer.toml`
|
||||
- `eval-framework/rubrics/role-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/game-designer-role-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"id": "role.game-designer.domain-boundary",
|
||||
"target": "role.game-designer",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/role-game-designer/domain-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/agents/game-designer.toml",
|
||||
"eval-framework/rubrics/role-behavior.json",
|
||||
"eval-framework/scenarios/role-game-designer/domain-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/game-designer-role-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scenario: role.gameplay-programmer domain-boundary
|
||||
|
||||
Evaluate gameplay programmer bounded implementation plan and verification evidence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/agents/gameplay-programmer.toml`
|
||||
- `eval-framework/rubrics/role-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/gameplay-programmer-role-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"id": "role.gameplay-programmer.domain-boundary",
|
||||
"target": "role.gameplay-programmer",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/role-gameplay-programmer/domain-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/agents/gameplay-programmer.toml",
|
||||
"eval-framework/rubrics/role-behavior.json",
|
||||
"eval-framework/scenarios/role-gameplay-programmer/domain-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/gameplay-programmer-role-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scenario: role.market-analyst domain-boundary
|
||||
|
||||
Evaluate market analyst positioning, evidence separation, and competitor caveats.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/agents/market-analyst.toml`
|
||||
- `eval-framework/rubrics/role-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/market-analyst-role-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"id": "role.market-analyst.domain-boundary",
|
||||
"target": "role.market-analyst",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/role-market-analyst/domain-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/agents/market-analyst.toml",
|
||||
"eval-framework/rubrics/role-behavior.json",
|
||||
"eval-framework/scenarios/role-market-analyst/domain-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/market-analyst-role-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scenario: role.producer domain-boundary
|
||||
|
||||
Evaluate producer role prioritization, milestone risk, and next-owner clarity.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/agents/producer.toml`
|
||||
- `eval-framework/rubrics/role-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/producer-role-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"id": "role.producer.domain-boundary",
|
||||
"target": "role.producer",
|
||||
"kind": "role",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/role-producer/domain-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/agents/producer.toml",
|
||||
"eval-framework/rubrics/role-behavior.json",
|
||||
"eval-framework/scenarios/role-producer/domain-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/producer-role-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scenario: role.qa-playtester domain-boundary
|
||||
|
||||
Evaluate QA playtester reproduction evidence and implementation avoidance.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/agents/qa-playtester.toml`
|
||||
- `eval-framework/rubrics/role-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/qa-playtester-role-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"id": "role.qa-playtester.domain-boundary",
|
||||
"target": "role.qa-playtester",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/role-qa-playtester/domain-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/agents/qa-playtester.toml",
|
||||
"eval-framework/rubrics/role-behavior.json",
|
||||
"eval-framework/scenarios/role-qa-playtester/domain-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/qa-playtester-role-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scenario: role.release-manager domain-boundary
|
||||
|
||||
Evaluate release manager blocker separation, rollback notes, and ship/no-ship verdict.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/agents/release-manager.toml`
|
||||
- `eval-framework/rubrics/role-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/release-manager-role-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"id": "role.release-manager.domain-boundary",
|
||||
"target": "role.release-manager",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/role-release-manager/domain-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/agents/release-manager.toml",
|
||||
"eval-framework/rubrics/role-behavior.json",
|
||||
"eval-framework/scenarios/role-release-manager/domain-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/release-manager-role-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# Scenario: role.technical-artist domain-boundary
|
||||
|
||||
Evaluate technical artist asset pipeline boundary and engine-specific evidence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/agents/technical-artist.toml`
|
||||
- `eval-framework/rubrics/role-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/technical-artist-role-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"id": "role.technical-artist.domain-boundary",
|
||||
"target": "role.technical-artist",
|
||||
"kind": "role",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/role-technical-artist/domain-boundary/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/agents/technical-artist.toml",
|
||||
"eval-framework/rubrics/role-behavior.json",
|
||||
"eval-framework/scenarios/role-technical-artist/domain-boundary/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/technical-artist-role-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"domain-boundary",
|
||||
"delegation-quality",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"handoff-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Scenario: workflow.architecture-review behavior
|
||||
|
||||
Evaluate architecture review verdict, risks, and traceability evidence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/architecture-review.md`
|
||||
- `.agents/skills/cgs-architecture-review/SKILL.md`
|
||||
- `templates/technical_design_template.md`
|
||||
- `templates/architecture_traceability_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/architecture-review-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"id": "workflow.architecture-review.behavior",
|
||||
"target": "workflow.architecture-review",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-architecture-review/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/architecture-review.md",
|
||||
".agents/skills/cgs-architecture-review/SKILL.md",
|
||||
"templates/technical_design_template.md",
|
||||
"templates/architecture_traceability_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-architecture-review/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/architecture-review-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: workflow.bugfix behavior
|
||||
|
||||
Evaluate bugfix workflow reproduction, bounded fix guidance, and verification evidence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/bugfix.md`
|
||||
- `.agents/skills/cgs-bugfix/SKILL.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/bugfix-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "workflow.bugfix.behavior",
|
||||
"target": "workflow.bugfix",
|
||||
"kind": "workflow",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-bugfix/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/bugfix.md",
|
||||
".agents/skills/cgs-bugfix/SKILL.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-bugfix/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/bugfix-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Scenario: workflow.design-spec behavior
|
||||
|
||||
Evaluate design-spec prompt requirements, acceptance criteria, and handoff clarity.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/design-spec.md`
|
||||
- `.agents/skills/cgs-design-system/SKILL.md`
|
||||
- `templates/gdd_template.md`
|
||||
- `templates/feature_spec_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/design-spec-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"id": "workflow.design-spec.behavior",
|
||||
"target": "workflow.design-spec",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-design-spec/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/design-spec.md",
|
||||
".agents/skills/cgs-design-system/SKILL.md",
|
||||
"templates/gdd_template.md",
|
||||
"templates/feature_spec_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-design-spec/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/design-spec-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
# Scenario: workflow.game-feel-tuning behavior
|
||||
|
||||
Evaluate game-feel tuning evidence, parameter discipline, and comparison notes.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/game-feel-tuning.md`
|
||||
- `.agents/skills/cgs-balance-check/SKILL.md`
|
||||
- `templates/game_feel_tuning_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/game-feel-tuning-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"id": "workflow.game-feel-tuning.behavior",
|
||||
"target": "workflow.game-feel-tuning",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-game-feel-tuning/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/game-feel-tuning.md",
|
||||
".agents/skills/cgs-balance-check/SKILL.md",
|
||||
"templates/game_feel_tuning_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-game-feel-tuning/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/game-feel-tuning-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Scenario: workflow.market-analysis behavior
|
||||
|
||||
Evaluate market-analysis prompt context selection and positioning output quality.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/market-analysis.md`
|
||||
- `.agents/skills/cgs-content-audit/SKILL.md`
|
||||
- `templates/market_analysis_template.md`
|
||||
- `templates/pitch_document_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/market-analysis-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"id": "workflow.market-analysis.behavior",
|
||||
"target": "workflow.market-analysis",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-market-analysis/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/market-analysis.md",
|
||||
".agents/skills/cgs-content-audit/SKILL.md",
|
||||
"templates/market_analysis_template.md",
|
||||
"templates/pitch_document_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-market-analysis/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/market-analysis-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Scenario: workflow.playtest behavior
|
||||
|
||||
Evaluate QA/playtest routing and evidence quality without implementation writes.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/playtest.md`
|
||||
- `.agents/skills/cgs-playtest-report/SKILL.md`
|
||||
- `templates/playtest_report_template.md`
|
||||
- `templates/test_evidence_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/playtest-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"id": "workflow.playtest.behavior",
|
||||
"target": "workflow.playtest",
|
||||
"kind": "workflow",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-playtest/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/playtest.md",
|
||||
".agents/skills/cgs-playtest-report/SKILL.md",
|
||||
"templates/playtest_report_template.md",
|
||||
"templates/test_evidence_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-playtest/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/playtest-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: workflow.prototype behavior
|
||||
|
||||
Evaluate prototype hypothesis framing, cleanup boundaries, and validation notes.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/prototype.md`
|
||||
- `.agents/skills/cgs-prototype/SKILL.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/prototype-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "workflow.prototype.behavior",
|
||||
"target": "workflow.prototype",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-prototype/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/prototype.md",
|
||||
".agents/skills/cgs-prototype/SKILL.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-prototype/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/prototype-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Scenario: workflow.release-checklist behavior
|
||||
|
||||
Evaluate release checklist blocker separation, risk handling, and verification evidence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/release-checklist.md`
|
||||
- `.agents/skills/cgs-release-checklist/SKILL.md`
|
||||
- `templates/release_notes_template.md`
|
||||
- `templates/risk_register_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/release-checklist-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"id": "workflow.release-checklist.behavior",
|
||||
"target": "workflow.release-checklist",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-release-checklist/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/release-checklist.md",
|
||||
".agents/skills/cgs-release-checklist/SKILL.md",
|
||||
"templates/release_notes_template.md",
|
||||
"templates/risk_register_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-release-checklist/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/release-checklist-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
# Scenario: workflow.ship-check behavior
|
||||
|
||||
Evaluate ship-check readiness judgment and release-manager handoff.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/ship-check.md`
|
||||
- `templates/ship_check_template.md`
|
||||
- `templates/risk_register_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/ship-check-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"id": "workflow.ship-check.behavior",
|
||||
"target": "workflow.ship-check",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-ship-check/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/ship-check.md",
|
||||
"templates/ship_check_template.md",
|
||||
"templates/risk_register_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-ship-check/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/ship-check-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
# Scenario: workflow.sprint-plan behavior
|
||||
|
||||
Evaluate sprint-plan prioritization, dependency handling, and risk visibility.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/sprint-plan.md`
|
||||
- `.agents/skills/cgs-sprint-plan/SKILL.md`
|
||||
- `templates/sprint_plan_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/sprint-plan-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"id": "workflow.sprint-plan.behavior",
|
||||
"target": "workflow.sprint-plan",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-sprint-plan/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/sprint-plan.md",
|
||||
".agents/skills/cgs-sprint-plan/SKILL.md",
|
||||
"templates/sprint_plan_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-sprint-plan/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/sprint-plan-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Scenario: workflow.ui-ux-review behavior
|
||||
|
||||
Evaluate UI/UX review output quality and accessibility coverage.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/ui-ux-review.md`
|
||||
- `.agents/skills/cgs-ui-ux-review/SKILL.md`
|
||||
- `templates/ui_ux_review_template.md`
|
||||
- `templates/accessibility_requirements_template.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/ui-ux-review-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"id": "workflow.ui-ux-review.behavior",
|
||||
"target": "workflow.ui-ux-review",
|
||||
"kind": "workflow",
|
||||
"priority": "high",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-ui-ux-review/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/ui-ux-review.md",
|
||||
".agents/skills/cgs-ui-ux-review/SKILL.md",
|
||||
"templates/ui_ux_review_template.md",
|
||||
"templates/accessibility_requirements_template.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-ui-ux-review/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/ui-ux-review-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scenario: workflow.vertical-slice behavior
|
||||
|
||||
Evaluate vertical-slice workflow planning, stop conditions, and verification evidence.
|
||||
|
||||
## Required context
|
||||
|
||||
- `.codex/workflows/vertical-slice.md`
|
||||
- `.agents/skills/cgs-vertical-slice/SKILL.md`
|
||||
- `eval-framework/rubrics/prompt-workflow-behavior.json`
|
||||
|
||||
## Required behavior
|
||||
|
||||
- Use the target skill, workflow, or role prompt only when it fits this task.
|
||||
- Select bounded context; do not load unrelated studio surfaces.
|
||||
- Produce `production/session-state/vertical-slice-eval-report.md` with verdict, evidence, changed files or proposed files, risks, verification notes, and next owner.
|
||||
- Keep source, templates, skills, workflows, and agent definitions unchanged during the evaluation.
|
||||
- Run `npm run validate` or explain the concrete blocker.
|
||||
|
||||
## Semantic review focus
|
||||
|
||||
Judge triggering, context selection, output quality, verification discipline, human-review usefulness, and token discipline.
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"id": "workflow.vertical-slice.behavior",
|
||||
"target": "workflow.vertical-slice",
|
||||
"kind": "workflow",
|
||||
"priority": "critical",
|
||||
"manualOnly": true,
|
||||
"prompt": "eval-framework/scenarios/workflow-vertical-slice/behavior/prompt.md",
|
||||
"expected": {
|
||||
"mustRead": [
|
||||
".codex/workflows/vertical-slice.md",
|
||||
".agents/skills/cgs-vertical-slice/SKILL.md",
|
||||
"eval-framework/rubrics/prompt-workflow-behavior.json",
|
||||
"eval-framework/scenarios/workflow-vertical-slice/behavior/prompt.md"
|
||||
],
|
||||
"mustChange": [
|
||||
"production/session-state/vertical-slice-eval-report.md"
|
||||
],
|
||||
"mustNotChange": [
|
||||
"src/**",
|
||||
".agents/skills/**",
|
||||
".codex/workflows/**",
|
||||
".codex/agents/**",
|
||||
"templates/**"
|
||||
],
|
||||
"mustRunOrExplain": [
|
||||
"npm run validate"
|
||||
],
|
||||
"report": {
|
||||
"required": true
|
||||
}
|
||||
},
|
||||
"grading": {
|
||||
"deterministic": [
|
||||
"required-read",
|
||||
"write-boundary",
|
||||
"required-change",
|
||||
"verification-evidence",
|
||||
"report-presence"
|
||||
],
|
||||
"semanticDimensions": [
|
||||
"triggering",
|
||||
"context-selection",
|
||||
"role-routing",
|
||||
"template-selection",
|
||||
"output-quality",
|
||||
"verification-discipline",
|
||||
"stop-condition-quality",
|
||||
"token-discipline"
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user