codex-game-studio/openspec/changes/expand-eval-framework-coverage/specs/performance-eval-framework/spec.md

MODIFIED Requirements

Requirement: Framework contract validation

The eval framework SHALL validate catalog targets, rubrics, scenarios, manual-only behavior, semantic dimensions, and the absence of existence-only success criteria.

Scenario: Framework validation reports coverage and behavior contracts

• WHEN repository validation runs
• THEN validation includes checks for manual-only behavior, target mappings, behavioral expectations, semantic dimensions, no existence-only checks, and first-pass coverage thresholds
• THEN validation fails if workflow, skill, or role coverage drops below the first-pass threshold

Requirement: Scenario grading contract

The scenario grader SHALL evaluate required reads, forbidden writes, required changed artifacts, verification evidence, report presence, and optional token usage.

Scenario: Forbidden write fails before semantic judging

• WHEN an observation changes a path matching a scenario mustNotChange pattern
• THEN the deterministic grading result is fail
• THEN the failure includes a forbidden-write label

Scenario: Token usage is preserved

• WHEN an observation includes raw token counts
• THEN the result includes those counts for comparison
• THEN the result does not enforce a token budget