mirror of
https://github.com/merlinhu1/codex-game-studio.git
synced 2026-08-25 07:54:34 +02:00
fix: honor review fix flags for custom roles
Render read-only review prompts and bounded fix prompts for project-local custom role runs instead of accepting --review/--fix without lifecycle prompts. Add regression coverage and update routed Truthmark docs for the custom role review/fix contract.
This commit is contained in:
@@ -37,7 +37,7 @@ This doc was created from the editable engineering-behavior template at docs/tru
|
||||
- Behavioral evaluation scenarios cover representative role/workflow prompt contracts by rendering prompts locally and checking required obligations, output-contract fields, selected context categories, required workflow templates, forbidden templates, forbidden future-only drift, and prompt-size bounds without LLM judges or hosted evaluators. Workflow scenarios without a project root still append the selected built-in workflow template bodies so repository-level behavioral validation exercises template rendering without requiring a temporary initialized project.
|
||||
- Market analyst, data scientist, game-feel, UI/UX, QA, release, audio, content, systems/economy, accessibility/localization/live-ops/community, and technical specialist roles have bounded default role packages; reusable template selection remains task-specific rather than loading every role or template.
|
||||
- Project-local customization packs may add `custom-*` role, workflow, and template IDs through `.codex/studio/config.json`; these overlays are extend-only and cannot replace built-in role/workflow/template IDs.
|
||||
- Custom role prompts render the same visible role ID, context strategy, expected outputs, review checklist, write-policy, sandbox, and selected-template sections as built-in role runs, while custom workflow prompts use the generic `workflow <id>` command, require the declared workflow markdown file to exist, include that file as selected context, and render its body as workflow instructions rather than adding unbounded shortcut commands. Custom workflows may target either a built-in studio role or a project-local custom role; built-in targets render through the standard Codex session prompt and role contract, while custom targets append the project-local custom role prompt.
|
||||
- Custom role prompts render the same visible role ID, context strategy, expected outputs, review checklist, write-policy, sandbox, and selected-template sections as built-in role runs. Custom role review/fix prompt rendering reuses the built-in QA review prompt surface for read-only review and appends the project-local custom role prompt for bounded fix passes. Custom workflow prompts use the generic `workflow <id>` command, require the declared workflow markdown file to exist, include that file as selected context, and render its body as workflow instructions rather than adding unbounded shortcut commands. Custom workflows may target either a built-in studio role or a project-local custom role; built-in targets render through the standard Codex session prompt and role contract, while custom targets append the project-local custom role prompt.
|
||||
- Generated workflow files carry deterministic source-input and rendered-body hash metadata that covers workflow definition fields and the owning role display name, expected outputs, and review checklist used in the rendered workflow body.
|
||||
|
||||
## Core Rules
|
||||
@@ -83,6 +83,7 @@ This doc was created from the editable engineering-behavior template at docs/tru
|
||||
- Decision (2026-06-17): Expand role prompt depth through structured compact role contracts and reusable shared fragments rendered in both standard Codex session prompts and generated project role prompts, while keeping role/task context bounded.
|
||||
- Decision (2026-06-17): Add representative behavioral-evaluation fixtures as deterministic prompt-contract checks instead of adopting hosted/LLM agent-evaluation infrastructure.
|
||||
- Decision (2026-06-17): Add project-local customization as an extend-only overlay so users can define local `custom-*` roles, workflows, and templates while preserving built-in registries and bounded context selection; custom workflows may reuse built-in roles rather than requiring duplicate local role definitions.
|
||||
- Decision (2026-06-17): Reuse the built-in QA review prompt surface for custom role review passes and the configured custom role prompt for custom fix passes, preserving extend-only customization without adding custom review-role schema.
|
||||
|
||||
## Rationale
|
||||
|
||||
|
||||
@@ -49,8 +49,8 @@ Runtime execution prepares bounded Codex prompts, evaluates studio write policy
|
||||
- Guided studio mutating runs require either a matching approval or `--approved-by-user`; the explicit override uses `override-write` provenance. Fast-prototype mutating runs use advisory write provenance.
|
||||
- Dry-run output includes the eligibility result, active write policy, sandbox, file-edit permission, approval/override/advisory provenance, and approval diagnostics for guided and strict studio modes. Dry-run does not write run metadata.
|
||||
- Non-dry runs write prompt and metadata before executing Codex.
|
||||
- Implementation, review, and fix prompts include a shared `# Context Contract` section that records project stage, studio mode, phase, write policy, sandbox, file-edit permission, selected context entries, and bounded omissions/blockers. Review prompts add read-only review instructions, and fix prompts preserve the same selected implementation context plus a bounded blocker placeholder for execution-time review/verification blockers.
|
||||
- Implementation and fix passes use `danger-full-access` by default when eligibility allows file edits; `workspace-write` is used only when the caller passes the explicit constrained-sandbox option. Review passes use a read-only Codex sandbox. Fix prompts receive the same generated role prompt and selected package templates as the primary implementation prompt, and review prompts receive the generated QA playtester prompt plus selected QA templates.
|
||||
- Built-in implementation, review, and fix prompts include a shared `# Context Contract` section that records project stage, studio mode, phase, write policy, sandbox, file-edit permission, selected context entries, and bounded omissions/blockers. Custom role review and fix prompts also use the shared context contract; custom implementation prompts keep their custom-role session shape while listing selected context and the active policy fields. Review prompts add read-only review instructions, and fix prompts preserve the same selected implementation context plus a bounded blocker placeholder for execution-time review/verification blockers.
|
||||
- Implementation and fix passes use `danger-full-access` by default when eligibility allows file edits; `workspace-write` is used only when the caller passes the explicit constrained-sandbox option. Review passes use a read-only Codex sandbox. Built-in fix prompts receive the same generated role prompt and selected package templates as the primary implementation prompt, custom fix prompts receive the configured custom role prompt and selected project-local templates, and review prompts receive the generated QA playtester prompt plus selected QA templates.
|
||||
- Prompt/session/run metadata records the active write policy, file-edit permission, sandbox, and eligibility metadata used for the runtime path.
|
||||
- `src/studio-policy.ts` defines pure, dependency-light project-stage/studio-mode policy primitives. These helpers keep lifecycle stage (`design`, `prototype`, `development`) separate from studio mode (`fast-prototype`, `guided-studio`, `strict-studio`), map plan/review/ship phases to read-only, classify implementation/fix eligibility, and map allowed mutating policies to `danger-full-access` by default unless a constrained sandbox option explicitly requests `workspace-write`.
|
||||
- Task runs mutate task status only for non-dry execution.
|
||||
@@ -100,6 +100,7 @@ Runtime execution prepares bounded Codex prompts, evaluates studio write policy
|
||||
- Decision (2026-06-14): Fail wrong-engine specialist role runs before constructing contradictory prompts or writing run metadata.
|
||||
- Decision (2026-06-17): Add task-relevant active-engine reference requests to role-run and workflow context contracts so module/plugin depth is selected by role/task relevance instead of broad prompt loading.
|
||||
- Decision (2026-06-17): Route project-local custom role runs through the same visible write-policy, sandbox, context-selection, prompt-cache, and template-selection contracts as built-in roles instead of introducing a separate plugin runtime.
|
||||
- Decision (2026-06-17): Honor review/fix flags for project-local custom role runs by rendering a read-only QA review prompt and a bounded custom-role fix prompt rather than accepting the flags without lifecycle prompts.
|
||||
|
||||
## Rationale
|
||||
|
||||
|
||||
@@ -2,6 +2,8 @@ import type { CodexStudioSession } from "./codex-session.js";
|
||||
import type { ContextManifestEntry } from "./context-manifest.js";
|
||||
import type { StudioProjectState } from "./projects.js";
|
||||
|
||||
type ContextContractSession = Pick<CodexStudioSession, "phase" | "writePolicy" | "sandbox" | "allowFileEdits">;
|
||||
|
||||
function formatEntries(entries: ContextManifestEntry[]): string[] {
|
||||
const selected = entries.filter((entry) => entry.status === "selected");
|
||||
if (!selected.length) return ["- None"];
|
||||
@@ -15,7 +17,7 @@ function formatOmissions(entries: ContextManifestEntry[]): string[] {
|
||||
}
|
||||
|
||||
export function renderContextContract(args: {
|
||||
session: CodexStudioSession;
|
||||
session: ContextContractSession;
|
||||
projectStage: StudioProjectState["mode"];
|
||||
studioMode: StudioProjectState["studioMode"];
|
||||
entries: ContextManifestEntry[];
|
||||
|
||||
+76
-3
@@ -375,6 +375,74 @@ function prepareCustomRun(role: ReturnType<typeof findCustomRole> extends infer
|
||||
renderSelectedTemplates(templateIds, "## Selected Templates", { projectRoot }),
|
||||
artifactBodies.join("")
|
||||
].join("\n");
|
||||
const reviewObjective = `Review the implementation for: ${task}. Inspect the diff and verification output. Return only JSON with blockers, warnings, summary, and needsFix.`;
|
||||
const reviewSelection = selectContextEntries(projectRoot, [
|
||||
{ sourcePath: "AGENTS.md", reason: "project instructions", required: true },
|
||||
{ sourcePath: ".codex/studio.json", reason: "project state", required: true },
|
||||
{ sourcePath: ".codex/prompts/qa-playtester.md", reason: "review role prompt", required: true },
|
||||
...contextSelection.selected.map((entry) => ({ sourcePath: entry.sourcePath, reason: `implementation context: ${entry.reason}`, required: entry.required }))
|
||||
]);
|
||||
const reviewSession = createCodexStudioSession({
|
||||
projectRoot,
|
||||
role: "qa-playtester",
|
||||
objective: reviewObjective,
|
||||
phase: "review",
|
||||
engine: studio.engine,
|
||||
contextFiles: contextFiles(reviewSelection.entries),
|
||||
expectedOutputs: ["Review JSON", "Blockers", "Warnings"],
|
||||
allowFileEdits: false,
|
||||
sandbox: "read-only",
|
||||
writePolicy: "read-only",
|
||||
reviewMode: "diff"
|
||||
});
|
||||
const reviewPrompt = options.review
|
||||
? `${renderCodexPrompt({
|
||||
...reviewSession,
|
||||
contextContract: renderContextContract({
|
||||
session: reviewSession,
|
||||
projectStage: studio.mode,
|
||||
studioMode: studio.studioMode,
|
||||
entries: reviewSelection.entries,
|
||||
readOnlyReview: true
|
||||
})
|
||||
})}${renderRuntimeContextBlock("qa-playtester", projectRoot, reviewObjective)}`
|
||||
: undefined;
|
||||
const fixContract = renderContextContract({
|
||||
session: { phase: "fix", writePolicy: eligibility.writePolicy, sandbox: eligibility.codexSandbox, allowFileEdits: eligibility.allowFileEdits },
|
||||
projectStage: studio.mode,
|
||||
studioMode: studio.studioMode,
|
||||
entries: contextSelection.entries,
|
||||
blockers: ["Review and verification blockers will be supplied at execution time."]
|
||||
});
|
||||
const fixPrompt =
|
||||
options.fix && maxFixPasses > 0
|
||||
? [
|
||||
"# Codex Game Studio Custom Role Session",
|
||||
"",
|
||||
`Role: ${role.displayName}`,
|
||||
`Role ID: ${role.id}`,
|
||||
`Context Strategy: ${role.contextStrategy}`,
|
||||
"Phase: fix",
|
||||
`Project Root: ${projectRoot}`,
|
||||
`Objective: Fix verification failures or review blockers for: ${task}`,
|
||||
`Engine: ${studio.engine}`,
|
||||
`Sandbox: ${eligibility.codexSandbox}`,
|
||||
`Write Policy: ${eligibility.writePolicy}`,
|
||||
`File Edits: ${eligibility.allowFileEdits ? "allowed" : "not allowed"}`,
|
||||
"",
|
||||
fixContract,
|
||||
"",
|
||||
"## Context Files",
|
||||
contextFilesForRun.length ? contextFilesForRun.map((file) => `- ${file}`).join("\n") : "- None selected",
|
||||
"",
|
||||
"## Verification",
|
||||
options.verifyCommand ? `${options.verifyCommand.command} ${options.verifyCommand.args.join(" ")}`.trim() : "No verification command provided.",
|
||||
"",
|
||||
renderCustomRolePrompt(projectRoot, role),
|
||||
renderSelectedTemplates(templateIds, "## Selected Templates", { projectRoot }),
|
||||
artifactBodies.join("")
|
||||
].join("\n")
|
||||
: undefined;
|
||||
const runId = `${new Date().toISOString().replace(/[-:.TZ]/g, "").slice(0, 17)}-${process.pid}-${++runSequence}`;
|
||||
const runDir = path.join(projectRoot, ".codex", "runs", `${runId}-${role.id}`);
|
||||
const promptPath = path.join(runDir, "prompt.md");
|
||||
@@ -384,11 +452,16 @@ function prepareCustomRun(role: ReturnType<typeof findCustomRole> extends infer
|
||||
writeFileSync(promptPath, prompt);
|
||||
writeFileSync(
|
||||
metadataPath,
|
||||
`${JSON.stringify({ timestamp: new Date().toISOString(), product: "codex-game-studio", project: path.relative(cwd, projectRoot) || ".", role: role.id, task, customRole: true, prompt_chars: prompt.length, prompt_cache_path: path.relative(cwd, promptPath), writePolicy: eligibility.writePolicy, allowFileEdits: eligibility.allowFileEdits, codexSandbox: eligibility.codexSandbox, eligibility }, null, 2)}\n`
|
||||
`${JSON.stringify({ timestamp: new Date().toISOString(), product: "codex-game-studio", project: path.relative(cwd, projectRoot) || ".", role: role.id, task, customRole: true, prompt_chars: prompt.length, prompt_cache_path: path.relative(cwd, promptPath), review: Boolean(reviewPrompt), fix: Boolean(fixPrompt), max_fix_passes: maxFixPasses, writePolicy: eligibility.writePolicy, allowFileEdits: eligibility.allowFileEdits, codexSandbox: eligibility.codexSandbox, eligibility }, null, 2)}\n`
|
||||
);
|
||||
}
|
||||
const codexBin = options.codexBin ?? resolveCodexCommand();
|
||||
const codexCommand = codexExecInvocation(projectRoot, codexBin, eligibility.codexSandbox);
|
||||
const reviewCodexCommand = reviewPrompt ? codexExecInvocation(projectRoot, codexBin, "read-only") : undefined;
|
||||
const dryRunExtra = [
|
||||
reviewPrompt ? `\n\nReview Codex command: ${reviewCodexCommand?.display}\n\nReview prompt:\n${reviewPrompt}\n\nExpected review JSON schema: {"blockers":[],"warnings":[],"summary":"","needsFix":false}` : "",
|
||||
fixPrompt ? `\n\nFix prompt (max passes: ${maxFixPasses}):\n${fixPrompt}` : ""
|
||||
].join("");
|
||||
const approvalDiagnostic =
|
||||
options.dryRun && studio.studioMode !== "fast-prototype"
|
||||
? formatApprovalDiagnostic(
|
||||
@@ -399,9 +472,9 @@ function prepareCustomRun(role: ReturnType<typeof findCustomRole> extends infer
|
||||
const output = options.printPrompt
|
||||
? prompt
|
||||
: options.dryRun
|
||||
? `Prompt cache (not written): ${promptPath}\nMetadata (not written): ${metadataPath}\n${formatEligibility(eligibility)}\nContext files:\n${contextFilesForRun.map((f) => `- ${f}`).join("\n")}\nCodex command: ${codexCommand.display}${approvalDiagnostic ? `\n\n${approvalDiagnostic}` : ""}`
|
||||
? `Prompt cache (not written): ${promptPath}\nMetadata (not written): ${metadataPath}\n${formatEligibility(eligibility)}\nContext files:\n${contextFilesForRun.map((f) => `- ${f}`).join("\n")}\nCodex command: ${codexCommand.display}${approvalDiagnostic ? `\n\n${approvalDiagnostic}` : ""}${dryRunExtra}`
|
||||
: `Prompt cache written: ${promptPath}\n${formatEligibility(eligibility)}\nExecuting Codex: ${codexCommand.display}`;
|
||||
return { prompt, promptPath, metadataPath, projectRoot, role: roleInput, task, contextFiles: contextFilesForRun, verification: options.verifyCommand, codexCommand, output, maxFixPasses, eligibility };
|
||||
return { prompt, promptPath, metadataPath, projectRoot, role: roleInput, task, contextFiles: contextFilesForRun, verification: options.verifyCommand, codexCommand, reviewCodexCommand, output, reviewPrompt, fixPrompt, maxFixPasses, eligibility };
|
||||
}
|
||||
|
||||
export function prepareRun(roleInput: string, options: RunOptions, cwd = process.cwd()): PreparedRun {
|
||||
|
||||
@@ -102,6 +102,23 @@ describe("project-local customization packs", () => {
|
||||
expect(run.output).toContain("Template: custom-boss-brief");
|
||||
});
|
||||
|
||||
test("custom role runs render review and fix prompts when requested", () => {
|
||||
const cwd = mkdtempSync(path.join(tmpdir(), "ogs-custom-review-fix-"));
|
||||
const { projectRoot } = initProject({ name: "Custom Review Fix", engine: "godot", mode: "prototype", studioMode: "fast-prototype", nonInteractive: true }, cwd);
|
||||
writeValidCustomPack(projectRoot, "implement");
|
||||
|
||||
const run = prepareRun("custom-boss-designer", { project: projectRoot, task: "Draft a boss fight brief with phase readability", dryRun: true, review: true, fix: true }, cwd);
|
||||
|
||||
expect(run.reviewPrompt).toContain("Read-only review: inspect diff and verification output; do not edit files.");
|
||||
expect(run.reviewPrompt).toContain(".codex/prompts/qa-playtester.md");
|
||||
expect(run.fixPrompt).toContain("# Project Custom Role Prompt: .codex/custom/roles/boss-designer.md");
|
||||
expect(run.fixPrompt).toContain("Bounded Blockers:");
|
||||
expect(run.reviewCodexCommand?.args).toEqual(expect.arrayContaining(["--sandbox", "read-only"]));
|
||||
expect(run.output).toContain("Review Codex command:");
|
||||
expect(run.output).toContain("Expected review JSON schema:");
|
||||
expect(run.output).toContain("Fix prompt (max passes: 1):");
|
||||
});
|
||||
|
||||
test("project-local workflows can render with built-in roles accepted by validation", () => {
|
||||
const cwd = mkdtempSync(path.join(tmpdir(), "ogs-custom-built-in-workflow-"));
|
||||
const { projectRoot } = initProject({ name: "Built In Workflow", engine: "godot", mode: "development", nonInteractive: true }, cwd);
|
||||
|
||||
Reference in New Issue
Block a user