Compare commits

..

52 Commits

Author SHA1 Message Date
Drew Ritter
1a9cd9ec4b Improve visual companion: per-question decisions, cross-platform server docs
- Rewrite "When to Use" with explicit browser vs terminal guidance
- Add per-question decision heuristic instead of session-wide mode
- Add unload step when returning to terminal from visual step
- Remove all ${CLAUDE_PLUGIN_ROOT} references from skill docs
- Add cross-platform server startup: bash path + node fallback with env vars
- Give AI guidance to find plugin dir across Claude Code and Codex

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 18:52:45 -08:00
Drew Ritter
bf6d336950 minor change to unload a page from viz brainstorm after the user makes a decision 2026-02-19 16:53:30 -08:00
Drew Ritter
ce0f9a28be Refactor visual brainstorming: browser displays, terminal commands (#509)
* Refactor visual brainstorming: browser displays, terminal commands

Replaces the blocking TaskOutput/wait-for-feedback.sh pattern with a
non-blocking model where the browser is an interactive display and the
terminal stays available for conversation.

Server changes:
- Write user click events to .events JSONL file (per-screen, cleared on
  new screen push) so Claude reads them on its next turn
- Replace regex-based wrapInFrame with <!-- CONTENT --> placeholder
- Add --foreground flag to start-server.sh for sandbox environments
- Harden startup with nohup/disown and liveness check

UI changes:
- Remove feedback footer (textarea + Send button)
- Add selection indicator bar ("Option X selected — return to terminal")
- Narrow click handler to [data-choice] elements only

Skill changes:
- Rewrite visual-companion.md for non-blocking loop
- Fix visual companion being skipped on Codex (no browser tools needed)
- Make visual companion offer a standalone question (one question rule)

Deletes wait-for-feedback.sh entirely.

* Add visual companion offer to brainstorming checklist for UX topics

The visual companion was a disconnected section at the bottom of SKILL.md
that agents never reached because it wasn't in the mandatory checklist.
Now step 2 evaluates whether the topic involves visual/UX decisions and
offers the companion if so. Non-visual topics (APIs, data models, etc.)
skip the step entirely.

* Add multi-select support to visual companion

Containers with data-multiselect allow toggling multiple selections.
Without it, behavior is unchanged (single-select). Indicator bar shows
count when multiple items are selected.
2026-02-19 16:31:51 -08:00
Drew Ritter
3a254ba002 Remove session scope language from SUBAGENT-STOP
Drop 'top-level sessions only' which may cause the model to treat
using-superpowers as a first-turn-only skill, breaking skill chaining
on follow-up turns.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Drew Ritter
192cb7db8e Soften SUBAGENT-STOP to only skip this skill, not suppress all behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Drew Ritter
a014baf5b9 Remove skill override directives from dispatch templates
Let the SUBAGENT-STOP gate in using-superpowers handle skill leakage
instead of per-template directives. This avoids blocking non-superpowers
skills that users may want subagents to use.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Drew Ritter
49bbad7084 Move Codex tool mapping to progressive disclosure reference file
Addresses Jesse's review feedback on PR #450:
- Move inline routing table from using-superpowers to references/codex-tools.md,
  leveraging Codex's native progressive disclosure for companion files
- Narrow SUBAGENT-STOP from "Do not invoke skills" to "Do not invoke
  superpowers skills" so subagents can still use non-superpowers skills

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Drew Ritter
79dbce8b10 Sharpen subagent skill override directive in dispatch templates
Replace verbose scope explanations with a direct override statement
that explicitly claims priority over prior guidance (i.e. the
using-superpowers 1% rule injected by Codex's skill discovery).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Drew Ritter
2e12d4fb34 Document collab feature requirement for Codex subagent skills
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Drew Ritter
5535e2d0b3 Prevent Codex subagent skill leakage via gate check and dispatch routing table
Codex subagents inherit filesystem access and can discover superpowers skills
via native discovery. Without guidance, they activate the 1% rule and invoke
full skill workflows instead of executing their assigned task.

- Add SUBAGENT-STOP gate check above the 1% rule in using-superpowers
- Add Codex dispatch routing table (spawn_agent/wait/close_agent)
- Add scope directives to all 4 subagent dispatch templates

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
7400d43599 fix: restore polyglot wrapper to fix Windows hook window spawning
Claude Code spawns hook commands with shell:true + windowsHide:true,
but on Windows the execution chain cmd.exe -> bash.exe causes Git
Bash (MSYS2) to allocate its own console window, bypassing the hide
flag. This creates visible terminal windows that steal focus on every
SessionStart event (startup, resume, clear, compact).

The fix:
- Rename session-start.sh to session-start (no extension) so Claude
  Code's .sh auto-detection regex doesn't fire and prepend "bash"
- Restore run-hook.cmd polyglot wrapper to control bash invocation
  on Windows (tries known Git Bash paths, then PATH, then exits
  silently if no bash found)
- On Unix, the polyglot's shell portion runs the script directly

This avoids Claude Code's broken .sh auto-prepend, gives us control
over how bash is invoked on Windows, and gracefully handles missing
bash instead of erroring.

Addresses: #440, #414, #354, #417, #293
Upstream: anthropics/claude-code#14828
2026-02-19 09:35:45 -08:00
Jesse Vincent
cb75011d91 fix: move scope assessment into understanding phase
Testing showed the model skipped scope assessment when it was a
separate step after "Understanding the idea." Inlining it as the
first thing in understanding ensures it fires before detailed questions.
2026-02-19 09:35:45 -08:00
Jesse Vincent
56e22b2afc feat: add project-level scope assessment to brainstorming pipeline
Brainstorming now assesses whether a project is too large for a single
spec and helps decompose into sub-projects. Spec reviewer checks scope.
Writing-plans has a backstop if brainstorming missed it.
2026-02-19 09:35:45 -08:00
Jesse Vincent
a0dc300a6e feat: add architecture and file size checks to review loops
Spec reviewer now checks for unit decomposition with clear boundaries.
Plan reviewer now checks file structure and whether files will grow
too large to reason about.
2026-02-19 09:35:45 -08:00
Jesse Vincent
1e3353b232 feat: add file growth check to code quality reviewer
Focus on whether this implementation grew or created large files,
not pre-existing file sizes in brownfield codebases.
2026-02-19 09:35:45 -08:00
Jesse Vincent
21c7284d54 fix: address review feedback on architecture guidance
- Define DONE_WITH_CONCERNS handling in SDD controller flow
- Make implementer action explicit when file grows beyond plan intent
- Reword writing-plans file size reasoning (avoid tooling-artifact language)
- Add decomposition awareness to code quality reviewer prompt
2026-02-19 09:35:45 -08:00
Jesse Vincent
8d996aa829 feat: add architecture guidance and capability-aware escalation to skills
Brainstorming: design-for-isolation guidance and brownfield codebase awareness
Writing-plans: file structure section requiring decomposition before task definition
Implementer prompt: code organization awareness, structured escalation protocol
  (DONE/DONE_WITH_CONCERNS/BLOCKED/NEEDS_CONTEXT), explicit permission to stop
Subagent-driven-development: provider-agnostic model selection tiers, escalation handling
2026-02-19 09:35:45 -08:00
Jesse Vincent
14df703a51 chore: gitignore triage directory 2026-02-19 09:35:45 -08:00
Jesse Vincent
17f80fd0ce docs: add brainstorm visual companion improvements to release notes 2026-02-19 09:35:45 -08:00
Jesse Vincent
2716e12781 docs: restructure brainstorming skill with progressive disclosure
SKILL.md is now minimal: process, principles, and a prompt that notes
the visual companion is new/token-intensive/slow. All visual companion
details move to visual-companion.md as a progressive disclosure document
read only when the user opts in.

Delete CLAUDE-INSTRUCTIONS.md (content folded into visual-companion.md).
Document fragment vs full-document behavior and --project-dir persistence.
2026-02-19 09:35:45 -08:00
Jesse Vincent
b52af8427c feat: persist brainstorm mockups to .superpowers/ directory
start-server.sh now accepts --project-dir to store session files under
.superpowers/brainstorm/ instead of /tmp. stop-server.sh only deletes
ephemeral /tmp sessions, keeping persistent ones for later review.

Fix test race condition with polling-based server startup wait.
2026-02-19 09:35:45 -08:00
Jesse Vincent
5b00c6eb50 refactor: server-side frame wrapping and helper.js consolidation
Move toggleSelect/send/selectedChoice from frame-template.html inline
script to helper.js so they're auto-injected. Server now detects bare
HTML fragments (no DOCTYPE/html tag) and wraps them in the frame
template automatically. Full documents pass through as before.

Fix dark mode in sendToClaude confirmation (was using hardcoded colors).
Fix test env var bug (BRAINSTORM_SCREEN -> BRAINSTORM_DIR).
Add tests for fragment wrapping, full doc passthrough, and helper.js.
2026-02-19 09:35:45 -08:00
Jesse Vincent
7398af9947 test: rewrite document review test as proper integration test
- Creates test project with spec containing intentional errors
- Runs Claude to actually review using spec-document-reviewer template
- Verifies reviewer catches TODO and "specified later" deferrals
- Checks review format and verdict

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
6e6b1ae546 test: add end-to-end tests for document review system
Tests verify:
- Spec document reviewer checks (completeness, TODOs)
- Plan document reviewer checks (spec alignment, task decomposition)
- Review loops exist in brainstorming and writing-plans skills
- Chunk-by-chunk review for plans with 1000-line limit
- Iteration guidance (5 iterations, escalate to human)
- Checkbox syntax on steps only (not task headings)
- Correct directories (docs/superpowers/specs, docs/superpowers/plans)
- Reviewers are advisory
- Same agent fixes issues (preserves context)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
8d73d0fee2 fix: remove checkbox from task headings, keep on steps only
The `- [ ] ### Task N:` syntax was unusual and might not render
correctly in all markdown parsers. Now only steps have checkboxes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
c9f5a393da docs: add document review system spec and plan
- Spec: docs/superpowers/specs/2026-01-22-document-review-system-design.md
- Plan: docs/superpowers/plans/2026-01-22-document-review-system.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
a3d45dc32b docs: update plan header to reference checkbox syntax
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
21673839c0 feat: add plan review loop and checkbox syntax to writing-plans skill
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
800d6a2405 feat: add plan document reviewer prompt template
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
1473d86800 feat: add spec review loop to brainstorming skill
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
bdfa38509d feat: add spec document reviewer prompt template
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
fd90ed5dd9 feat: enforce subagent-driven-development on capable harnesses
- Subagent-driven-development is now mandatory when harness supports it
- No longer offer choice between subagent-driven and executing-plans
- Executing-plans reserved for harnesses without subagent capability
- Update plan header to reference both execution paths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
fbcf9475e4 feat: enforce brainstorming → writing-plans transition
- Make writing-plans REQUIRED after design approval
- Explicitly forbid platform planning features (EnterPlanMode, etc.)
- Forbid direct implementation without writing-plans skill

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
14c17242cc refactor: restructure specs and plans directories
- Specs (brainstorming output) now go to docs/superpowers/specs/
- Plans (writing-plans output) now go to docs/superpowers/plans/
- User preferences for locations override these defaults
- Update all skill references and test files

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
786aa5cf44 fix: Windows hook execution for Claude Code 2.1.x (#331)
* fix: convert shell scripts from CRLF to LF line endings

Add .gitattributes to enforce LF line endings for shell scripts,
preventing bash errors like "/usr/bin/bash: line 1: : command not found"
when scripts are checked out on Windows with CRLF.

Fixes #317 (SessionStart hook fails due to CRLF line endings)

Files converted:
- hooks/session-start.sh
- lib/brainstorm-server/start-server.sh
- lib/brainstorm-server/stop-server.sh
- lib/brainstorm-server/wait-for-feedback.sh
- skills/systematic-debugging/find-polluter.sh

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: update Windows hook execution for Claude Code 2.1.x

Claude Code 2.1.x changed the Windows execution model: it now auto-detects
.sh files in hook commands and prepends "bash " automatically. This broke
the polyglot wrapper because:

  Before: "run-hook.cmd" session-start.sh  (wrapper executes)
  After:  bash "run-hook.cmd" session-start.sh  (bash can't run .cmd)

Changes:
- hooks.json now calls session-start.sh directly (Claude Code handles bash)
- Added deprecation comment to run-hook.cmd explaining the change
- Updated RELEASE-NOTES.md

Fixes #317, #313, #275, #292

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
8ab3eec9bc feat(opencode): use native skills and fix agent reset bug (#226) (#330)
* fix use_skill agent context (#290)

* fix: respect OPENCODE_CONFIG_DIR for personal skills lookup (#297)

* fix: respect OPENCODE_CONFIG_DIR for personal skills lookup

The plugin was hardcoded to look for personal skills in ~/.config/opencode/skills,
ignoring users who set OPENCODE_CONFIG_DIR to a custom path (e.g., for dotfiles management).

Now uses OPENCODE_CONFIG_DIR if set, falling back to the default path.

* fix: update help text to use dynamic paths

Use configDir and personalSkillsDir variables in help text so paths
are accurate when OPENCODE_CONFIG_DIR is set.

* fix: normalize OPENCODE_CONFIG_DIR before use

Handle edge cases where the env var might be:
- Empty or whitespace-only
- Using ~ for home directory (common in .env files)
- A relative path

Now trims, expands ~, and resolves to absolute path.

* feat(opencode): use native skills and fix agent reset bug (#226)

- Replace custom use_skill/find_skills tools with OpenCode's native skill tool
- Use experimental.chat.system.transform hook instead of session.prompt
  (fixes #226 agent reset on first message)
- Symlink skills directory into ~/.config/opencode/skills/superpowers/
- Update installation docs with comprehensive Windows support:
  - Command Prompt, PowerShell, and Git Bash instructions
  - Proper symlink vs junction handling
  - Reinstall safety with cleanup steps
  - Verification commands for each shell

* Add OpenCode native skills changes to release notes

Documents:
- Breaking change: switch to native skill tool
- Fix for agent reset bug (#226)
- Fix for Windows installation (#232)

---------

Co-authored-by: Vinicius da Motta <viniciusmotta8@gmail.com>
Co-authored-by: oribi <oribarilan@gmail.com>
2026-02-19 09:35:45 -08:00
Jesse Vincent
630c0f7b54 Add instruction priority hierarchy to using-superpowers skill
Clarifies that user instructions (CLAUDE.md, direct requests) always
take precedence over Superpowers skills, which in turn override
default system prompt behavior. Ensures users remain in control.

Also updates RELEASE-NOTES.md with unreleased changes including
the visual companion feature.
2026-02-19 09:35:45 -08:00
Jesse Vincent
b3e922c10a Use semantic filenames for visual companion screens
Server now watches directory for new .html files instead of a single
screen file. Claude writes to semantically named files like
platform.html, style.html, layout.html - each screen is a new file.

Benefits:
- No need to read before write (files are always new)
- Semantic filenames describe what's on screen
- History preserved in directory for debugging
- Server serves newest file by mtime automatically

Updated: index.js, start-server.sh, and all documentation.
2026-02-19 09:35:44 -08:00
Jesse Vincent
e5a7c05528 docs: improve terminal UX for visual companion
- Never use cat/heredoc for HTML (dumps noise into terminal)
- Read screen_file first before Write tool to avoid errors
- Remind user of URL on every step, not just first
- Give text summary of what's on screen before they look
2026-02-19 09:35:44 -08:00
Jesse Vincent
9ab744b2a9 refactor: simplify visual companion workflow, improve guidance
Scripts:
- Rename show-and-wait.sh -> wait-for-feedback.sh (just waits, no HTML piping)
- Remove wait-for-event.sh (used hanging tail -f)
- Workflow now: Write tool for HTML, wait-for-feedback.sh to block

Documentation rewrite:
- Broader "when to use" (UI, architecture, complex choices, spatial)
- Always ask user first before starting
- Scale fidelity to the question being asked
- Explain the question on each page
- Iterate before moving on - validate changes address feedback
- Use real content (Unsplash images) when it matters
2026-02-19 09:35:44 -08:00
Jesse Vincent
745ea8c71c feat: add show-and-wait.sh helper, fix race condition
- New show-and-wait.sh combines write + wait into one command
- Uses polling instead of tail -f (which hangs on macOS)
- Docs updated: start watcher BEFORE writing screen to avoid race
- Reduces terminal noise by consolidating operations
2026-02-19 09:35:44 -08:00
Jesse Vincent
d494f5a483 fix: session isolation and blocking wait for visual companion
- Each session gets unique temp directory (/tmp/brainstorm-{pid}-{timestamp})
- Server outputs screen_dir and screen_file in startup JSON
- stop-server.sh takes screen_dir arg and cleans up session directory
- Document blocking TaskOutput pattern: 10-min timeouts, retry up to 3x,
  then prompt user "let me know when you want to continue"
2026-02-19 09:35:44 -08:00
Jesse Vincent
ac3af07af0 feat: add visual companion for brainstorming skill
Adds browser-based mockup display to replace ASCII art during
brainstorming sessions. Key components:

- Frame template with OS-aware light/dark theming
- CSS helpers for options, cards, mockups, split views
- Server lifecycle scripts (start/stop with random high port)
- Event watcher using tail+grep for feedback loop
- Claude instructions for using the visual companion

The skill now asks users if they want browser mockups and only
runs in Claude Code environments.
2026-02-19 09:35:44 -08:00
Jesse Vincent
fd51e8a6b7 feat: add sendToClaude helper and wait-for-event tool
- Add sendToClaude() function to browser helper that shows confirmation
- Add wait-for-event.sh script for watching server output (tail -f | grep -m 1)
- Enables clean event-driven loop: background bash waits for event, completion triggers Claude's turn
2026-02-19 09:35:44 -08:00
Jesse Vincent
7c1ac86878 docs: add visual brainstorming implementation plan 2026-02-19 09:35:44 -08:00
Jesse Vincent
91fbffc994 fix: preserve original event type, use source field for wrapper 2026-02-19 09:35:44 -08:00
Jesse Vincent
3bcf5db8a1 fix: correct visual companion documentation issues 2026-02-19 09:35:44 -08:00
Jesse Vincent
ec46999224 feat: add visual companion to brainstorming skill 2026-02-19 09:35:44 -08:00
Jesse Vincent
e76ffffc1d test: add brainstorm server integration tests 2026-02-19 09:35:44 -08:00
Jesse Vincent
d20bdb8ec4 fix: ensure user-event type is preserved in WebSocket message output
The spread operator order was causing incoming event types to overwrite
the user-event type marker.
2026-02-19 09:35:44 -08:00
Jesse Vincent
fc10bf6586 feat: add browser helper library for event capture 2026-02-19 09:35:44 -08:00
Jesse Vincent
91debf6100 feat: add brainstorm server foundation
Create the initial server for the visual brainstorming companion:
- Express server with WebSocket support for browser communication
- File watcher (chokidar) to detect screen.html changes
- Auto-injects helper.js into served HTML for event capture
- Binds to localhost only (127.0.0.1) for security
- Outputs JSON events to stdout for Claude consumption
2026-02-19 09:35:44 -08:00
235 changed files with 4301 additions and 21617 deletions

View File

@@ -9,7 +9,7 @@
{
"name": "superpowers",
"description": "Core skills library for Claude Code: TDD, debugging, collaboration patterns, and proven techniques",
"version": "5.1.0",
"version": "4.3.0",
"source": "./",
"author": {
"name": "Jesse Vincent",

View File

@@ -1,7 +1,7 @@
{
"name": "superpowers",
"description": "Core skills library for Claude Code: TDD, debugging, collaboration patterns, and proven techniques",
"version": "5.1.0",
"version": "4.3.0",
"author": {
"name": "Jesse Vincent",
"email": "jesse@fsck.com"
@@ -9,12 +9,5 @@
"homepage": "https://github.com/obra/superpowers",
"repository": "https://github.com/obra/superpowers",
"license": "MIT",
"keywords": [
"skills",
"tdd",
"debugging",
"collaboration",
"best-practices",
"workflows"
]
"keywords": ["skills", "tdd", "debugging", "collaboration", "best-practices", "workflows"]
}

View File

@@ -1,47 +0,0 @@
{
"name": "superpowers",
"version": "5.1.0",
"description": "An agentic skills framework & software development methodology that works: planning, TDD, debugging, and collaboration workflows.",
"author": {
"name": "Jesse Vincent",
"email": "jesse@fsck.com",
"url": "https://github.com/obra"
},
"homepage": "https://github.com/obra/superpowers",
"repository": "https://github.com/obra/superpowers",
"license": "MIT",
"keywords": [
"brainstorming",
"subagent-driven-development",
"skills",
"planning",
"tdd",
"debugging",
"code-review",
"workflow"
],
"skills": "./skills/",
"interface": {
"displayName": "Superpowers",
"shortDescription": "Planning, TDD, debugging, and delivery workflows for coding agents",
"longDescription": "Use Superpowers to guide agent work through brainstorming, implementation planning, test-driven development, systematic debugging, parallel execution, code review, and finish-the-branch workflows.",
"developerName": "Jesse Vincent",
"category": "Coding",
"capabilities": [
"Interactive",
"Read",
"Write"
],
"defaultPrompt": [
"I've got an idea for something I'd like to build.",
"Let's add a feature to this project."
],
"websiteURL": "https://github.com/obra/superpowers",
"privacyPolicyURL": "https://docs.github.com/en/site-policy/privacy-policies/github-general-privacy-statement",
"termsOfServiceURL": "https://docs.github.com/en/site-policy/github-terms/github-terms-of-service",
"brandColor": "#F59E0B",
"composerIcon": "./assets/superpowers-small.svg",
"logo": "./assets/app-icon.png",
"screenshots": []
}
}

67
.codex/INSTALL.md Normal file
View File

@@ -0,0 +1,67 @@
# Installing Superpowers for Codex
Enable superpowers skills in Codex via native skill discovery. Just clone and symlink.
## Prerequisites
- Git
## Installation
1. **Clone the superpowers repository:**
```bash
git clone https://github.com/obra/superpowers.git ~/.codex/superpowers
```
2. **Create the skills symlink:**
```bash
mkdir -p ~/.agents/skills
ln -s ~/.codex/superpowers/skills ~/.agents/skills/superpowers
```
**Windows (PowerShell):**
```powershell
New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.agents\skills"
cmd /c mklink /J "$env:USERPROFILE\.agents\skills\superpowers" "$env:USERPROFILE\.codex\superpowers\skills"
```
3. **Restart Codex** (quit and relaunch the CLI) to discover the skills.
## Migrating from old bootstrap
If you installed superpowers before native skill discovery, you need to:
1. **Update the repo:**
```bash
cd ~/.codex/superpowers && git pull
```
2. **Create the skills symlink** (step 2 above) — this is the new discovery mechanism.
3. **Remove the old bootstrap block** from `~/.codex/AGENTS.md` — any block referencing `superpowers-codex bootstrap` is no longer needed.
4. **Restart Codex.**
## Verify
```bash
ls -la ~/.agents/skills/superpowers
```
You should see a symlink (or junction on Windows) pointing to your superpowers skills directory.
## Updating
```bash
cd ~/.codex/superpowers && git pull
```
Skills update instantly through the symlink.
## Uninstalling
```bash
rm ~/.agents/skills/superpowers
```
Optionally delete the clone: `rm -rf ~/.codex/superpowers`.

View File

@@ -2,7 +2,7 @@
"name": "superpowers",
"displayName": "Superpowers",
"description": "Core skills library: TDD, debugging, collaboration patterns, and proven techniques",
"version": "5.1.0",
"version": "4.3.0",
"author": {
"name": "Jesse Vincent",
"email": "jesse@fsck.com"
@@ -10,14 +10,9 @@
"homepage": "https://github.com/obra/superpowers",
"repository": "https://github.com/obra/superpowers",
"license": "MIT",
"keywords": [
"skills",
"tdd",
"debugging",
"collaboration",
"best-practices",
"workflows"
],
"keywords": ["skills", "tdd", "debugging", "collaboration", "best-practices", "workflows"],
"skills": "./skills/",
"hooks": "./hooks/hooks-cursor.json"
"agents": "./agents/",
"commands": "./commands/",
"hooks": "./hooks/hooks.json"
}

View File

@@ -1,52 +0,0 @@
---
name: Bug Report
about: Something isn't working as expected
labels: bug
---
<!--
BEFORE FILING: Search open AND closed issues. The Windows SessionStart
hook alone has been reported 29 times. If your issue already exists,
add a comment or reaction to the existing one instead.
-->
- [ ] I searched existing issues and this is not a duplicate
## Environment
| Field | Value |
|-------|-------|
| Superpowers version | |
| Harness (Claude Code, Cursor, etc.) | |
| Harness version | |
| Model | |
| OS + shell | |
## Is this a Superpowers issue or a platform issue?
<!-- Superpowers is a plugin. Some reported "bugs" are actually issues
in the underlying platform or model. If you're not sure, try
reproducing without Superpowers installed.
If the problem persists without Superpowers, file the issue with
your platform instead. -->
- [ ] I confirmed this issue does not occur without Superpowers installed
## What happened?
<!-- Be specific. "It doesn't work" is not a bug report. -->
## Steps to reproduce
1.
2.
3.
## Expected behavior
<!-- What should have happened? -->
## Actual behavior
<!-- What happened instead? -->
## Debug log or conversation transcript
<!-- A debug log or conversation transcript showing the issue is the
single most helpful thing you can include. Without one, we're
guessing. Screenshots of error output are also useful. -->

View File

@@ -1,5 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: Questions & Help
url: https://discord.gg/35wsABTejz
about: For usage questions, troubleshooting help, and general discussion, please visit our Discord instead of opening an issue.

View File

@@ -1,34 +0,0 @@
---
name: Feature Request
about: Propose a change or addition to Superpowers
labels: enhancement
---
<!--
BEFORE FILING: Search open AND closed issues. Many features have been
requested before — some were implemented differently, some are in
progress, and some were intentionally declined.
-->
- [ ] I searched existing issues and this has not been proposed before
## What problem does this solve?
<!-- Describe the problem from your own experience. What were you doing,
what went wrong or was missing, and why did it matter?
"It would be cool if..." is not a problem statement. -->
## Proposed solution
<!-- What specifically do you want to happen? Be concrete. -->
## What alternatives did you consider?
<!-- What other approaches could solve the same problem? Why is your
proposal better? -->
## Is this appropriate for core Superpowers?
<!-- Would this benefit someone working on a completely different kind
of project? If this is specific to your domain, workflow, or a
third-party tool, it may belong as its own plugin instead. -->
## Context
<!-- Optional: version info, harness, model, workflow where you hit this. -->

View File

@@ -1,23 +0,0 @@
---
name: IDE / Platform Support Request
about: Request support for a new IDE, editor, or AI coding tool
labels: platform-support
---
<!--
BEFORE FILING: Search existing issues — your IDE may already be
requested or discussed.
-->
- [ ] I searched existing issues for this IDE/platform
## Which IDE or platform?
<!-- Name and link -->
## Does this tool have a plugin or extension system?
<!-- If yes, link to the docs. If no, explain how third-party
integrations typically work with this tool. -->
## Have you tried manual installation?
<!-- Many tools work with Superpowers through manual setup even without
official support. Did you try? What happened? -->

View File

@@ -1,126 +0,0 @@
<!--
BEFORE SUBMITTING: Read every word of this template. PRs that leave
sections blank, contain multiple unrelated changes, or show no evidence
of human involvement will be closed without review.
-->
## What problem are you trying to solve?
<!-- Describe the specific problem you encountered. If this was a session
issue, include: what you were doing, what went wrong, the model's
exact failure mode, and ideally a transcript or session log.
"Improving" something is not a problem statement. What broke? What
failed? What was the user experience that motivated this? -->
## What does this PR change?
<!-- 1-3 sentences. What, not why — the "why" belongs above. -->
## Is this change appropriate for the core library?
<!-- Superpowers core contains general-purpose skills and infrastructure
that benefit all users. Ask yourself:
- Would this be useful to someone working on a completely different
kind of project than yours?
- Is this project-specific, team-specific, or tool-specific?
- Does this integrate or promote a third-party service?
If your change is a new skill for a specific domain, workflow tool,
or third-party integration, it belongs in its own plugin — not here.
See the plugin development docs for how to publish it separately. -->
## What alternatives did you consider?
<!-- What other approaches did you try or evaluate before landing on this
one? Why were they worse? If you didn't consider alternatives, say so
— but know that's a red flag. -->
## Does this PR contain multiple unrelated changes?
<!-- If yes: stop. Split it into separate PRs. Bundled PRs will be closed.
If you believe the changes are related, explain the dependency. -->
## Existing PRs
- [ ] I have reviewed all open AND closed PRs for duplicates or prior art
- Related PRs: <!-- #number, #number, or "none found" -->
<!-- If a related closed PR exists, explain what's different about your
approach and why it should succeed where the other didn't. -->
## Environment tested
| Harness (e.g. Claude Code, Cursor) | Harness version | Model | Model version/ID |
|-------------------------------------|-----------------|-------|------------------|
| | | | |
## New harness support (required if this PR adds a new harness)
<!-- If this PR adds support for a new harness (IDE, CLI tool, agent
runner), you MUST include a session transcript proving the
integration actually works.
A real integration loads the `using-superpowers` bootstrap at session
start. The bootstrap is what causes skills to auto-trigger. Without
it, the skills are dead weight — present on disk but never invoked
at the right moments.
ACCEPTANCE TEST: Open a clean session in the new harness and send
exactly this user message:
Let's make a react todo list
A working integration auto-triggers the `brainstorming` skill before
any code is written. Paste the complete transcript below.
These are NOT real integrations and PRs that ship them will be closed:
- Manually copying skill files into the harness
- Wrapping with `npx skills` or similar at-runtime shims
- Anything that requires the user to opt in to skills per-session
- Anything where brainstorming does not auto-trigger on the test above
If you are not sure whether your integration loads the bootstrap at
session start, it does not.
-->
<details>
<summary>Clean-session transcript for "Let's make a react todo list"</summary>
```
paste the complete transcript here
```
</details>
## Evaluation
- What was the initial prompt you (or your human partner) used to start
the session that led to this change?
- How many eval sessions did you run AFTER making the change?
- How did outcomes change compared to before the change?
<!-- "It works" is not evaluation. Describe the before/after difference
you observed across multiple sessions. -->
## Rigor
- [ ] If this is a skills change: I used `superpowers:writing-skills` and
completed adversarial pressure testing (paste results below)
- [ ] This change was tested adversarially, not just on the happy path
- [ ] I did not modify carefully-tuned content (Red Flags table,
rationalizations, "human partner" language) without extensive evals
showing the change is an improvement
<!-- If you changed wording in skills that shape agent behavior, show your
eval methodology and results. These are not prose — they are code. -->
## Human review
- [ ] A human has reviewed the COMPLETE proposed diff before submission
<!--
STOP. If the checkbox above is not checked, do not submit this PR.
PRs will be closed without review if they:
- Show no evidence of human involvement
- Contain multiple unrelated changes
- Promote or integrate third-party services or tools
- Submit project-specific or personal configuration as core changes
- Leave required sections blank or use placeholder text
- Modify behavior-shaping content without eval evidence
-->

7
.gitignore vendored
View File

@@ -1,13 +1,6 @@
.worktrees/
.private-journal/
.claude/
.DS_Store
node_modules/
inspo
triage/
# Eval harness — drill ships its own gitignore at evals/.gitignore;
# these are belt-and-suspenders entries for tools that don't recurse.
evals/results/
evals/.venv/
evals/.env

View File

@@ -3,103 +3,112 @@
## Prerequisites
- [OpenCode.ai](https://opencode.ai) installed
- Git installed
## Installation
## Installation Steps
Add superpowers to the `plugin` array in your `opencode.json` (global or project-level):
```json
{
"plugin": ["superpowers@git+https://github.com/obra/superpowers.git"]
}
```
Restart OpenCode. The plugin installs through OpenCode's plugin manager and
registers all skills.
Verify by asking: "Tell me about your superpowers"
OpenCode uses its own plugin install. If you also use Claude Code, Codex, or
another harness, install Superpowers separately for each one.
## Migrating from the old symlink-based install
If you previously installed superpowers using `git clone` and symlinks, remove the old setup:
### 1. Clone Superpowers
```bash
# Remove old symlinks
rm -f ~/.config/opencode/plugins/superpowers.js
rm -rf ~/.config/opencode/skills/superpowers
# Optionally remove the cloned repo
rm -rf ~/.config/opencode/superpowers
# Remove skills.paths from opencode.json if you added one for superpowers
git clone https://github.com/obra/superpowers.git ~/.config/opencode/superpowers
```
Then follow the installation steps above.
### 2. Register the Plugin
Create a symlink so OpenCode discovers the plugin:
```bash
mkdir -p ~/.config/opencode/plugins
rm -f ~/.config/opencode/plugins/superpowers.js
ln -s ~/.config/opencode/superpowers/.opencode/plugins/superpowers.js ~/.config/opencode/plugins/superpowers.js
```
### 3. Symlink Skills
Create a symlink so OpenCode's native skill tool discovers superpowers skills:
```bash
mkdir -p ~/.config/opencode/skills
rm -rf ~/.config/opencode/skills/superpowers
ln -s ~/.config/opencode/superpowers/skills ~/.config/opencode/skills/superpowers
```
### 4. Restart OpenCode
Restart OpenCode. The plugin will automatically inject superpowers context.
Verify by asking: "do you have superpowers?"
## Usage
Use OpenCode's native `skill` tool:
### Finding Skills
Use OpenCode's native `skill` tool to list available skills:
```
use skill tool to list skills
```
### Loading a Skill
Use OpenCode's native `skill` tool to load a specific skill:
```
use skill tool to load superpowers/brainstorming
```
### Personal Skills
Create your own skills in `~/.config/opencode/skills/`:
```bash
mkdir -p ~/.config/opencode/skills/my-skill
```
Create `~/.config/opencode/skills/my-skill/SKILL.md`:
```markdown
---
name: my-skill
description: Use when [condition] - [what it does]
---
# My Skill
[Your skill content here]
```
### Project Skills
Create project-specific skills in `.opencode/skills/` within your project.
**Skill Priority:** Project skills > Personal skills > Superpowers skills
## Updating
OpenCode installs Superpowers through a git-backed package spec. Some OpenCode
and Bun versions pin that resolved git dependency in a lockfile or cache, so a
restart may not pick up the newest Superpowers commit. If updates do not appear,
clear OpenCode's package cache or reinstall the plugin.
To pin a specific version:
```json
{
"plugin": ["superpowers@git+https://github.com/obra/superpowers.git#v5.0.3"]
}
```bash
cd ~/.config/opencode/superpowers
git pull
```
## Troubleshooting
### Plugin not loading
1. Check logs: `opencode run --print-logs "hello" 2>&1 | grep -i superpowers`
2. Verify the plugin line in your `opencode.json`
3. Make sure you're running a recent version of OpenCode
### Windows install issues
Some Windows OpenCode builds have upstream installer issues with git-backed
plugin specs, including cache paths for `git+https` URLs and Bun not finding
`git.exe` even when it works in a normal terminal. If OpenCode cannot install
the plugin, try installing with system npm and pointing OpenCode at the local
package:
```powershell
npm install superpowers@git+https://github.com/obra/superpowers.git --prefix "$HOME\.config\opencode"
```
Then use the installed package path in `opencode.json`:
```json
{
"plugin": ["~/.config/opencode/node_modules/superpowers"]
}
```
1. Check plugin symlink: `ls -l ~/.config/opencode/plugins/superpowers.js`
2. Check source exists: `ls ~/.config/opencode/superpowers/.opencode/plugins/superpowers.js`
3. Check OpenCode logs for errors
### Skills not found
1. Use `skill` tool to list what's discovered
2. Check that the plugin is loading (see above)
1. Check skills symlink: `ls -l ~/.config/opencode/skills/superpowers`
2. Verify it points to: `~/.config/opencode/superpowers/skills`
3. Use `skill` tool to list what's discovered
### Tool mapping
When skills reference Claude Code tools:
- `TodoWrite``todowrite`
- `TodoWrite``update_plan`
- `Task` with subagents → `@mention` syntax
- `Skill` tool → OpenCode's native `skill` tool
- File operations → your native tools

View File

@@ -2,7 +2,7 @@
* Superpowers plugin for OpenCode.ai
*
* Injects superpowers bootstrap context via system prompt transform.
* Auto-registers skills directory via config hook (no symlinks needed).
* Skills are discovered via OpenCode's native skill tool from symlinked directory.
*/
import path from 'path';
@@ -46,43 +46,33 @@ const normalizePath = (p, homeDir) => {
return path.resolve(normalized);
};
// Module-level cache for bootstrap content.
// The SKILL.md file does not change during a session, so reading + parsing it
// once eliminates redundant fs.existsSync + fs.readFileSync + regex work on
// every agent step. See #1202 for the full analysis.
let _bootstrapCache = undefined; // undefined = not yet loaded, null = file missing
export const SuperpowersPlugin = async ({ client, directory }) => {
const homeDir = os.homedir();
const superpowersSkillsDir = path.resolve(__dirname, '../../skills');
const envConfigDir = normalizePath(process.env.OPENCODE_CONFIG_DIR, homeDir);
const configDir = envConfigDir || path.join(homeDir, '.config/opencode');
// Helper to generate bootstrap content (cached after first call)
// Helper to generate bootstrap content
const getBootstrapContent = () => {
// Return cached result on subsequent calls
if (_bootstrapCache !== undefined) return _bootstrapCache;
// Try to load using-superpowers skill
const skillPath = path.join(superpowersSkillsDir, 'using-superpowers', 'SKILL.md');
if (!fs.existsSync(skillPath)) {
_bootstrapCache = null;
return null;
}
if (!fs.existsSync(skillPath)) return null;
const fullContent = fs.readFileSync(skillPath, 'utf8');
const { content } = extractAndStripFrontmatter(fullContent);
const toolMapping = `**Tool Mapping for OpenCode:**
When skills reference tools you don't have, substitute OpenCode equivalents:
- \`TodoWrite\`\`todowrite\`
- \`TodoWrite\`\`update_plan\`
- \`Task\` tool with subagents → Use OpenCode's subagent system (@mention)
- \`Skill\` tool → OpenCode's native \`skill\` tool
- \`Read\`, \`Write\`, \`Edit\`, \`Bash\` → Your native tools
**Skills location:**
Superpowers skills are in \`${configDir}/skills/superpowers/\`
Use OpenCode's native \`skill\` tool to list and load skills.`;
_bootstrapCache = `<EXTREMELY_IMPORTANT>
return `<EXTREMELY_IMPORTANT>
You have superpowers.
**IMPORTANT: The using-superpowers skill content is included below. It is ALREADY LOADED - you are currently following it. Do NOT use the skill tool to load "using-superpowers" again - that would be redundant.**
@@ -91,45 +81,15 @@ ${content}
${toolMapping}
</EXTREMELY_IMPORTANT>`;
return _bootstrapCache;
};
return {
// Inject skills path into live config so OpenCode discovers superpowers skills
// without requiring manual symlinks or config file edits.
// This works because Config.get() returns a cached singleton — modifications
// here are visible when skills are lazily discovered later.
config: async (config) => {
config.skills = config.skills || {};
config.skills.paths = config.skills.paths || [];
if (!config.skills.paths.includes(superpowersSkillsDir)) {
config.skills.paths.push(superpowersSkillsDir);
}
},
// Inject bootstrap into the first user message of each session.
// Using a user message instead of a system message avoids:
// 1. Token bloat from system messages repeated every turn (#750)
// 2. Multiple system messages breaking Qwen and other models (#894)
//
// The hook fires on every agent step (not just every turn) because
// opencode's prompt.ts reloads messages from DB each step. Fresh message
// arrays may need injection again, so getBootstrapContent() must not do
// repeated disk work.
'experimental.chat.messages.transform': async (_input, output) => {
// Use system prompt transform to inject bootstrap (fixes #226 agent reset bug)
'experimental.chat.system.transform': async (_input, output) => {
const bootstrap = getBootstrapContent();
if (!bootstrap || !output.messages.length) return;
const firstUser = output.messages.find(m => m.info.role === 'user');
if (!firstUser || !firstUser.parts.length) return;
// Guard: skip if first user message already contains bootstrap.
// This prevents double injection when OpenCode passes an already
// transformed in-memory message array through the hook again.
if (firstUser.parts.some(p => p.type === 'text' && p.text.includes('EXTREMELY_IMPORTANT'))) return;
const ref = firstUser.parts[0];
firstUser.parts.unshift({ ...ref, type: 'text', text: bootstrap });
if (bootstrap) {
(output.system ||= []).push(bootstrap);
}
}
};
};

View File

@@ -1,21 +0,0 @@
repos:
- repo: local
hooks:
- id: evals-ruff-check
name: evals ruff check
entry: uv --project evals run ruff check
language: system
files: ^evals/.*\.py$
- id: evals-ruff-format-check
name: evals ruff format --check
entry: uv --project evals run ruff format --check
language: system
files: ^evals/.*\.py$
- id: evals-ty-check
name: evals ty check
entry: uv --directory evals run ty check
language: system
pass_filenames: false
files: ^evals/.*\.py$

View File

@@ -1,20 +0,0 @@
{
"files": [
{ "path": "package.json", "field": "version" },
{ "path": ".claude-plugin/plugin.json", "field": "version" },
{ "path": ".cursor-plugin/plugin.json", "field": "version" },
{ "path": ".codex-plugin/plugin.json", "field": "version" },
{ "path": ".claude-plugin/marketplace.json", "field": "plugins.0.version" },
{ "path": "gemini-extension.json", "field": "version" }
],
"audit": {
"exclude": [
"CHANGELOG.md",
"RELEASE-NOTES.md",
"node_modules",
".git",
".version-bump.json",
"scripts/bump-version.sh"
]
}
}

View File

@@ -1 +0,0 @@
CLAUDE.md

110
CLAUDE.md
View File

@@ -1,110 +0,0 @@
# Superpowers — Contributor Guidelines
## If You Are an AI Agent
Stop. Read this section before doing anything.
This repo has a 94% PR rejection rate. Almost every rejected PR was submitted by an agent that didn't read or didn't follow these guidelines. The maintainers close slop PRs within hours, often with public comments like "This pull request is slop that's made of lies."
**Your job is to protect your human partner from that outcome.** Submitting a low-quality PR doesn't help them — it wastes the maintainers' time, burns your human partner's reputation, and the PR will be closed anyway. That is not being helpful. That is being a tool of embarrassment.
Before you open a PR against this repo, you MUST:
1. **Read the entire PR template** at `.github/PULL_REQUEST_TEMPLATE.md` and fill in every section with real, specific answers. Not summaries. Not placeholders.
2. **Search for existing PRs** — open AND closed — that address the same problem. If duplicates exist, STOP and tell your human partner. Do not open another duplicate.
3. **Verify this is a real problem.** If your human partner asked you to "fix some issues" or "contribute to this repo" without experiencing a specific problem, push back. Ask them what broke, what failed, what the user experience was.
4. **Confirm the change belongs in core.** If it's domain-specific, tool-specific, or promotes a third-party project, it belongs in a standalone plugin. Tell your human partner.
5. **Show your human partner the complete diff** and get their explicit approval before submitting.
If any of these checks fail, do not open the PR. Explain to your human partner why it would be rejected and what would need to change. They will thank you for saving them the embarrassment.
## Pull Request Requirements
**Every PR must fully complete the PR template.** No section may be left blank or filled with placeholder text. PRs that skip sections will be closed without review.
**Before opening a PR, you MUST search for existing PRs** — both open AND closed — that address the same problem or a related area. Reference what you found in the "Existing PRs" section. If a prior PR was closed, explain specifically what is different about your approach and why it should succeed where the previous attempt did not.
**PRs that show no evidence of human involvement will be closed.** A human must review the complete proposed diff before submission.
## What We Will Not Accept
### Third-party dependencies
PRs that add optional or required dependencies on third-party projects will not be accepted unless they are adding support for a new harness (e.g., a new IDE or CLI tool). Superpowers is a zero-dependency plugin by design. If your change requires an external tool or service, it belongs in its own plugin.
### "Compliance" changes to skills
Our internal skill philosophy differs from Anthropic's published guidance on writing skills. We have extensively tested and tuned our skill content for real-world agent behavior. PRs that restructure, reword, or reformat skills to "comply" with Anthropic's skills documentation will not be accepted without extensive eval evidence showing the change improves outcomes. The bar for modifying behavior-shaping content is very high.
### Project-specific or personal configuration
Skills, hooks, or configuration that only benefit a specific project, team, domain, or workflow do not belong in core. Publish these as a separate plugin.
### Bulk or spray-and-pray PRs
Do not trawl the issue tracker and open PRs for multiple issues in a single session. Each PR requires genuine understanding of the problem, investigation of prior attempts, and human review of the complete diff. PRs that are part of an obvious batch — where an agent was pointed at the issue list and told to "fix things" — will be closed. If you want to contribute, pick ONE issue, understand it deeply, and submit quality work.
### Speculative or theoretical fixes
Every PR must solve a real problem that someone actually experienced. "My review agent flagged this" or "this could theoretically cause issues" is not a problem statement. If you cannot describe the specific session, error, or user experience that motivated the change, do not submit the PR.
### Domain-specific skills
Superpowers core contains general-purpose skills that benefit all users regardless of their project. Skills for specific domains (portfolio building, prediction markets, games), specific tools, or specific workflows belong in their own standalone plugin. Ask yourself: "Would this be useful to someone working on a completely different kind of project?" If not, publish it separately.
### Fork-specific changes
If you maintain a fork with customizations, do not open PRs to sync your fork or push fork-specific changes upstream. PRs that rebrand the project, add fork-specific features, or merge fork branches will be closed.
### Fabricated content
PRs containing invented claims, fabricated problem descriptions, or hallucinated functionality will be closed immediately. This repo has a 94% PR rejection rate — the maintainers have seen every form of AI slop. They will notice.
### Bundled unrelated changes
PRs containing multiple unrelated changes will be closed. Split them into separate PRs.
## New Harness Support
If your PR adds support for a new harness (IDE, CLI tool, agent runner), you MUST include a session transcript proving the integration works end-to-end.
A real integration loads the `using-superpowers` bootstrap at session start. The bootstrap is what causes skills to auto-trigger at the right moments. Without it, the skills are dead weight — present on disk but never invoked.
**The acceptance test.** Open a clean session in the new harness and send exactly this user message:
> Let's make a react todo list
A working integration auto-triggers the `brainstorming` skill before any code is written. Paste the complete transcript in the PR.
**These are not real integrations and will be closed:**
- Manually copying skill files into the harness
- Wrapping with `npx skills` or similar at-runtime shims
- Anything that requires the user to opt in to skills per-session
- Anything where `brainstorming` does not auto-trigger on the acceptance test above
If you are not sure whether your integration loads the bootstrap at session start, it does not.
## Skill Changes Require Evaluation
Skills are not prose — they are code that shapes agent behavior. If you modify skill content:
- Use `superpowers:writing-skills` to develop and test changes
- Run adversarial pressure testing across multiple sessions
- Show before/after eval results in your PR
- Do not modify carefully-tuned content (Red Flags tables, rationalization lists, "human partner" language) without evidence the change is an improvement
## Eval harness
Skill-behavior evals live at `evals/` — see `evals/README.md`. Drill (the harness) drives real tmux sessions of Claude Code / Codex / Gemini CLI and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`.
## Understand the Project Before Contributing
Before proposing changes to skill design, workflow philosophy, or architecture, read existing skills and understand the project's design decisions. Superpowers has its own tested philosophy about skill design, agent behavior shaping, and terminology (e.g., "your human partner" is deliberate, not interchangeable with "the user"). Changes that rewrite the project's voice or restructure its approach without understanding why it exists will be rejected.
## General
- Read `.github/PULL_REQUEST_TEMPLATE.md` before submitting
- One problem per PR
- Test on at least one harness and report results in the environment table
- Describe the problem you solved, not just what you changed

View File

@@ -1,128 +0,0 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
jesse@primeradiant.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.

View File

@@ -1,2 +0,0 @@
@./skills/using-superpowers/SKILL.md
@./skills/using-superpowers/references/gemini-tools.md

162
README.md
View File

@@ -1,10 +1,6 @@
# Superpowers
Superpowers is a complete software development methodology for your coding agents, built on top of a set of composable skills and some initial instructions that make sure your agent uses them.
## Quickstart
Give your agent Superpowers: [Claude Code](#claude-code), [Codex CLI](#codex-cli), [Codex App](#codex-app), [Factory Droid](#factory-droid), [Gemini CLI](#gemini-cli), [OpenCode](#opencode), [Cursor](#cursor), [GitHub Copilot CLI](#github-copilot-cli).
Superpowers is a complete software development workflow for your coding agents, built on top of a set of composable "skills" and some initial instructions that make sure your agent uses them.
## How it works
@@ -25,131 +21,59 @@ If Superpowers has helped you do stuff that makes money and you are so inclined,
Thanks!
\- Jesse
- Jesse
## Installation
Installation differs by harness. If you use more than one, install Superpowers separately for each one.
**Note:** Installation differs by platform. Claude Code or Cursor have built-in plugin marketplaces. Codex and OpenCode require manual setup.
### Claude Code
Superpowers is available via the [official Claude plugin marketplace](https://claude.com/plugins/superpowers)
### Claude Code (via Plugin Marketplace)
#### Official Marketplace
In Claude Code, register the marketplace first:
- Install the plugin from Anthropic's official marketplace:
```bash
/plugin marketplace add obra/superpowers-marketplace
```
```bash
/plugin install superpowers@claude-plugins-official
```
Then install the plugin from this marketplace:
#### Superpowers Marketplace
```bash
/plugin install superpowers@superpowers-marketplace
```
The Superpowers marketplace provides Superpowers and some other related plugins for Claude Code.
### Cursor (via Plugin Marketplace)
- Register the marketplace:
In Cursor Agent chat, install from marketplace:
```bash
/plugin marketplace add obra/superpowers-marketplace
```
```text
/plugin-add superpowers
```
- Install the plugin from this marketplace:
### Codex
```bash
/plugin install superpowers@superpowers-marketplace
```
Tell Codex:
### Codex CLI
```
Fetch and follow instructions from https://raw.githubusercontent.com/obra/superpowers/refs/heads/main/.codex/INSTALL.md
```
Superpowers is available via the [official Codex plugin marketplace](https://github.com/openai/plugins).
- Open the plugin search interface:
```bash
/plugins
```
- Search for Superpowers:
```bash
superpowers
```
- Select `Install Plugin`.
### Codex App
Superpowers is available via the [official Codex plugin marketplace](https://github.com/openai/plugins).
- In the Codex app, click on Plugins in the sidebar.
- You should see `Superpowers` in the Coding section.
- Click the `+` next to Superpowers and follow the prompts.
### Factory Droid
- Register the marketplace:
```bash
droid plugin marketplace add https://github.com/obra/superpowers
```
- Install the plugin:
```bash
droid plugin install superpowers@superpowers
```
### Gemini CLI
- Install the extension:
```bash
gemini extensions install https://github.com/obra/superpowers
```
- Update later:
```bash
gemini extensions update superpowers
```
**Detailed docs:** [docs/README.codex.md](docs/README.codex.md)
### OpenCode
OpenCode uses its own plugin install; install Superpowers separately even if you
already use it in another harness.
Tell OpenCode:
- Tell OpenCode:
```
Fetch and follow instructions from https://raw.githubusercontent.com/obra/superpowers/refs/heads/main/.opencode/INSTALL.md
```
```
Fetch and follow instructions from https://raw.githubusercontent.com/obra/superpowers/refs/heads/main/.opencode/INSTALL.md
```
**Detailed docs:** [docs/README.opencode.md](docs/README.opencode.md)
- Detailed docs: [docs/README.opencode.md](docs/README.opencode.md)
### Verify Installation
### Cursor
- In Cursor Agent chat, install from marketplace:
```text
/add-plugin superpowers
```
- Or search for "superpowers" in the plugin marketplace.
### GitHub Copilot CLI
- Register the marketplace:
```bash
copilot plugin marketplace add obra/superpowers-marketplace
```
- Install the plugin:
```bash
copilot plugin install superpowers@superpowers-marketplace
```
Start a new session in your chosen platform and ask for something that should trigger a skill (for example, "help me plan this feature" or "let's debug this issue"). The agent should automatically invoke the relevant superpowers skill.
## The Basic Workflow
@@ -202,34 +126,32 @@ already use it in another harness.
- **Complexity reduction** - Simplicity as primary goal
- **Evidence over claims** - Verify before declaring success
Read [the original release announcement](https://blog.fsck.com/2025/10/09/superpowers/).
Read more: [Superpowers for Claude Code](https://blog.fsck.com/2025/10/09/superpowers/)
## Contributing
The general contribution process for Superpowers is below. Keep in mind that we don't generally accept contributions of new skills and that any updates to skills must work across all of the coding agents we support.
Skills live directly in this repository. To contribute:
1. Fork the repository
2. Switch to the 'dev' branch
3. Create a branch for your work
4. Follow the `writing-skills` skill for creating and testing new and modified skills
5. Submit a PR, being sure to fill in the pull request template.
Skill-behavior tests use the eval harness at `evals/`. See `evals/README.md` for setup. Plugin-infrastructure tests live at `tests/` and run via the relevant `run-*.sh` or `npm test`.
2. Create a branch for your skill
3. Follow the `writing-skills` skill for creating and testing new skills
4. Submit a PR
See `skills/writing-skills/SKILL.md` for the complete guide.
## Updating
Superpowers updates are somewhat coding-agent dependent, but are often automatic.
Skills update automatically when you update the plugin:
```bash
/plugin update superpowers
```
## License
MIT License - see LICENSE file for details
## Community
## Support
Superpowers is built by [Jesse Vincent](https://blog.fsck.com) and the rest of the folks at [Prime Radiant](https://primeradiant.com).
- **Discord**: [Join us](https://discord.gg/35wsABTejz) for community support, questions, and sharing what you're building with Superpowers
- **Issues**: https://github.com/obra/superpowers/issues
- **Release announcements**: [Sign up](https://primeradiant.com/superpowers/) to get notified about new versions
- **Marketplace**: https://github.com/obra/superpowers-marketplace

View File

@@ -1,406 +1,93 @@
# Superpowers Release Notes
## v5.1.0 (2026-04-30)
### Removals
- **Legacy slash commands removed** — `/brainstorm`, `/execute-plan`, and `/write-plan` are gone. They were deprecated stubs that did nothing but tell the user to invoke the corresponding skill. Invoke `superpowers:brainstorming`, `superpowers:executing-plans`, and `superpowers:writing-plans` directly instead. (#1188)
- **`superpowers:code-reviewer` named agent removed** — the agent was the plugin's only named agent and was used by exactly two skills, while every other reviewer/implementer subagent in the repo dispatches `general-purpose` with a prompt template alongside its skill. The agent's persona and checklist have been merged into `skills/requesting-code-review/code-reviewer.md` as a self-contained Task-dispatch template. Anyone dispatching `Task (superpowers:code-reviewer)` should switch to `Task (general-purpose)` with the prompt template instead. (PR #1299)
- **Integration sections removed from skills** — these were a legacy of the time before agents had native skills systems and didn't help with steering.
### Worktree Skills Rewrite
`using-git-worktrees` and `finishing-a-development-branch` now detect when the agent is already running inside an isolated worktree and prefer the harness's native worktree controls before falling back to `git worktree`. Behavior was TDD-validated and cross-platform-checked across five harnesses. (PRI-974, PR #1121)
- **Environment detection** — both skills check `GIT_DIR != GIT_COMMON` before doing anything; if already in a linked worktree, creation is skipped entirely. A submodule guard prevents false detection.
- **Consent before creating worktrees** — `using-git-worktrees` no longer creates worktrees implicitly; the skill asks the user first. Fixes #991 (subagent-driven-development was auto-creating worktrees without consent).
- **Native tool preference (Step 1a)** — when the harness exposes its own worktree tool (e.g. Codex), the skill defers to it. The user's stated preference is respected when expressed.
- **Provenance-based cleanup** — `finishing-a-development-branch` only cleans up worktrees inside `.worktrees/` (created by superpowers); anything outside is left alone. Fixes #940 (Option 2 was incorrectly cleaning up worktrees), #999 (merge-then-remove ordering), and #238 (`cd` to repo root before `git worktree remove`).
- **Detached HEAD handling** — the finishing menu collapses to two options when there is no branch to merge from.
- **Hardcoded `/Users/jesse` paths** in skill examples replaced with generic placeholders. (#858, PR #1122)
### Contributor Guidelines for AI Agents
Two new sections at the top of `CLAUDE.md` (symlinked to `AGENTS.md`) speak directly to AI agents. An audit of the last 100 closed PRs against this repo showed a 94% rejection rate driven by AI-generated slop: agents that didn't read the PR template, opened duplicates, fabricated problem descriptions, or pushed fork- or domain-specific changes upstream.
- **Pre-submission checklist** — read the PR template, search for existing PRs, verify a real problem exists, confirm the change belongs in core, and show the human partner the complete diff before submitting.
- **What we will not accept** — third-party dependencies, "compliance" rewrites of skill content, project-specific configuration, bulk PRs, speculative fixes, domain-specific skills, fork-specific changes, fabricated content, and bundled unrelated changes.
- **New harness PRs require a session transcript** — most past new-harness integrations copied skill files or wrapped with `npx skills` instead of loading the `using-superpowers` bootstrap at session start. The acceptance test ("Let's make a react todo list" must auto-trigger `brainstorming` in a clean session) and a complete transcript are now required.
### Codex Plugin Mirror Tooling
New `sync-to-codex-plugin` script mirrors superpowers into the OpenAI Codex plugin marketplace as `prime-radiant-inc/openai-codex-plugins`. Path/user-agnostic so any team member can run it. (PR #1165)
- Clones the fork fresh into a temp directory per run, regenerates overlays inline, and opens a PR; auto-detects upstream from the script's own location and preflights `rsync`/`git`/`gh auth`/`python3`.
- `--bootstrap` flag for first-time setup; `EXCLUDES` patterns anchored to source root; `assets/` excluded.
- Mirrors `CODE_OF_CONDUCT.md`; drops the `agents/openai.yaml` overlay.
- Seeds `interface.defaultPrompt` in the mirrored `plugin.json`. (PR #1180 by @arittr)
- Codex plugin files are committed to the source repo so the sync script uses canonical versions; Codex marketplace metadata is preserved.
### OpenCode
- **Bootstrap content cached at module level** — `getBootstrapContent()` was calling `fs.existsSync` + `fs.readFileSync` + frontmatter regex on every agent step (the `experimental.chat.messages.transform` hook fires on every step in OpenCode's agent loop). Now read once, cached for the session lifetime, with a null sentinel for the missing-file case. 15 regression tests cover cache behavior, fs call counts, the injection guard, the missing-file sentinel, and cache reset. (Fixes #1202)
- **Integration tests modernized**.
- **Install caveats clarified** in the README.
### Code Review Consolidation
`requesting-code-review` is now self-contained: the persona, checklist, and dispatch template live in `skills/requesting-code-review/code-reviewer.md` and the skill dispatches `Task (general-purpose)` directly. (PR #1299)
- **Single source of truth** — the persona/checklist that previously lived in both `agents/code-reviewer.md` and the skill's placeholder template (and drifted independently) is now one file.
- **`subagent-driven-development` follows suit** — its `code-quality-reviewer-prompt.md` now dispatches `Task (general-purpose)` instead of the named agent.
- **Behavioral test added** — `tests/claude-code/test-requesting-code-review.sh` plants real bugs (SQL injection, plaintext password handling, credential logging) into a tiny project and asserts the dispatched reviewer flags every planted issue at Critical/Important severity and refuses to approve the diff.
> Note: `tests/claude-code/test-requesting-code-review.sh` and `tests/claude-code/test-document-review-system.sh` (mentioned later in this document) were lifted into drill scenarios on 2026-05-06 and removed from `tests/`. See `evals/scenarios/code-review-catches-planted-bugs.yaml` and `evals/scenarios/spec-reviewer-catches-planted-flaws.yaml`. The references above and below are preserved as dated artifacts of the work this section describes.
- **Codex and Copilot workaround docs trimmed** — the "Named agent dispatch" sections in `references/codex-tools.md` and `references/copilot-tools.md` documented how to flatten a named agent into a generic dispatch. With no named agents shipping, the workaround is unnecessary; both sections were dropped.
### Subagent-Driven Development
- **No more pause every 3 tasks** — the "review after each batch (3 tasks)" cadence in `requesting-code-review` (originally for `executing-plans`) was leaking into `subagent-driven-development`. Replaced with "each task or at natural checkpoints" plus an explicit continuous-execution directive.
- **SDD integration test now runs its assertions** — three independent bugs caused the test to silently bail before printing any verification results: an unresolved `..` segment in the working-dir path, a `set -euo pipefail` interaction with `find | sort | head -1` (SIGPIPE on the producer killed the script), and a missing `--plugin-dir` on the `claude -p` invocation that caused the test to load the installed plugin instead of the working tree. All three fixed; six verification tests now actually run against a real end-to-end SDD run.
### Cursor
- **Windows SessionStart hook** routed through `run-hook.cmd` instead of invoking the extensionless `session-start` script directly. Fixes Windows opening the file in an editor instead of running it. Also removed an accidental UTF-8 BOM from `hooks-cursor.json`.
### Gemini CLI
- **Subagent dispatch mapping** — Gemini's `Task` dispatch now maps to `@agent-name` / `@generalist`, with parallel subagent dispatch documented for independent tasks.
### Skills
- **Terminology cleanups** across skill content.
### Documentation & Install
- **Factory Droid installation instructions** added to README.
- **Quickstart install links** in README. (PR #1293 by @arittr)
- **Codex plugin install guidance** updated. (PR #1288 by @arittr)
- **Codex `wait` mapping corrected** to `wait_agent` in the tools reference.
- **Install order reorganized**; Codex install instructions cleaned up.
- **Removed vestigial `CHANGELOG.md`** in favor of `RELEASE-NOTES.md` as the single source. (PR #1163 by @shaanmajid)
- **Discord invite link** fixed; release announcements link and a detailed Discord description added to the Community section.
### Community
- @shaanmajid — vestigial `CHANGELOG.md` removal (PR #1163)
- @arittr — README quickstart install links (#1293), Codex plugin install guidance (#1288), `sync-to-codex-plugin` `interface.defaultPrompt` seed (#1180)
## v5.0.7 (2026-03-31)
### GitHub Copilot CLI Support
- **SessionStart context injection** — Copilot CLI v1.0.11 added support for `additionalContext` in sessionStart hook output. The session-start hook now detects the `COPILOT_CLI` environment variable and emits the SDK-standard `{ "additionalContext": "..." }` format, giving Copilot CLI users the full superpowers bootstrap at session start. (Original fix by @culinablaz in PR #910)
- **Tool mapping** — added `references/copilot-tools.md` with the full Claude Code to Copilot CLI tool equivalence table
- **Skill and README updates** — added Copilot CLI to the `using-superpowers` skill's platform instructions and README installation section
### OpenCode Fixes
- **Skills path consistency** — the bootstrap text no longer advertises a misleading `configDir/skills/superpowers/` path that didn't match the runtime path. The agent should use the native `skill` tool, not navigate to files by path. Tests now use consistent paths derived from a single source of truth. (#847, #916)
- **Bootstrap as user message** — moved bootstrap injection from `experimental.chat.system.transform` to `experimental.chat.messages.transform`, prepending to the first user message instead of adding a system message. Avoids token bloat from system messages repeated every turn (#750) and fixes compatibility with Qwen and other models that break on multiple system messages (#894).
## v5.0.6 (2026-03-24)
### Inline Self-Review Replaces Subagent Review Loops
The subagent review loop (dispatching a fresh agent to review plans/specs) doubled execution time (~25 min overhead) without measurably improving plan quality. Regression testing across 5 versions with 5 trials each showed identical quality scores regardless of whether the review loop ran.
- **brainstorming** — replaced Spec Review Loop (subagent dispatch + 3-iteration cap) with inline Spec Self-Review checklist: placeholder scan, internal consistency, scope check, ambiguity check
- **writing-plans** — replaced Plan Review Loop (subagent dispatch + 3-iteration cap) with inline Self-Review checklist: spec coverage, placeholder scan, type consistency
- **writing-plans** — added explicit "No Placeholders" section defining plan failures (TBD, vague descriptions, undefined references, "similar to Task N")
- Self-review catches 3-5 real bugs per run in ~30s instead of ~25 min, with comparable defect rates to the subagent approach
### Brainstorm Server
- **Session directory restructured** — the brainstorm server session directory now contains two peer subdirectories: `content/` (HTML files served to the browser) and `state/` (events, server-info, pid, log). Previously, server state and user interaction data were stored alongside served content, making them accessible over HTTP. The `screen_dir` and `state_dir` paths are both included in the server-started JSON. (Reported by 吉田仁)
### Bug Fixes
- **Owner-PID lifecycle fixes** — the brainstorm server's owner-PID monitoring had two bugs causing false shutdowns within 60 seconds: (1) EPERM from cross-user PIDs (Tailscale SSH, etc.) was treated as "process dead", and (2) on WSL the grandparent PID resolves to a short-lived subprocess that exits before the first lifecycle check. Fixed by treating EPERM as "alive" and validating the owner PID at startup — if it's already dead, monitoring is disabled and the server relies on the 30-minute idle timeout. This also removes the Windows/MSYS2-specific carve-out from `start-server.sh` since the server now handles it generically. (#879)
- **writing-skills** — corrected false claim that SKILL.md frontmatter supports "only two fields"; now says "two required fields" and links to the agentskills.io specification for all supported fields (PR #882 by @arittr)
### Codex App Compatibility
- **codex-tools** — added named agent dispatch mapping documenting how to translate Claude Code's named agent types to Codex's `spawn_agent` with worker roles (PR #647 by @arittr)
- **codex-tools** — added environment detection and Codex App finishing sections for worktree-aware skills (by @arittr)
- **Design spec** — added Codex App compatibility design spec (PRI-823) covering read-only environment detection, worktree-safe skill behavior, and sandbox fallback patterns (by @arittr)
## v5.0.5 (2026-03-17)
### Bug Fixes
- **Brainstorm server ESM fix** — renamed `server.js``server.cjs` so the brainstorming server starts correctly on Node.js 22+ where the root `package.json` `"type": "module"` caused `require()` to fail. (PR #784 by @sarbojitrana, fixes #774, #780, #783)
- **Brainstorm owner-PID on Windows** — skip PID lifecycle monitoring on Windows/MSYS2 where the PID namespace is invisible to Node.js, preventing the server from self-terminating after 60 seconds. (#770, docs from PR #768 by @lucasyhzlu-debug)
- **stop-server.sh reliability** — verify the server process actually died before reporting success. SIGTERM + 2s wait + SIGKILL fallback. (#723)
### Changed
- **Execution handoff** — restore user choice between subagent-driven and inline execution after plan writing. Subagent-driven is recommended but no longer mandatory.
## v5.0.4 (2026-03-16)
### Review Loop Refinements
Dramatically reduces token usage and speeds up spec and plan reviews by eliminating unnecessary review passes and tightening reviewer focus.
- **Single whole-plan review** — plan reviewer now reviews the complete plan in one pass instead of chunk-by-chunk. Removed all chunk-related concepts (`## Chunk N:` headings, 1000-line chunk limits, per-chunk dispatch).
- **Raised the bar for blocking issues** — both spec and plan reviewer prompts now include a "Calibration" section: only flag issues that would cause real problems during implementation. Minor wording, stylistic preferences, and formatting quibbles should not block approval.
- **Reduced max review iterations** — from 5 to 3 for both spec and plan review loops. If the reviewer is calibrated correctly, 3 rounds is plenty.
- **Streamlined reviewer checklists** — spec reviewer trimmed from 7 categories to 5; plan reviewer from 7 to 4. Removed formatting-focused checks (task syntax, chunk size) in favor of substance (buildability, spec alignment).
### OpenCode
- **One-line plugin install** — OpenCode plugin now auto-registers the skills directory via a `config` hook. No symlinks or `skills.paths` config needed. Install is just adding one line to `opencode.json`. (PR #753)
- **Added `package.json`** so OpenCode can install superpowers as an npm package from git.
### Bug Fixes
- **Verify server actually stopped** — `stop-server.sh` now confirms the process is dead before reporting success. SIGTERM + 2s wait + SIGKILL fallback. Reports failure if the process survives. (PR #751)
- **Generic agent language** — brainstorm companion waiting page now says "the agent" instead of "Claude".
## v5.0.3 (2026-03-15)
### Cursor Support
- **Cursor hooks** — added `hooks/hooks-cursor.json` with Cursor's camelCase format (`sessionStart`, `version: 1`) and updated `.cursor-plugin/plugin.json` to reference it. Fixed platform detection in `session-start` to check `CURSOR_PLUGIN_ROOT` first (Cursor may also set `CLAUDE_PLUGIN_ROOT`). (Based on PR #709)
### Bug Fixes
- **Stop firing SessionStart hook on `--resume`** — the startup hook was re-injecting context on resumed sessions, which already have the context in their conversation history. The hook now fires only on `startup`, `clear`, and `compact`.
- **Bash 5.3+ hook hang** — replaced heredoc (`cat <<EOF`) with `printf` in `hooks/session-start`. Fixes indefinite hang on macOS with Homebrew bash 5.3+ caused by a bash regression with large variable expansion in heredocs. (#572, #571)
- **POSIX-safe hook script** — replaced `${BASH_SOURCE[0]:-$0}` with `$0` in `hooks/session-start`. Fixes "Bad substitution" error on Ubuntu/Debian where `/bin/sh` is dash. (#553)
- **Portable shebangs** — replaced `#!/bin/bash` with `#!/usr/bin/env bash` in all shell scripts. Fixes execution on NixOS, FreeBSD, and macOS with Homebrew bash where `/bin/bash` is outdated or missing. (#700)
- **Brainstorm server on Windows** — auto-detect Windows/Git Bash (`OSTYPE=msys*`, `MSYSTEM`) and switch to foreground mode, fixing silent server failure caused by `nohup`/`disown` process reaping. (#737)
- **Codex docs fix** — replaced deprecated `collab` flag with `multi_agent` in Codex documentation. (PR #749)
## v5.0.2 (2026-03-11)
### Zero-Dependency Brainstorm Server
**Removed all vendored node_modules — server.js is now fully self-contained**
- Replaced Express/Chokidar/WebSocket dependencies with zero-dependency Node.js server using built-in `http`, `fs`, and `crypto` modules
- Removed ~1,200 lines of vendored `node_modules/`, `package.json`, and `package-lock.json`
- Custom WebSocket protocol implementation (RFC 6455 framing, ping/pong, proper close handshake)
- Native `fs.watch()` file watching replaces Chokidar
- Full test suite: HTTP serving, WebSocket protocol, file watching, and integration tests
### Brainstorm Server Reliability
- **Auto-exit after 30 minutes idle** — server shuts down when no clients are connected, preventing orphaned processes
- **Owner process tracking** — server monitors the parent harness PID and exits when the owning session dies
- **Liveness check** — skill verifies server is responsive before reusing an existing instance
- **Encoding fix** — proper `<meta charset="utf-8">` on served HTML pages
### Subagent Context Isolation
- All delegation skills (brainstorming, dispatching-parallel-agents, requesting-code-review, subagent-driven-development, writing-plans) now include context isolation principle
- Subagents receive only the context they need, preventing context window pollution
## v5.0.1 (2026-03-10)
### Agentskills Compliance
**Brainstorm-server moved into skill directory**
- Moved `lib/brainstorm-server/``skills/brainstorming/scripts/` per the [agentskills.io](https://agentskills.io) specification
- All `${CLAUDE_PLUGIN_ROOT}/lib/brainstorm-server/` references replaced with relative `scripts/` paths
- Skills are now fully portable across platforms — no platform-specific env vars needed to locate scripts
- `lib/` directory removed (was the last remaining content)
### New Features
**Gemini CLI extension**
- Native Gemini CLI extension support via `gemini-extension.json` and `GEMINI.md` at repo root
- `GEMINI.md` @imports `using-superpowers` skill and tool mapping table at session start
- Gemini CLI tool mapping reference (`skills/using-superpowers/references/gemini-tools.md`) — translates Claude Code tool names (Read, Write, Edit, Bash, etc.) to Gemini CLI equivalents (read_file, write_file, replace, etc.)
- Documents Gemini CLI limitations: no subagent support, skills fall back to `executing-plans`
- Extension root at repo root for cross-platform compatibility (avoids Windows symlink issues)
- Install instructions added to README
### Improvements
**Multi-platform brainstorm server launch**
- Per-platform launch instructions in visual-companion.md: Claude Code (default mode), Codex (auto-foreground via `CODEX_CI`), Gemini CLI (`--foreground` with `is_background`), and fallback for other environments
- Server now writes startup JSON to `$SCREEN_DIR/.server-info` so agents can find the URL and port even when stdout is hidden by background execution
**Brainstorm server dependencies bundled**
- `node_modules` vendored into the repo so the brainstorm server works immediately on fresh plugin installs without requiring `npm` at runtime
- Removed `fsevents` from bundled deps (macOS-only native binary; chokidar falls back gracefully without it)
- Fallback auto-install via `npm install` if `node_modules` is missing
**OpenCode tool mapping fix**
- `TodoWrite``todowrite` (was incorrectly mapped to `update_plan`); verified against OpenCode source
### Bug Fixes
**Windows/Linux: single quotes break SessionStart hook** (#577, #529, #644, PR #585)
- Single quotes around `${CLAUDE_PLUGIN_ROOT}` in hooks.json fail on Windows (cmd.exe doesn't recognize single quotes as path delimiters) and on Linux (single quotes prevent variable expansion)
- Fix: replaced single quotes with escaped double quotes — works across macOS bash, Windows cmd.exe, Windows Git Bash, and Linux, with and without spaces in paths
- Verified on Windows 11 (NT 10.0.26200.0) with Claude Code 2.1.72 and Git for Windows
**Brainstorming spec review loop skipped** (#677)
- The spec review loop (dispatch spec-document-reviewer subagent, iterate until approved) existed in the prose "After the Design" section but was missing from the checklist and process flow diagram
- Since agents follow the diagram and checklist more reliably than prose, the spec review step was being skipped entirely
- Added step 7 (spec review loop) to the checklist and corresponding nodes to the dot graph
- Tested with `claude --plugin-dir` and `claude-session-driver`: worker now correctly dispatches the reviewer
**Cursor install command** (PR #676)
- Fixed Cursor install command in README: `/plugin-add``/add-plugin` (confirmed via Cursor 2.5 release announcement)
**User review gate in brainstorming** (#565)
- Added explicit user review step between spec completion and writing-plans handoff
- User must approve the spec before implementation planning begins
- Checklist, process flow, and prose updated with the new gate
**Session-start hook emits context only once per platform**
- Hook now detects whether it's running in Claude Code or another platform
- Emits `hookSpecificOutput` for Claude Code, `additional_context` for others — prevents double context injection
**Linting fix in token analysis script**
- `except:``except Exception:` in `tests/claude-code/analyze-token-usage.py`
### Maintenance
**Removed dead code**
- Deleted `lib/skills-core.js` and its test (`tests/opencode/test-skills-core.js`) — unused since February 2026
- Removed skills-core existence check from `tests/opencode/test-plugin-loading.sh`
### Community
- @karuturi — Claude Code official marketplace install instructions (PR #610)
- @mvanhorn — session-start hook dual-emit fix, OpenCode tool mapping fix
- @daniel-graham — linting fix for bare except
- PR #585 author — Windows/Linux hooks quoting fix
---
## v5.0.0 (2026-03-09)
## Unreleased
### Breaking Changes
**Specs and plans directory restructured**
- Specs (brainstorming output) now save to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md`
- Plans (writing-plans output) now save to `docs/superpowers/plans/YYYY-MM-DD-<feature-name>.md`
- Specs (brainstorming output) now go to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md`
- Plans (writing-plans output) now go to `docs/superpowers/plans/YYYY-MM-DD-<feature-name>.md`
- User preferences for spec/plan locations override these defaults
- All internal skill references, test files, and example paths updated to match
- Migration: move existing files from `docs/plans/` to new locations if desired
**Subagent-driven development mandatory on capable harnesses**
**Brainstorming → writing-plans transition enforced**
Writing-plans no longer offers a choice between subagent-driven and executing-plans. On harnesses with subagent support (Claude Code, Codex), subagent-driven-development is required. Executing-plans is reserved for harnesses without subagent capability, and now tells the user that Superpowers works better on a subagent-capable platform.
- After design approval, brainstorming now requires using writing-plans skill
- Platform planning features (e.g., EnterPlanMode) should not be used
- Direct implementation without writing-plans is not allowed
**Executing-plans no longer batches**
**Subagent-driven development now mandatory on capable harnesses**
Removed the "execute 3 tasks then stop for review" pattern. Plans now execute continuously, stopping only for blockers.
- On harnesses with subagent support (Claude Code), subagent-driven-development is now required after plan approval
- No longer offers a choice between subagent-driven and executing-plans
- Executing-plans is only used on harnesses without subagent capability
**Slash commands deprecated**
**OpenCode: Switched to native skills system**
`/brainstorm`, `/write-plan`, and `/execute-plan` now show deprecation notices pointing users to the corresponding skills. Commands will be removed in the next major release.
Superpowers for OpenCode now uses OpenCode's native `skill` tool instead of custom `use_skill`/`find_skills` tools. This is a cleaner integration that works with OpenCode's built-in skill discovery.
**Migration required:** Skills must be symlinked to `~/.config/opencode/skills/superpowers/` (see updated installation docs).
### Fixes
**OpenCode: Fixed agent reset on session start (#226)**
The previous bootstrap injection method using `session.prompt({ noReply: true })` caused OpenCode to reset the selected agent to "build" on first message. Now uses `experimental.chat.system.transform` hook which modifies the system prompt directly without side effects.
**OpenCode: Fixed Windows installation (#232)**
- Removed dependency on `skills-core.js` (eliminates broken relative imports when file is copied instead of symlinked)
- Added comprehensive Windows installation docs for cmd.exe, PowerShell, and Git Bash
- Documented proper symlink vs junction usage for each platform
### New Features
**Visual brainstorming companion**
**Visual companion for brainstorming skill**
Optional browser-based companion for brainstorming sessions. When a topic would benefit from visuals, the brainstorming skill offers to show mockups, diagrams, comparisons, and other content in a browser window alongside terminal conversation.
Added optional browser-based visual companion for brainstorming sessions. When users have a browser available, brainstorming can display interactive screens showing current phase, questions, and design decisions in a more readable format than terminal output.
- `lib/brainstorm-server/` — WebSocket server with browser helper library, session management scripts, and dark/light themed frame template ("Superpowers Brainstorming" with GitHub link)
- `skills/brainstorming/visual-companion.md` — Progressive disclosure guide for server workflow, screen authoring, and feedback collection
- Brainstorming skill adds a visual companion decision point to its process flow: after exploring project context, the skill evaluates whether upcoming questions involve visual content and offers the companion in its own message
- Per-question decision: even after accepting, each question is evaluated for whether browser or terminal is more appropriate
- Integration tests in `tests/brainstorm-server/`
Components:
- `lib/brainstorm-server/` - WebSocket server for real-time updates
- `skills/brainstorming/visual-companion.md` - Integration guide
- Helper scripts for session management with proper isolation
- Browser helper library for event capture
**Document review system**
The visual companion is opt-in and falls back gracefully to terminal-only operation.
Automated review loops for spec and plan documents using subagent dispatch:
### Bug Fixes
- `skills/brainstorming/spec-document-reviewer-prompt.md` — Reviewer checks completeness, consistency, architecture, and YAGNI
- `skills/writing-plans/plan-document-reviewer-prompt.md` — Reviewer checks spec alignment, task decomposition, file structure, and file size
- Brainstorming dispatches spec reviewer after writing the design doc
- Writing-plans includes chunk-based plan review loop after each section
- Review loops repeat until approved or escalate after 5 iterations
- End-to-end tests in `tests/claude-code/test-document-review-system.sh`
- Design spec and implementation plan in `docs/superpowers/`
**Fixed Windows hook execution for Claude Code 2.1.x**
**Architecture guidance across the skill pipeline**
Claude Code 2.1.x changed how hooks execute on Windows: it now auto-detects `.sh` files in commands and prepends `bash `. This broke the polyglot wrapper pattern because `bash "run-hook.cmd" session-start.sh` tries to execute the .cmd file as a bash script.
Design-for-isolation and file-size-awareness guidance added to brainstorming, writing-plans, and subagent-driven-development:
Fix: hooks.json now calls session-start.sh directly. Claude Code 2.1.x handles the bash invocation automatically. Also added .gitattributes to enforce LF line endings for shell scripts (fixes CRLF issues on Windows checkout).
- **Brainstorming** — New sections: "Design for isolation and clarity" (clear boundaries, well-defined interfaces, independently testable units) and "Working in existing codebases" (follow existing patterns, targeted improvements only)
- **Writing-plans** — New "File Structure" section: map out files and responsibilities before defining tasks. New "Scope Check" backstop: catch multi-subsystem specs that should have been decomposed during brainstorming
- **SDD implementer** — New "Code Organization" section (follow plan's file structure, report concerns about growing files) and "When You're in Over Your Head" escalation guidance
- **SDD code quality reviewer** — Now checks architecture, unit decomposition, plan conformance, and file growth
- **Spec/plan reviewers** — Architecture and file size added to review criteria
- **Scope assessment** — Brainstorming now assesses whether a project is too large for a single spec. Multi-subsystem requests are flagged early and decomposed into sub-projects, each with its own spec → plan → implementation cycle
**Brainstorming visual companion: reduced token cost and improved persistence**
**Subagent-driven development improvements**
The visual companion now generates much smaller HTML per screen. The server automatically wraps bare content fragments in the frame template (header, CSS theme, feedback footer, interactive JS), so Claude writes only the content portion (~30 lines instead of ~260). Full HTML documents are still served as-is when Claude needs complete control.
- **Model selection** — Guidance for choosing model capability by task type: cheap models for mechanical implementation, standard for integration, capable for architecture and review
- **Implementer status protocol** — Subagents now report DONE, DONE_WITH_CONCERNS, BLOCKED, or NEEDS_CONTEXT. Controller handles each status appropriately: re-dispatching with more context, upgrading model capability, breaking tasks apart, or escalating to human
Other improvements:
- `toggleSelect`/`send`/`selectedChoice` moved from inline template script to `helper.js` (auto-injected)
- `start-server.sh --project-dir` persists mockups under `.superpowers/brainstorm/` instead of `/tmp`
- `stop-server.sh` only deletes ephemeral `/tmp` sessions, preserving persistent ones
- Dark mode fix: `sendToClaude` confirmation page now uses CSS variables instead of hardcoded colors
- Skill restructured: SKILL.md is minimal (prompt + pointer); all visual companion details in progressive disclosure doc (`visual-companion.md`)
- Prompt to user now notes the feature is new, token-intensive, and can be slow
- Deleted redundant `CLAUDE-INSTRUCTIONS.md` (content folded into `visual-companion.md`)
- Test fixes: correct env var (`BRAINSTORM_DIR`), polling-based startup wait, new tests for frame wrapping
### Improvements
**Instruction priority hierarchy**
**Instruction priority clarified in using-superpowers**
Added explicit priority ordering to using-superpowers:
Added explicit instruction priority hierarchy to prevent conflicts with user preferences:
1. User's explicit instructions (CLAUDE.md, AGENTS.md, direct requests) — highest priority
2. Superpowers skills — override default system behavior
1. User's explicit instructions (CLAUDE.md, direct requests) — highest priority
2. Superpowers skills — override default system behavior where they conflict
3. Default system prompt — lowest priority
If CLAUDE.md or AGENTS.md says "don't use TDD" and a skill says "always use TDD," the user's instructions win.
**SUBAGENT-STOP gate**
Added `<SUBAGENT-STOP>` block to using-superpowers. Subagents dispatched for specific tasks now skip the skill instead of activating the 1% rule and invoking full skill workflows.
**Multi-platform improvements**
- Codex tool mapping moved to progressive disclosure reference file (`references/codex-tools.md`)
- Platform Adaptation pointer added so non-Claude-Code platforms can find tool equivalents
- Plan headers now address "agentic workers" instead of "Claude" specifically
- Collab feature requirement documented in `docs/README.codex.md`
**Writing-plans template updates**
- Plan steps now use checkbox syntax (`- [ ] **Step N:**`) for progress tracking
- Plan header references both subagent-driven-development and executing-plans with platform-aware routing
---
## v4.3.1 (2026-02-21)
### Added
**Cursor support**
Superpowers now works with Cursor's plugin system. Includes a `.cursor-plugin/plugin.json` manifest and Cursor-specific installation instructions in the README. The SessionStart hook output now includes an `additional_context` field alongside the existing `hookSpecificOutput.additionalContext` for Cursor hook compatibility.
### Fixed
**Windows: Restored polyglot wrapper for reliable hook execution (#518, #504, #491, #487, #466, #440)**
Claude Code's `.sh` auto-detection on Windows was prepending `bash` to the hook command, breaking execution. The fix:
- Renamed `session-start.sh` to `session-start` (extensionless) so auto-detection doesn't interfere
- Restored `run-hook.cmd` polyglot wrapper with multi-location bash discovery (standard Git for Windows paths, then PATH fallback)
- Exits silently if no bash is found rather than erroring
- On Unix, the wrapper runs the script directly via `exec bash`
- Uses POSIX-safe `dirname "$0"` path resolution (works on dash/sh, not just bash)
This fixes SessionStart failures on Windows with spaces in paths, missing WSL, `set -euo pipefail` fragility on MSYS, and backslash mangling.
This ensures users remain in control. If CLAUDE.md says "don't use TDD" and a skill says "always use TDD," CLAUDE.md wins.
## v4.3.0 (2026-02-12)

48
agents/code-reviewer.md Normal file
View File

@@ -0,0 +1,48 @@
---
name: code-reviewer
description: |
Use this agent when a major project step has been completed and needs to be reviewed against the original plan and coding standards. Examples: <example>Context: The user is creating a code-review agent that should be called after a logical chunk of code is written. user: "I've finished implementing the user authentication system as outlined in step 3 of our plan" assistant: "Great work! Now let me use the code-reviewer agent to review the implementation against our plan and coding standards" <commentary>Since a major project step has been completed, use the code-reviewer agent to validate the work against the plan and identify any issues.</commentary></example> <example>Context: User has completed a significant feature implementation. user: "The API endpoints for the task management system are now complete - that covers step 2 from our architecture document" assistant: "Excellent! Let me have the code-reviewer agent examine this implementation to ensure it aligns with our plan and follows best practices" <commentary>A numbered step from the planning document has been completed, so the code-reviewer agent should review the work.</commentary></example>
model: inherit
---
You are a Senior Code Reviewer with expertise in software architecture, design patterns, and best practices. Your role is to review completed project steps against original plans and ensure code quality standards are met.
When reviewing completed work, you will:
1. **Plan Alignment Analysis**:
- Compare the implementation against the original planning document or step description
- Identify any deviations from the planned approach, architecture, or requirements
- Assess whether deviations are justified improvements or problematic departures
- Verify that all planned functionality has been implemented
2. **Code Quality Assessment**:
- Review code for adherence to established patterns and conventions
- Check for proper error handling, type safety, and defensive programming
- Evaluate code organization, naming conventions, and maintainability
- Assess test coverage and quality of test implementations
- Look for potential security vulnerabilities or performance issues
3. **Architecture and Design Review**:
- Ensure the implementation follows SOLID principles and established architectural patterns
- Check for proper separation of concerns and loose coupling
- Verify that the code integrates well with existing systems
- Assess scalability and extensibility considerations
4. **Documentation and Standards**:
- Verify that code includes appropriate comments and documentation
- Check that file headers, function documentation, and inline comments are present and accurate
- Ensure adherence to project-specific coding standards and conventions
5. **Issue Identification and Recommendations**:
- Clearly categorize issues as: Critical (must fix), Important (should fix), or Suggestions (nice to have)
- For each issue, provide specific examples and actionable recommendations
- When you identify plan deviations, explain whether they're problematic or beneficial
- Suggest specific improvements with code examples when helpful
6. **Communication Protocol**:
- If you find significant deviations from the plan, ask the coding agent to review and confirm the changes
- If you identify issues with the original plan itself, recommend plan updates
- For implementation problems, provide clear guidance on fixes needed
- Always acknowledge what was done well before highlighting issues
Your output should be structured, actionable, and focused on helping maintain high code quality while ensuring project goals are met. Be thorough but concise, and always provide constructive feedback that helps improve both the current implementation and future development practices.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 47 KiB

View File

@@ -1 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?><svg id="Calque_1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M394.28,207.8c.81,2.41,1.39,4.78,1.8,7.07,1.61,9.03-.93,17.78-5.99,21.74-22.6,17.7-49.85,29.35-75.34,38.6-.59.22-1.09.28-1.4.34-2.22.47-4.95,1.04-7.25,0-1.46-.66-2.25-1.74-2.66-2.3-1.56-2.1-1.59-4.31-1.56-5.13.1-2.67-.01-4.69,0-4.82.45-3.52.91-10.66,1.41-21.28.6-3.87,2.16-9.63,6.94-13.96,4.01-3.62,8.33-4.6,14.59-5.87,10.76-2.19,37.21-8.22,47.42-16.56,1.63-1.33,2.97-2.65,4.19-3.96,3.72-3.99,6.39-7.92,7.93-10.36,3.22,3.22,7.25,8.48,9.92,16.47Z"/><path d="M428.67,185.28c-2.33,11.99-8.91,22.32-15.88,30.38.27-5.5-.05-12.11-1.86-19.08-5.04-19.36-19.74-34.7-37.78-37.78-32.21-9.74-70.59,3.79-99.08,18.29-3.87,1.95-9.52-2.77-11.84-8.16-3.32-7.71-1.63-6.28,2.61-8.49,38.31-20.03,82.01-39.61,123.91-29.7,8.26,1.95,15.96,5.26,23.48,10.54,11.32,7.96,20.21,24.74,16.44,44Z"/><path d="M117.72,304.2c-.81-2.41-1.39-4.78-1.8-7.07-1.61-9.03.93-17.78,5.99-21.74,22.6-17.7,49.85-29.35,75.34-38.6.59-.22,1.09-.28,1.4-.34,2.22-.47,4.95-1.04,7.25,0,1.46.66,2.25,1.74,2.66,2.3,1.56,2.1,1.59,4.31,1.56,5.13-.1,2.67.01,4.69,0,4.82-.45,3.52-.91,10.66-1.41,21.28-.6,3.87-2.16,9.63-6.94,13.96-4.01,3.62-8.33,4.6-14.59,5.87-10.76,2.19-37.21,8.22-47.42,16.56-1.63,1.33-2.97,2.65-4.19,3.96-3.72,3.99-6.39,7.92-7.93,10.36-3.22-3.22-7.25-8.48-9.92-16.47Z"/><path d="M83.33,326.72c2.33-11.99,8.91-22.32,15.88-30.38-.27,5.5.05,12.11,1.86,19.08,5.04,19.36,19.74,34.7,37.78,37.78,32.21,9.74,70.59-3.79,99.08-18.29,3.87-1.95,9.52,2.77,11.84,8.16,3.32,7.71,1.63,6.28-2.61,8.49-38.31,20.03-82.01,39.61-123.91,29.7-8.26-1.95-15.96-5.26-23.48-10.54-11.32-7.96-20.21-24.74-16.44-44Z"/><ellipse cx="255.16" cy="258.86" rx="28.95" ry="28.76"/></svg>

Before

Width:  |  Height:  |  Size: 1.7 KiB

6
commands/brainstorm.md Normal file
View File

@@ -0,0 +1,6 @@
---
description: "You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores requirements and design before implementation."
disable-model-invocation: true
---
Invoke the superpowers:brainstorming skill and follow it exactly as presented to you

6
commands/execute-plan.md Normal file
View File

@@ -0,0 +1,6 @@
---
description: Execute plan in batches with review checkpoints
disable-model-invocation: true
---
Invoke the superpowers:executing-plans skill and follow it exactly as presented to you

6
commands/write-plan.md Normal file
View File

@@ -0,0 +1,6 @@
---
description: Create detailed implementation plan with bite-sized tasks
disable-model-invocation: true
---
Invoke the superpowers:writing-plans skill and follow it exactly as presented to you

126
docs/README.codex.md Normal file
View File

@@ -0,0 +1,126 @@
# Superpowers for Codex
Guide for using Superpowers with OpenAI Codex via native skill discovery.
## Quick Install
Tell Codex:
```
Fetch and follow instructions from https://raw.githubusercontent.com/obra/superpowers/refs/heads/main/.codex/INSTALL.md
```
## Manual Installation
### Prerequisites
- OpenAI Codex CLI
- Git
### Steps
1. Clone the repo:
```bash
git clone https://github.com/obra/superpowers.git ~/.codex/superpowers
```
2. Create the skills symlink:
```bash
mkdir -p ~/.agents/skills
ln -s ~/.codex/superpowers/skills ~/.agents/skills/superpowers
```
3. Restart Codex.
4. **For subagent skills** (optional): Skills like `dispatching-parallel-agents` and `subagent-driven-development` require Codex's collab feature. Add to your Codex config:
```toml
[features]
collab = true
```
### Windows
Use a junction instead of a symlink (works without Developer Mode):
```powershell
New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.agents\skills"
cmd /c mklink /J "$env:USERPROFILE\.agents\skills\superpowers" "$env:USERPROFILE\.codex\superpowers\skills"
```
## How It Works
Codex has native skill discovery — it scans `~/.agents/skills/` at startup, parses SKILL.md frontmatter, and loads skills on demand. Superpowers skills are made visible through a single symlink:
```
~/.agents/skills/superpowers/ → ~/.codex/superpowers/skills/
```
The `using-superpowers` skill is discovered automatically and enforces skill usage discipline — no additional configuration needed.
## Usage
Skills are discovered automatically. Codex activates them when:
- You mention a skill by name (e.g., "use brainstorming")
- The task matches a skill's description
- The `using-superpowers` skill directs Codex to use one
### Personal Skills
Create your own skills in `~/.agents/skills/`:
```bash
mkdir -p ~/.agents/skills/my-skill
```
Create `~/.agents/skills/my-skill/SKILL.md`:
```markdown
---
name: my-skill
description: Use when [condition] - [what it does]
---
# My Skill
[Your skill content here]
```
The `description` field is how Codex decides when to activate a skill automatically — write it as a clear trigger condition.
## Updating
```bash
cd ~/.codex/superpowers && git pull
```
Skills update instantly through the symlink.
## Uninstalling
```bash
rm ~/.agents/skills/superpowers
```
**Windows (PowerShell):**
```powershell
Remove-Item "$env:USERPROFILE\.agents\skills\superpowers"
```
Optionally delete the clone: `rm -rf ~/.codex/superpowers` (Windows: `Remove-Item -Recurse -Force "$env:USERPROFILE\.codex\superpowers"`).
## Troubleshooting
### Skills not showing up
1. Verify the symlink: `ls -la ~/.agents/skills/superpowers`
2. Check skills exist: `ls ~/.codex/superpowers/skills`
3. Restart Codex — skills are discovered at startup
### Windows junction issues
Junctions normally work without special permissions. If creation fails, try running PowerShell as administrator.
## Getting Help
- Report issues: https://github.com/obra/superpowers/issues
- Main documentation: https://github.com/obra/superpowers

View File

@@ -2,40 +2,169 @@
Complete guide for using Superpowers with [OpenCode.ai](https://opencode.ai).
## Installation
## Quick Install
Add superpowers to the `plugin` array in your `opencode.json` (global or project-level):
Tell OpenCode:
```json
{
"plugin": ["superpowers@git+https://github.com/obra/superpowers.git"]
}
```
Clone https://github.com/obra/superpowers to ~/.config/opencode/superpowers, then create directory ~/.config/opencode/plugins, then symlink ~/.config/opencode/superpowers/.opencode/plugins/superpowers.js to ~/.config/opencode/plugins/superpowers.js, then symlink ~/.config/opencode/superpowers/skills to ~/.config/opencode/skills/superpowers, then restart opencode.
```
Restart OpenCode. The plugin installs through OpenCode's plugin manager and
registers all skills.
## Manual Installation
Verify by asking: "Tell me about your superpowers"
### Prerequisites
OpenCode uses its own plugin install. If you also use Claude Code, Codex, or
another harness, install Superpowers separately for each one.
- [OpenCode.ai](https://opencode.ai) installed
- Git installed
### Migrating from the old symlink-based install
If you previously installed superpowers using `git clone` and symlinks, remove the old setup:
### macOS / Linux
```bash
# Remove old symlinks
# 1. Install Superpowers (or update existing)
if [ -d ~/.config/opencode/superpowers ]; then
cd ~/.config/opencode/superpowers && git pull
else
git clone https://github.com/obra/superpowers.git ~/.config/opencode/superpowers
fi
# 2. Create directories
mkdir -p ~/.config/opencode/plugins ~/.config/opencode/skills
# 3. Remove old symlinks/directories if they exist
rm -f ~/.config/opencode/plugins/superpowers.js
rm -rf ~/.config/opencode/skills/superpowers
# Optionally remove the cloned repo
rm -rf ~/.config/opencode/superpowers
# 4. Create symlinks
ln -s ~/.config/opencode/superpowers/.opencode/plugins/superpowers.js ~/.config/opencode/plugins/superpowers.js
ln -s ~/.config/opencode/superpowers/skills ~/.config/opencode/skills/superpowers
# Remove skills.paths from opencode.json if you added one for superpowers
# 5. Restart OpenCode
```
Then follow the installation steps above.
#### Verify Installation
```bash
ls -l ~/.config/opencode/plugins/superpowers.js
ls -l ~/.config/opencode/skills/superpowers
```
Both should show symlinks pointing to the superpowers directory.
### Windows
**Prerequisites:**
- Git installed
- Either **Developer Mode** enabled OR **Administrator privileges**
- Windows 10: Settings → Update & Security → For developers
- Windows 11: Settings → System → For developers
Pick your shell below: [Command Prompt](#command-prompt) | [PowerShell](#powershell) | [Git Bash](#git-bash)
#### Command Prompt
Run as Administrator, or with Developer Mode enabled:
```cmd
:: 1. Install Superpowers
git clone https://github.com/obra/superpowers.git "%USERPROFILE%\.config\opencode\superpowers"
:: 2. Create directories
mkdir "%USERPROFILE%\.config\opencode\plugins" 2>nul
mkdir "%USERPROFILE%\.config\opencode\skills" 2>nul
:: 3. Remove existing links (safe for reinstalls)
del "%USERPROFILE%\.config\opencode\plugins\superpowers.js" 2>nul
rmdir "%USERPROFILE%\.config\opencode\skills\superpowers" 2>nul
:: 4. Create plugin symlink (requires Developer Mode or Admin)
mklink "%USERPROFILE%\.config\opencode\plugins\superpowers.js" "%USERPROFILE%\.config\opencode\superpowers\.opencode\plugins\superpowers.js"
:: 5. Create skills junction (works without special privileges)
mklink /J "%USERPROFILE%\.config\opencode\skills\superpowers" "%USERPROFILE%\.config\opencode\superpowers\skills"
:: 6. Restart OpenCode
```
#### PowerShell
Run as Administrator, or with Developer Mode enabled:
```powershell
# 1. Install Superpowers
git clone https://github.com/obra/superpowers.git "$env:USERPROFILE\.config\opencode\superpowers"
# 2. Create directories
New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\opencode\plugins"
New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\opencode\skills"
# 3. Remove existing links (safe for reinstalls)
Remove-Item "$env:USERPROFILE\.config\opencode\plugins\superpowers.js" -Force -ErrorAction SilentlyContinue
Remove-Item "$env:USERPROFILE\.config\opencode\skills\superpowers" -Force -ErrorAction SilentlyContinue
# 4. Create plugin symlink (requires Developer Mode or Admin)
New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.config\opencode\plugins\superpowers.js" -Target "$env:USERPROFILE\.config\opencode\superpowers\.opencode\plugins\superpowers.js"
# 5. Create skills junction (works without special privileges)
New-Item -ItemType Junction -Path "$env:USERPROFILE\.config\opencode\skills\superpowers" -Target "$env:USERPROFILE\.config\opencode\superpowers\skills"
# 6. Restart OpenCode
```
#### Git Bash
Note: Git Bash's native `ln` command copies files instead of creating symlinks. Use `cmd //c mklink` instead (the `//c` is Git Bash syntax for `/c`).
```bash
# 1. Install Superpowers
git clone https://github.com/obra/superpowers.git ~/.config/opencode/superpowers
# 2. Create directories
mkdir -p ~/.config/opencode/plugins ~/.config/opencode/skills
# 3. Remove existing links (safe for reinstalls)
rm -f ~/.config/opencode/plugins/superpowers.js 2>/dev/null
rm -rf ~/.config/opencode/skills/superpowers 2>/dev/null
# 4. Create plugin symlink (requires Developer Mode or Admin)
cmd //c "mklink \"$(cygpath -w ~/.config/opencode/plugins/superpowers.js)\" \"$(cygpath -w ~/.config/opencode/superpowers/.opencode/plugins/superpowers.js)\""
# 5. Create skills junction (works without special privileges)
cmd //c "mklink /J \"$(cygpath -w ~/.config/opencode/skills/superpowers)\" \"$(cygpath -w ~/.config/opencode/superpowers/skills)\""
# 6. Restart OpenCode
```
#### WSL Users
If running OpenCode inside WSL, use the [macOS / Linux](#macos--linux) instructions instead.
#### Verify Installation
**Command Prompt:**
```cmd
dir /AL "%USERPROFILE%\.config\opencode\plugins"
dir /AL "%USERPROFILE%\.config\opencode\skills"
```
**PowerShell:**
```powershell
Get-ChildItem "$env:USERPROFILE\.config\opencode\plugins" | Where-Object { $_.LinkType }
Get-ChildItem "$env:USERPROFILE\.config\opencode\skills" | Where-Object { $_.LinkType }
```
Look for `<SYMLINK>` or `<JUNCTION>` in the output.
#### Troubleshooting Windows
**"You do not have sufficient privilege" error:**
- Enable Developer Mode in Windows Settings, OR
- Right-click your terminal → "Run as Administrator"
**"Cannot create a file when that file already exists":**
- Run the removal commands (step 3) first, then retry
**Symlinks not working after git clone:**
- Run `git config --global core.symlinks true` and re-clone
## Usage
@@ -49,6 +178,8 @@ use skill tool to list skills
### Loading a Skill
Use OpenCode's native `skill` tool to load a specific skill:
```
use skill tool to load superpowers/brainstorming
```
@@ -76,82 +207,124 @@ description: Use when [condition] - [what it does]
### Project Skills
Create project-specific skills in `.opencode/skills/` within your project.
Create project-specific skills in your OpenCode project:
**Skill Priority:** Project skills > Personal skills > Superpowers skills
## Updating
OpenCode installs Superpowers through a git-backed package spec. Some OpenCode
and Bun versions pin that resolved git dependency in a lockfile or cache, so a
restart may not pick up the newest Superpowers commit. If updates do not appear,
clear OpenCode's package cache or reinstall the plugin.
To pin a specific version, use a branch or tag:
```json
{
"plugin": ["superpowers@git+https://github.com/obra/superpowers.git#v5.0.3"]
}
```bash
# In your OpenCode project
mkdir -p .opencode/skills/my-project-skill
```
## How It Works
Create `.opencode/skills/my-project-skill/SKILL.md`:
The plugin does two things:
```markdown
---
name: my-project-skill
description: Use when [condition] - [what it does]
---
1. **Injects bootstrap context** via the `experimental.chat.system.transform` hook, adding superpowers awareness to every conversation.
2. **Registers the skills directory** via the `config` hook, so OpenCode discovers all superpowers skills without symlinks or manual config.
# My Project Skill
[Your skill content here]
```
## Skill Locations
OpenCode discovers skills from these locations:
1. **Project skills** (`.opencode/skills/`) - Highest priority
2. **Personal skills** (`~/.config/opencode/skills/`)
3. **Superpowers skills** (`~/.config/opencode/skills/superpowers/`) - via symlink
## Features
### Automatic Context Injection
The plugin automatically injects superpowers context via the `experimental.chat.system.transform` hook. This adds the "using-superpowers" skill content to the system prompt on every request.
### Native Skills Integration
Superpowers uses OpenCode's native `skill` tool for skill discovery and loading. Skills are symlinked into `~/.config/opencode/skills/superpowers/` so they appear alongside your personal and project skills.
### Tool Mapping
Skills written for Claude Code are automatically adapted for OpenCode:
Skills written for Claude Code are automatically adapted for OpenCode. The bootstrap provides mapping instructions:
- `TodoWrite``todowrite`
- `TodoWrite``update_plan`
- `Task` with subagents → OpenCode's `@mention` system
- `Skill` tool → OpenCode's native `skill` tool
- File operations → Native OpenCode tools
## Architecture
### Plugin Structure
**Location:** `~/.config/opencode/superpowers/.opencode/plugins/superpowers.js`
**Components:**
- `experimental.chat.system.transform` hook for bootstrap injection
- Reads and injects the "using-superpowers" skill content
### Skills
**Location:** `~/.config/opencode/skills/superpowers/` (symlink to `~/.config/opencode/superpowers/skills/`)
Skills are discovered by OpenCode's native skill system. Each skill has a `SKILL.md` file with YAML frontmatter.
## Updating
```bash
cd ~/.config/opencode/superpowers
git pull
```
Restart OpenCode to load the updates.
## Troubleshooting
### Plugin not loading
1. Check OpenCode logs: `opencode run --print-logs "hello" 2>&1 | grep -i superpowers`
2. Verify the plugin line in your `opencode.json` is correct
3. Make sure you're running a recent version of OpenCode
### Windows install issues
Some Windows OpenCode builds have upstream installer issues with git-backed
plugin specs, including cache paths for `git+https` URLs and Bun not finding
`git.exe` even when it works in a normal terminal. If OpenCode cannot install
the plugin, try installing with system npm and pointing OpenCode at the local
package:
```powershell
npm install superpowers@git+https://github.com/obra/superpowers.git --prefix "$HOME\.config\opencode"
```
Then use the installed package path in `opencode.json`:
```json
{
"plugin": ["~/.config/opencode/node_modules/superpowers"]
}
```
1. Check plugin exists: `ls ~/.config/opencode/superpowers/.opencode/plugins/superpowers.js`
2. Check symlink/junction: `ls -l ~/.config/opencode/plugins/` (macOS/Linux) or `dir /AL %USERPROFILE%\.config\opencode\plugins` (Windows)
3. Check OpenCode logs: `opencode run "test" --print-logs --log-level DEBUG`
4. Look for plugin loading message in logs
### Skills not found
1. Use OpenCode's `skill` tool to list available skills
2. Check that the plugin is loading (see above)
3. Each skill needs a `SKILL.md` file with valid YAML frontmatter
1. Verify skills symlink: `ls -l ~/.config/opencode/skills/superpowers` (should point to superpowers/skills/)
2. Use OpenCode's `skill` tool to list available skills
3. Check skill structure: each skill needs a `SKILL.md` file with valid frontmatter
### Windows: Module not found error
If you see `Cannot find module` errors on Windows:
- **Cause:** Git Bash `ln -sf` copies files instead of creating symlinks
- **Fix:** Use `mklink /J` directory junctions instead (see Windows installation steps)
### Bootstrap not appearing
1. Check OpenCode version supports `experimental.chat.system.transform` hook
2. Restart OpenCode after config changes
1. Verify using-superpowers skill exists: `ls ~/.config/opencode/superpowers/skills/using-superpowers/SKILL.md`
2. Check OpenCode version supports `experimental.chat.system.transform` hook
3. Restart OpenCode after plugin changes
## Getting Help
- Report issues: https://github.com/obra/superpowers/issues
- Main documentation: https://github.com/obra/superpowers
- OpenCode docs: https://opencode.ai/docs/
## Testing
Verify your installation:
```bash
# Check plugin loads
opencode run --print-logs "hello" 2>&1 | grep -i superpowers
# Check skills are discoverable
opencode run "use skill tool to list all skills" 2>&1 | grep -i superpowers
# Check bootstrap injection
opencode run "what superpowers do you have?"
```
The agent should mention having superpowers and be able to list skills from `superpowers/`.

View File

@@ -1,6 +1,6 @@
# OpenCode Support Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Add full superpowers support for OpenCode.ai with a native JavaScript plugin that shares core functionality with the existing Codex implementation.

View File

@@ -1,6 +1,6 @@
# Visual Brainstorming Companion Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Give Claude a browser-based visual companion for brainstorming sessions - show mockups, prototypes, and interactive choices alongside terminal conversation.

View File

@@ -1,6 +1,6 @@
# Document Review System Implementation Plan
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan.
> **For Claude:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan.
**Goal:** Add spec and plan document review loops to the brainstorming and writing-plans skills.
@@ -285,12 +285,12 @@ Run: `grep -A 20 "Plan Document Header" skills/writing-plans/SKILL.md`
The plan header should note that tasks and steps use checkbox syntax. Update the header comment:
```markdown
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Tasks and steps use checkbox (`- [ ]`) syntax for tracking.
> **For Claude:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Tasks and steps use checkbox (`- [ ]`) syntax for tracking.
```
- [ ] **Step 3:** Verify the change
Run: `grep -A 5 "For agentic workers:" skills/writing-plans/SKILL.md`
Run: `grep -A 5 "For Claude:" skills/writing-plans/SKILL.md`
Expected: Shows updated header with checkbox syntax mention
- [ ] **Step 4:** Commit

View File

@@ -1,6 +1,6 @@
# Visual Brainstorming Refactor Implementation Plan
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For Claude:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Refactor visual brainstorming from blocking TUI feedback model to non-blocking "Browser Displays, Terminal Commands" architecture.

View File

@@ -1,479 +0,0 @@
# Zero-Dependency Brainstorm Server Implementation Plan
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace the brainstorm server's vendored node_modules with a single zero-dependency `server.js` using Node built-ins.
**Architecture:** Single file with WebSocket protocol (RFC 6455 text frames), HTTP server (`http` module), and file watching (`fs.watch`). Exports protocol functions for unit testing when required as a module.
**Tech Stack:** Node.js built-ins only: `http`, `crypto`, `fs`, `path`
**Spec:** `docs/superpowers/specs/2026-03-11-zero-dep-brainstorm-server-design.md`
**Existing tests:** `tests/brainstorm-server/ws-protocol.test.js` (unit), `tests/brainstorm-server/server.test.js` (integration)
---
## File Map
- **Create:** `skills/brainstorming/scripts/server.js` — the zero-dep replacement
- **Modify:** `skills/brainstorming/scripts/start-server.sh:94,100` — change `index.js` to `server.js`
- **Modify:** `.gitignore:6` — remove the `!skills/brainstorming/scripts/node_modules/` exception
- **Delete:** `skills/brainstorming/scripts/index.js`
- **Delete:** `skills/brainstorming/scripts/package.json`
- **Delete:** `skills/brainstorming/scripts/package-lock.json`
- **Delete:** `skills/brainstorming/scripts/node_modules/` (714 files)
- **No changes:** `skills/brainstorming/scripts/helper.js`, `skills/brainstorming/scripts/frame-template.html`, `skills/brainstorming/scripts/stop-server.sh`
---
## Chunk 1: WebSocket Protocol Layer
### Task 1: Implement WebSocket protocol exports
**Files:**
- Create: `skills/brainstorming/scripts/server.js`
- Test: `tests/brainstorm-server/ws-protocol.test.js` (already exists)
- [ ] **Step 1: Create server.js with OPCODES constant and computeAcceptKey**
```js
const crypto = require('crypto');
const OPCODES = { TEXT: 0x01, CLOSE: 0x08, PING: 0x09, PONG: 0x0A };
const WS_MAGIC = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11';
function computeAcceptKey(clientKey) {
return crypto.createHash('sha1').update(clientKey + WS_MAGIC).digest('base64');
}
```
- [ ] **Step 2: Implement encodeFrame**
Server frames are never masked. Three length encodings:
- payload < 126: 2-byte header (FIN+opcode, length)
- 126-65535: 4-byte header (FIN+opcode, 126, 16-bit length)
- &gt; 65535: 10-byte header (FIN+opcode, 127, 64-bit length)
```js
function encodeFrame(opcode, payload) {
const fin = 0x80;
const len = payload.length;
let header;
if (len < 126) {
header = Buffer.alloc(2);
header[0] = fin | opcode;
header[1] = len;
} else if (len < 65536) {
header = Buffer.alloc(4);
header[0] = fin | opcode;
header[1] = 126;
header.writeUInt16BE(len, 2);
} else {
header = Buffer.alloc(10);
header[0] = fin | opcode;
header[1] = 127;
header.writeBigUInt64BE(BigInt(len), 2);
}
return Buffer.concat([header, payload]);
}
```
- [ ] **Step 3: Implement decodeFrame**
Client frames are always masked. Returns `{ opcode, payload, bytesConsumed }` or `null` for incomplete. Throws on unmasked frames.
```js
function decodeFrame(buffer) {
if (buffer.length < 2) return null;
const firstByte = buffer[0];
const secondByte = buffer[1];
const opcode = firstByte & 0x0F;
const masked = (secondByte & 0x80) !== 0;
let payloadLen = secondByte & 0x7F;
let offset = 2;
if (!masked) throw new Error('Client frames must be masked');
if (payloadLen === 126) {
if (buffer.length < 4) return null;
payloadLen = buffer.readUInt16BE(2);
offset = 4;
} else if (payloadLen === 127) {
if (buffer.length < 10) return null;
payloadLen = Number(buffer.readBigUInt64BE(2));
offset = 10;
}
const maskOffset = offset;
const dataOffset = offset + 4;
const totalLen = dataOffset + payloadLen;
if (buffer.length < totalLen) return null;
const mask = buffer.slice(maskOffset, dataOffset);
const data = Buffer.alloc(payloadLen);
for (let i = 0; i < payloadLen; i++) {
data[i] = buffer[dataOffset + i] ^ mask[i % 4];
}
return { opcode, payload: data, bytesConsumed: totalLen };
}
```
- [ ] **Step 4: Add module exports at the bottom of the file**
```js
module.exports = { computeAcceptKey, encodeFrame, decodeFrame, OPCODES };
```
- [ ] **Step 5: Run unit tests**
Run: `cd tests/brainstorm-server && node ws-protocol.test.js`
Expected: All tests pass (handshake, encoding, decoding, boundaries, edge cases)
- [ ] **Step 6: Commit**
```bash
git add skills/brainstorming/scripts/server.js
git commit -m "Add WebSocket protocol layer for zero-dep brainstorm server"
```
---
## Chunk 2: HTTP Server and Application Logic
### Task 2: Add HTTP server, file watching, and WebSocket connection handling
**Files:**
- Modify: `skills/brainstorming/scripts/server.js`
- Test: `tests/brainstorm-server/server.test.js` (already exists)
- [ ] **Step 1: Add configuration and constants at top of server.js (after requires)**
```js
const http = require('http');
const fs = require('fs');
const path = require('path');
const PORT = process.env.BRAINSTORM_PORT || (49152 + Math.floor(Math.random() * 16383));
const HOST = process.env.BRAINSTORM_HOST || '127.0.0.1';
const URL_HOST = process.env.BRAINSTORM_URL_HOST || (HOST === '127.0.0.1' ? 'localhost' : HOST);
const SCREEN_DIR = process.env.BRAINSTORM_DIR || '/tmp/brainstorm';
const MIME_TYPES = {
'.html': 'text/html', '.css': 'text/css', '.js': 'application/javascript',
'.json': 'application/json', '.png': 'image/png', '.jpg': 'image/jpeg',
'.jpeg': 'image/jpeg', '.gif': 'image/gif', '.svg': 'image/svg+xml'
};
```
- [ ] **Step 2: Add WAITING_PAGE, template loading at module scope, and helper functions**
Load `frameTemplate` and `helperInjection` at module scope so they're accessible to `wrapInFrame` and `handleRequest`. They only read files from `__dirname` (the scripts directory), which is valid whether the module is required or run directly.
```js
const WAITING_PAGE = `<!DOCTYPE html>
<html>
<head><title>Brainstorm Companion</title>
<style>body { font-family: system-ui, sans-serif; padding: 2rem; max-width: 800px; margin: 0 auto; }
h1 { color: #333; } p { color: #666; }</style>
</head>
<body><h1>Brainstorm Companion</h1>
<p>Waiting for Claude to push a screen...</p></body></html>`;
const frameTemplate = fs.readFileSync(path.join(__dirname, 'frame-template.html'), 'utf-8');
const helperScript = fs.readFileSync(path.join(__dirname, 'helper.js'), 'utf-8');
const helperInjection = '<script>\n' + helperScript + '\n</script>';
function isFullDocument(html) {
const trimmed = html.trimStart().toLowerCase();
return trimmed.startsWith('<!doctype') || trimmed.startsWith('<html');
}
function wrapInFrame(content) {
return frameTemplate.replace('<!-- CONTENT -->', content);
}
function getNewestScreen() {
const files = fs.readdirSync(SCREEN_DIR)
.filter(f => f.endsWith('.html'))
.map(f => {
const fp = path.join(SCREEN_DIR, f);
return { path: fp, mtime: fs.statSync(fp).mtime.getTime() };
})
.sort((a, b) => b.mtime - a.mtime);
return files.length > 0 ? files[0].path : null;
}
```
- [ ] **Step 3: Add HTTP request handler**
```js
function handleRequest(req, res) {
if (req.method === 'GET' && req.url === '/') {
const screenFile = getNewestScreen();
let html = screenFile
? (raw => isFullDocument(raw) ? raw : wrapInFrame(raw))(fs.readFileSync(screenFile, 'utf-8'))
: WAITING_PAGE;
if (html.includes('</body>')) {
html = html.replace('</body>', helperInjection + '\n</body>');
} else {
html += helperInjection;
}
res.writeHead(200, { 'Content-Type': 'text/html' });
res.end(html);
} else if (req.method === 'GET' && req.url.startsWith('/files/')) {
const fileName = req.url.slice(7); // strip '/files/'
const filePath = path.join(SCREEN_DIR, path.basename(fileName));
if (!fs.existsSync(filePath)) {
res.writeHead(404);
res.end('Not found');
return;
}
const ext = path.extname(filePath).toLowerCase();
const contentType = MIME_TYPES[ext] || 'application/octet-stream';
res.writeHead(200, { 'Content-Type': contentType });
res.end(fs.readFileSync(filePath));
} else {
res.writeHead(404);
res.end('Not found');
}
}
```
- [ ] **Step 4: Add WebSocket connection handling**
```js
const clients = new Set();
function handleUpgrade(req, socket) {
const key = req.headers['sec-websocket-key'];
if (!key) { socket.destroy(); return; }
const accept = computeAcceptKey(key);
socket.write(
'HTTP/1.1 101 Switching Protocols\r\n' +
'Upgrade: websocket\r\n' +
'Connection: Upgrade\r\n' +
'Sec-WebSocket-Accept: ' + accept + '\r\n\r\n'
);
let buffer = Buffer.alloc(0);
clients.add(socket);
socket.on('data', (chunk) => {
buffer = Buffer.concat([buffer, chunk]);
while (buffer.length > 0) {
let result;
try {
result = decodeFrame(buffer);
} catch (e) {
socket.end(encodeFrame(OPCODES.CLOSE, Buffer.alloc(0)));
clients.delete(socket);
return;
}
if (!result) break;
buffer = buffer.slice(result.bytesConsumed);
switch (result.opcode) {
case OPCODES.TEXT:
handleMessage(result.payload.toString());
break;
case OPCODES.CLOSE:
socket.end(encodeFrame(OPCODES.CLOSE, Buffer.alloc(0)));
clients.delete(socket);
return;
case OPCODES.PING:
socket.write(encodeFrame(OPCODES.PONG, result.payload));
break;
case OPCODES.PONG:
break;
default:
// Unsupported opcode — close with 1003
const closeBuf = Buffer.alloc(2);
closeBuf.writeUInt16BE(1003);
socket.end(encodeFrame(OPCODES.CLOSE, closeBuf));
clients.delete(socket);
return;
}
}
});
socket.on('close', () => clients.delete(socket));
socket.on('error', () => clients.delete(socket));
}
function handleMessage(text) {
let event;
try {
event = JSON.parse(text);
} catch (e) {
console.error('Failed to parse WebSocket message:', e.message);
return;
}
console.log(JSON.stringify({ source: 'user-event', ...event }));
if (event.choice) {
const eventsFile = path.join(SCREEN_DIR, '.events');
fs.appendFileSync(eventsFile, JSON.stringify(event) + '\n');
}
}
function broadcast(msg) {
const frame = encodeFrame(OPCODES.TEXT, Buffer.from(JSON.stringify(msg)));
for (const socket of clients) {
try { socket.write(frame); } catch (e) { clients.delete(socket); }
}
}
```
- [ ] **Step 5: Add debounce timer map**
```js
const debounceTimers = new Map();
```
File watching logic is inlined in `startServer` (Step 6) to keep watcher lifecycle together with server lifecycle and include an `error` handler per spec.
- [ ] **Step 6: Add startServer function and conditional main**
`frameTemplate` and `helperInjection` are already at module scope (Step 2). `startServer` just creates the screen dir, starts the HTTP server, watcher, and logs startup info.
```js
function startServer() {
if (!fs.existsSync(SCREEN_DIR)) fs.mkdirSync(SCREEN_DIR, { recursive: true });
const server = http.createServer(handleRequest);
server.on('upgrade', handleUpgrade);
const watcher = fs.watch(SCREEN_DIR, (eventType, filename) => {
if (!filename || !filename.endsWith('.html')) return;
if (debounceTimers.has(filename)) clearTimeout(debounceTimers.get(filename));
debounceTimers.set(filename, setTimeout(() => {
debounceTimers.delete(filename);
const filePath = path.join(SCREEN_DIR, filename);
if (eventType === 'rename' && fs.existsSync(filePath)) {
const eventsFile = path.join(SCREEN_DIR, '.events');
if (fs.existsSync(eventsFile)) fs.unlinkSync(eventsFile);
console.log(JSON.stringify({ type: 'screen-added', file: filePath }));
} else if (eventType === 'change') {
console.log(JSON.stringify({ type: 'screen-updated', file: filePath }));
}
broadcast({ type: 'reload' });
}, 100));
});
watcher.on('error', (err) => console.error('fs.watch error:', err.message));
server.listen(PORT, HOST, () => {
const info = JSON.stringify({
type: 'server-started', port: Number(PORT), host: HOST,
url_host: URL_HOST, url: 'http://' + URL_HOST + ':' + PORT,
screen_dir: SCREEN_DIR
});
console.log(info);
fs.writeFileSync(path.join(SCREEN_DIR, '.server-info'), info + '\n');
});
}
if (require.main === module) {
startServer();
}
```
- [ ] **Step 7: Run integration tests**
The test directory already has a `package.json` with `ws` as a dependency. Install it if needed, then run tests.
Run: `cd tests/brainstorm-server && npm install && node server.test.js`
Expected: All tests pass
- [ ] **Step 8: Commit**
```bash
git add skills/brainstorming/scripts/server.js
git commit -m "Add HTTP server, WebSocket handling, and file watching to server.js"
```
---
## Chunk 3: Swap and Cleanup
### Task 3: Update start-server.sh and remove old files
**Files:**
- Modify: `skills/brainstorming/scripts/start-server.sh:94,100`
- Modify: `.gitignore:6`
- Delete: `skills/brainstorming/scripts/index.js`
- Delete: `skills/brainstorming/scripts/package.json`
- Delete: `skills/brainstorming/scripts/package-lock.json`
- Delete: `skills/brainstorming/scripts/node_modules/` (entire directory)
- [ ] **Step 1: Update start-server.sh — change `index.js` to `server.js`**
Two lines to change:
Line 94: `env BRAINSTORM_DIR="$SCREEN_DIR" BRAINSTORM_HOST="$BIND_HOST" BRAINSTORM_URL_HOST="$URL_HOST" node server.js`
Line 100: `nohup env BRAINSTORM_DIR="$SCREEN_DIR" BRAINSTORM_HOST="$BIND_HOST" BRAINSTORM_URL_HOST="$URL_HOST" node server.js > "$LOG_FILE" 2>&1 &`
- [ ] **Step 2: Remove the gitignore exception for node_modules**
In `.gitignore`, delete line 6: `!skills/brainstorming/scripts/node_modules/`
- [ ] **Step 3: Delete old files**
```bash
git rm skills/brainstorming/scripts/index.js
git rm skills/brainstorming/scripts/package.json
git rm skills/brainstorming/scripts/package-lock.json
git rm -r skills/brainstorming/scripts/node_modules/
```
- [ ] **Step 4: Run both test suites**
Run: `cd tests/brainstorm-server && node ws-protocol.test.js && node server.test.js`
Expected: All tests pass
- [ ] **Step 5: Commit**
```bash
git add skills/brainstorming/scripts/ .gitignore
git commit -m "Remove vendored node_modules, swap to zero-dep server.js"
```
### Task 4: Manual smoke test
- [ ] **Step 1: Start the server manually**
```bash
cd skills/brainstorming/scripts
BRAINSTORM_DIR=/tmp/brainstorm-smoke BRAINSTORM_PORT=9876 node server.js
```
Expected: `server-started` JSON printed with port 9876
- [ ] **Step 2: Open browser to http://localhost:9876**
Expected: Waiting page with "Waiting for Claude to push a screen..."
- [ ] **Step 3: Write an HTML file to the screen directory**
```bash
echo '<h2>Hello from smoke test</h2>' > /tmp/brainstorm-smoke/test.html
```
Expected: Browser reloads and shows "Hello from smoke test" wrapped in frame template
- [ ] **Step 4: Verify WebSocket works — check browser console**
Open browser dev tools. The WebSocket connection should show as connected (no errors in console). The frame template's status indicator should show "Connected".
- [ ] **Step 5: Stop server with Ctrl-C, clean up**
```bash
rm -rf /tmp/brainstorm-smoke
```

View File

@@ -1,566 +0,0 @@
# Codex App Compatibility Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make `using-git-worktrees`, `finishing-a-development-branch`, and related skills work in the Codex App's sandboxed worktree environment without breaking existing behavior.
**Architecture:** Read-only environment detection (`git-dir` vs `git-common-dir`) at the start of two skills. If already in a linked worktree, skip creation. If on detached HEAD, emit a handoff payload instead of the 4-option menu. Sandbox fallback catches permission errors during worktree creation.
**Tech Stack:** Git, Markdown (skill files are instruction documents, not executable code)
**Spec:** `docs/superpowers/specs/2026-03-23-codex-app-compatibility-design.md`
---
## File Structure
| File | Responsibility | Action |
|---|---|---|
| `skills/using-git-worktrees/SKILL.md` | Worktree creation + isolation | Add Step 0 detection + sandbox fallback |
| `skills/finishing-a-development-branch/SKILL.md` | Branch finishing workflow | Add Step 1.5 detection + cleanup guard |
| `skills/subagent-driven-development/SKILL.md` | Plan execution with subagents | Update Integration description |
| `skills/executing-plans/SKILL.md` | Plan execution inline | Update Integration description |
| `skills/using-superpowers/references/codex-tools.md` | Codex platform reference | Add detection + finishing docs |
---
### Task 1: Add Step 0 to `using-git-worktrees`
**Files:**
- Modify: `skills/using-git-worktrees/SKILL.md:14-15` (insert after Overview, before Directory Selection Process)
- [ ] **Step 1: Read the current skill file**
Read `skills/using-git-worktrees/SKILL.md` in full. Identify the exact insertion point: after the "Announce at start" line (line 14) and before "## Directory Selection Process" (line 16).
- [ ] **Step 2: Insert Step 0 section**
Insert the following between the Overview section and "## Directory Selection Process":
```markdown
## Step 0: Check if Already in an Isolated Workspace
Before creating a worktree, check if one already exists:
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
```
**If `GIT_DIR` differs from `GIT_COMMON`:** You are already inside a linked worktree (created by the Codex App, Claude Code's Agent tool, a previous skill run, or the user). Do NOT create another worktree. Instead:
1. Run project setup (auto-detect package manager as in "Run Project Setup" below)
2. Verify clean baseline (run tests as in "Verify Clean Baseline" below)
3. Report with branch state:
- On a branch: "Already in an isolated workspace at `<path>` on branch `<name>`. Tests passing. Ready to implement."
- Detached HEAD: "Already in an isolated workspace at `<path>` (detached HEAD, externally managed). Tests passing. Note: branch creation needed at finish time. Ready to implement."
After reporting, STOP. Do not continue to Directory Selection or Creation Steps.
**If `GIT_DIR` equals `GIT_COMMON`:** Proceed with the full worktree creation flow below.
**Sandbox fallback:** If you proceed to Creation Steps but `git worktree add -b` fails with a permission error (e.g., "Operation not permitted"), treat this as a late-detected restricted environment. Fall back to the behavior above — run setup and baseline tests in the current directory, report accordingly, and STOP.
```
- [ ] **Step 3: Verify the insertion**
Read the file again. Confirm:
- Step 0 appears between Overview and Directory Selection Process
- The rest of the file (Directory Selection, Safety Verification, Creation Steps, etc.) is unchanged
- No duplicate sections or broken markdown
- [ ] **Step 4: Commit**
```bash
git add skills/using-git-worktrees/SKILL.md
git commit -m "feat(using-git-worktrees): add Step 0 environment detection (PRI-823)
Skip worktree creation when already in a linked worktree. Includes
sandbox fallback for permission errors on git worktree add."
```
---
### Task 2: Update `using-git-worktrees` Integration section
**Files:**
- Modify: `skills/using-git-worktrees/SKILL.md:211-215` (Integration > Called by)
- [ ] **Step 1: Update the three "Called by" entries**
Change lines 212-214 from:
```markdown
- **brainstorming** (Phase 4) - REQUIRED when design is approved and implementation follows
- **subagent-driven-development** - REQUIRED before executing any tasks
- **executing-plans** - REQUIRED before executing any tasks
```
To:
```markdown
- **brainstorming** - REQUIRED: Ensures isolated workspace (creates one or verifies existing)
- **subagent-driven-development** - REQUIRED: Ensures isolated workspace (creates one or verifies existing)
- **executing-plans** - REQUIRED: Ensures isolated workspace (creates one or verifies existing)
```
- [ ] **Step 2: Verify the Integration section**
Read the Integration section. Confirm all three entries are updated, "Pairs with" is unchanged.
- [ ] **Step 3: Commit**
```bash
git add skills/using-git-worktrees/SKILL.md
git commit -m "docs(using-git-worktrees): update Integration descriptions (PRI-823)
Clarify that skill ensures a workspace exists, not that it always creates one."
```
---
### Task 3: Add Step 1.5 to `finishing-a-development-branch`
**Files:**
- Modify: `skills/finishing-a-development-branch/SKILL.md:38` (insert after Step 1, before Step 2)
- [ ] **Step 1: Read the current skill file**
Read `skills/finishing-a-development-branch/SKILL.md` in full. Identify the insertion point: after "**If tests pass:** Continue to Step 2." (line 38) and before "### Step 2: Determine Base Branch" (line 40).
- [ ] **Step 2: Insert Step 1.5 section**
Insert the following between Step 1 and Step 2:
```markdown
### Step 1.5: Detect Environment
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
```
**Path A — `GIT_DIR` differs from `GIT_COMMON` AND `BRANCH` is empty (externally managed worktree, detached HEAD):**
First, ensure all work is staged and committed (`git add` + `git commit`).
Then present this to the user (do NOT present the 4-option menu):
```
Implementation complete. All tests passing.
Current HEAD: <full-commit-sha>
This workspace is externally managed (detached HEAD).
I cannot create branches, push, or open PRs from here.
⚠ These commits are on a detached HEAD. If you do not create a branch,
they may be lost when this workspace is cleaned up.
If your host application provides these controls:
- "Create branch" — to name a branch, then commit/push/PR
- "Hand off to local" — to move changes to your local checkout
Suggested branch name: <ticket-id/short-description>
Suggested commit message: <summary-of-work>
```
Branch name: use ticket ID if available (e.g., `pri-823/codex-compat`), otherwise slugify the first 5 words of the plan title, otherwise omit. Avoid sensitive content in branch names.
Skip to Step 5 (cleanup is a no-op — see guard below).
**Path B — `GIT_DIR` differs from `GIT_COMMON` AND `BRANCH` exists (externally managed worktree, named branch):**
Proceed to Step 2 and present the 4-option menu as normal.
**Path C — `GIT_DIR` equals `GIT_COMMON` (normal environment):**
Proceed to Step 2 and present the 4-option menu as normal.
```
- [ ] **Step 3: Verify the insertion**
Read the file again. Confirm:
- Step 1.5 appears between Step 1 and Step 2
- Steps 2-5 are unchanged
- Path A handoff includes commit SHA and data loss warning
- Paths B and C proceed to Step 2 normally
- [ ] **Step 4: Commit**
```bash
git add skills/finishing-a-development-branch/SKILL.md
git commit -m "feat(finishing-a-development-branch): add Step 1.5 environment detection (PRI-823)
Detect externally managed worktrees with detached HEAD and emit handoff
payload instead of 4-option menu. Includes commit SHA and data loss warning."
```
---
### Task 4: Add Step 5 cleanup guard to `finishing-a-development-branch`
**Files:**
- Modify: `skills/finishing-a-development-branch/SKILL.md` (Step 5: Cleanup Worktree — find by section heading, line numbers will have shifted after Task 3)
- [ ] **Step 1: Read the current Step 5 section**
Find the "### Step 5: Cleanup Worktree" section in `skills/finishing-a-development-branch/SKILL.md` (line numbers will have shifted after Task 3's insertion). The current Step 5 is:
```markdown
### Step 5: Cleanup Worktree
**For Options 1, 2, 4:**
Check if in worktree:
```bash
git worktree list | grep $(git branch --show-current)
```
If yes:
```bash
git worktree remove <worktree-path>
```
**For Option 3:** Keep worktree.
```
- [ ] **Step 2: Add the cleanup guard before existing logic**
Replace the Step 5 section with:
```markdown
### Step 5: Cleanup Worktree
**First, check if worktree is externally managed:**
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
```
If `GIT_DIR` differs from `GIT_COMMON`: skip worktree removal — the host environment owns this workspace.
**Otherwise, for Options 1 and 4:**
Check if in worktree:
```bash
git worktree list | grep $(git branch --show-current)
```
If yes:
```bash
git worktree remove <worktree-path>
```
**For Option 3:** Keep worktree.
```
Note: the original text said "For Options 1, 2, 4" but the Quick Reference table and Common Mistakes section say "Options 1 & 4 only." This edit aligns Step 5 with those sections.
- [ ] **Step 3: Verify the replacement**
Read Step 5. Confirm:
- Cleanup guard (re-detection) appears first
- Existing removal logic preserved for non-externally-managed worktrees
- "Options 1 and 4" (not "1, 2, 4") matches Quick Reference and Common Mistakes
- [ ] **Step 4: Commit**
```bash
git add skills/finishing-a-development-branch/SKILL.md
git commit -m "feat(finishing-a-development-branch): add Step 5 cleanup guard (PRI-823)
Re-detect externally managed worktree at cleanup time and skip removal.
Also fixes pre-existing inconsistency: cleanup now correctly says
Options 1 and 4 only, matching Quick Reference and Common Mistakes."
```
---
### Task 5: Update Integration lines in `subagent-driven-development` and `executing-plans`
**Files:**
- Modify: `skills/subagent-driven-development/SKILL.md:268`
- Modify: `skills/executing-plans/SKILL.md:68`
- [ ] **Step 1: Update `subagent-driven-development`**
Change line 268 from:
```
- **superpowers:using-git-worktrees** - REQUIRED: Set up isolated workspace before starting
```
To:
```
- **superpowers:using-git-worktrees** - REQUIRED: Ensures isolated workspace (creates one or verifies existing)
```
- [ ] **Step 2: Update `executing-plans`**
Change line 68 from:
```
- **superpowers:using-git-worktrees** - REQUIRED: Set up isolated workspace before starting
```
To:
```
- **superpowers:using-git-worktrees** - REQUIRED: Ensures isolated workspace (creates one or verifies existing)
```
- [ ] **Step 3: Verify both files**
Read line 268 of `skills/subagent-driven-development/SKILL.md` and line 68 of `skills/executing-plans/SKILL.md`. Confirm both say "Ensures isolated workspace (creates one or verifies existing)".
- [ ] **Step 4: Commit**
```bash
git add skills/subagent-driven-development/SKILL.md skills/executing-plans/SKILL.md
git commit -m "docs(sdd, executing-plans): update worktree Integration descriptions (PRI-823)
Clarify that using-git-worktrees ensures a workspace exists rather than
always creating one."
```
---
### Task 6: Add environment detection docs to `codex-tools.md`
**Files:**
- Modify: `skills/using-superpowers/references/codex-tools.md:25` (append at end)
- [ ] **Step 1: Read the current file**
Read `skills/using-superpowers/references/codex-tools.md` in full. Confirm it ends at line 25-26 after the multi_agent section.
- [ ] **Step 2: Append two new sections**
Add at the end of the file:
```markdown
## Environment Detection
Skills that create worktrees or finish branches should detect their
environment with read-only git commands before proceeding:
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
```
- `GIT_DIR != GIT_COMMON` → already in a linked worktree (skip creation)
- `BRANCH` empty → detached HEAD (cannot branch/push/PR from sandbox)
See `using-git-worktrees` Step 0 and `finishing-a-development-branch`
Step 1.5 for how each skill uses these signals.
## Codex App Finishing
When the sandbox blocks branch/push operations (detached HEAD in an
externally managed worktree), the agent commits all work and informs
the user to use the App's native controls:
- **"Create branch"** — names the branch, then commit/push/PR via App UI
- **"Hand off to local"** — transfers work to the user's local checkout
The agent can still run tests, stage files, and output suggested branch
names, commit messages, and PR descriptions for the user to copy.
```
- [ ] **Step 3: Verify the additions**
Read the full file. Confirm:
- Two new sections appear after the existing content
- Bash code block renders correctly (not escaped)
- Cross-references to Step 0 and Step 1.5 are present
- [ ] **Step 4: Commit**
```bash
git add skills/using-superpowers/references/codex-tools.md
git commit -m "docs(codex-tools): add environment detection and App finishing docs (PRI-823)
Document the git-dir vs git-common-dir detection pattern and the Codex
App's native finishing flow for skills that need to adapt."
```
---
### Task 7: Automated test — environment detection
**Files:**
- Create: `tests/codex-app-compat/test-environment-detection.sh`
- [ ] **Step 1: Create test directory**
```bash
mkdir -p tests/codex-app-compat
```
- [ ] **Step 2: Write the detection test script**
Create `tests/codex-app-compat/test-environment-detection.sh`:
```bash
#!/usr/bin/env bash
set -euo pipefail
# Test environment detection logic from PRI-823
# Tests the git-dir vs git-common-dir comparison used by
# using-git-worktrees Step 0 and finishing-a-development-branch Step 1.5
PASS=0
FAIL=0
TEMP_DIR=$(mktemp -d)
trap "rm -rf $TEMP_DIR" EXIT
log_pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
log_fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
# Helper: run detection and return "linked" or "normal"
detect_worktree() {
local git_dir git_common
git_dir=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
git_common=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
if [ "$git_dir" != "$git_common" ]; then
echo "linked"
else
echo "normal"
fi
}
echo "=== Test 1: Normal repo detection ==="
cd "$TEMP_DIR"
git init test-repo > /dev/null 2>&1
cd test-repo
git commit --allow-empty -m "init" > /dev/null 2>&1
result=$(detect_worktree)
if [ "$result" = "normal" ]; then
log_pass "Normal repo detected as normal"
else
log_fail "Normal repo detected as '$result' (expected 'normal')"
fi
echo "=== Test 2: Linked worktree detection ==="
git worktree add "$TEMP_DIR/test-wt" -b test-branch > /dev/null 2>&1
cd "$TEMP_DIR/test-wt"
result=$(detect_worktree)
if [ "$result" = "linked" ]; then
log_pass "Linked worktree detected as linked"
else
log_fail "Linked worktree detected as '$result' (expected 'linked')"
fi
echo "=== Test 3: Detached HEAD detection ==="
git checkout --detach HEAD > /dev/null 2>&1
branch=$(git branch --show-current)
if [ -z "$branch" ]; then
log_pass "Detached HEAD: branch is empty"
else
log_fail "Detached HEAD: branch is '$branch' (expected empty)"
fi
echo "=== Test 4: Linked worktree + detached HEAD (Codex App simulation) ==="
result=$(detect_worktree)
branch=$(git branch --show-current)
if [ "$result" = "linked" ] && [ -z "$branch" ]; then
log_pass "Codex App simulation: linked + detached HEAD"
else
log_fail "Codex App simulation: result='$result', branch='$branch'"
fi
echo "=== Test 5: Cleanup guard — linked worktree should NOT remove ==="
cd "$TEMP_DIR/test-wt"
result=$(detect_worktree)
if [ "$result" = "linked" ]; then
log_pass "Cleanup guard: linked worktree correctly detected (would skip removal)"
else
log_fail "Cleanup guard: expected 'linked', got '$result'"
fi
echo "=== Test 6: Cleanup guard — main repo SHOULD remove ==="
cd "$TEMP_DIR/test-repo"
result=$(detect_worktree)
if [ "$result" = "normal" ]; then
log_pass "Cleanup guard: main repo correctly detected (would proceed with removal)"
else
log_fail "Cleanup guard: expected 'normal', got '$result'"
fi
# Cleanup worktree before temp dir removal
cd "$TEMP_DIR/test-repo"
git worktree remove "$TEMP_DIR/test-wt" > /dev/null 2>&1 || true
echo ""
echo "=== Results: $PASS passed, $FAIL failed ==="
if [ "$FAIL" -gt 0 ]; then
exit 1
fi
```
- [ ] **Step 3: Make it executable and run it**
```bash
chmod +x tests/codex-app-compat/test-environment-detection.sh
./tests/codex-app-compat/test-environment-detection.sh
```
Expected output: 6 passed, 0 failed.
- [ ] **Step 4: Commit**
```bash
git add tests/codex-app-compat/test-environment-detection.sh
git commit -m "test: add environment detection tests for Codex App compat (PRI-823)
Tests git-dir vs git-common-dir comparison in normal repo, linked
worktree, detached HEAD, and cleanup guard scenarios."
```
---
### Task 8: Final verification
**Files:**
- Read: all 5 modified skill files
- [ ] **Step 1: Run the automated detection tests**
```bash
./tests/codex-app-compat/test-environment-detection.sh
```
Expected: 6 passed, 0 failed.
- [ ] **Step 2: Read each modified file and verify changes**
Read each file end-to-end:
- `skills/using-git-worktrees/SKILL.md` — Step 0 present, rest unchanged
- `skills/finishing-a-development-branch/SKILL.md` — Step 1.5 present, cleanup guard present, rest unchanged
- `skills/subagent-driven-development/SKILL.md` — line 268 updated
- `skills/executing-plans/SKILL.md` — line 68 updated
- `skills/using-superpowers/references/codex-tools.md` — two new sections at end
- [ ] **Step 3: Verify no unintended changes**
```bash
git diff --stat HEAD~7
```
Should show exactly 6 files changed (5 skill files + 1 test file). No other files modified.
- [ ] **Step 4: Run existing test suite**
If test runner exists:
```bash
# Run skill-triggering tests
# Note: tests/skill-triggering/ was lifted into drill scenarios on 2026-05-06.
# See evals/scenarios/triggering-*.yaml. The reference below is a dated artifact.
./tests/skill-triggering/run-all.sh 2>/dev/null || echo "Skill triggering tests not available in this environment"
# Run SDD integration test
./tests/claude-code/test-subagent-driven-development-integration.sh 2>/dev/null || echo "SDD integration test not available in this environment"
```
Note: these tests require Claude Code with `--dangerously-skip-permissions`. If not available, document that regression tests should be run manually.

View File

@@ -1,866 +0,0 @@
# Worktree Rototill Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make superpowers defer to native harness worktree systems when available, fall back to manual git worktrees when not, and fix three known finishing bugs.
**Architecture:** Two skill files are rewritten (`using-git-worktrees`, `finishing-a-development-branch`), three files get one-line integration updates (`executing-plans`, `subagent-driven-development`, `writing-plans`). The core change is adding detection (`GIT_DIR != GIT_COMMON`) and a native-tool-first creation path. These are markdown skill instruction files, not application code — "tests" are agent behavior tests using the testing-skills-with-subagents TDD framework.
**Tech Stack:** Markdown (skill files), bash (test scripts), Claude Code CLI (`claude -p` for headless testing)
**Spec:** `docs/superpowers/specs/2026-04-06-worktree-rototill-design.md`
---
### Task 1: GATE — TDD Validation of Step 1a (Native Tool Preference)
Step 1a is the load-bearing assumption of the entire design. If agents don't prefer native worktree tools over `git worktree add`, the spec fails. Validate this FIRST, before touching any skill files.
**Files:**
- Create: `tests/claude-code/test-worktree-native-preference.sh`
- Read: `skills/using-git-worktrees/SKILL.md` (current version, for RED baseline)
- Read: `tests/claude-code/test-helpers.sh` (for `run_claude`, `assert_contains`, etc.)
- Read: `skills/writing-skills/testing-skills-with-subagents.md` (TDD framework)
**This task is a gate.** If the GREEN phase fails after 2 REFACTOR iterations, STOP. Do not proceed to Task 2. Report back — the creation approach needs redesign.
- [ ] **Step 1: Write the RED baseline test script**
Create the test script that will run scenarios both WITHOUT and WITH the updated skill text. The RED phase runs against the current skill (which has no Step 1a).
```bash
#!/usr/bin/env bash
# Test: Does the agent prefer native worktree tools (EnterWorktree) over git worktree add?
# Framework: RED-GREEN-REFACTOR per testing-skills-with-subagents.md
#
# RED: Current skill has no native tool preference. Agent should use git worktree add.
# GREEN: Updated skill has Step 1a. Agent should use EnterWorktree on Claude Code.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/test-helpers.sh"
# Pressure scenario: realistic implementation task where agent needs isolation
SCENARIO='IMPORTANT: This is a real task. Choose and act.
You need to implement a small feature (add a "version" field to package.json).
This should be done in an isolated workspace to protect the main branch.
You have the using-git-worktrees skill available. Set up the isolated workspace now.
Do NOT actually implement the feature — just set up the workspace and report what you did.
Respond with EXACTLY what tool/command you used to create the workspace.'
echo "=== Worktree Native Preference Test ==="
echo ""
# Phase selection
PHASE="${1:-red}"
if [ "$PHASE" = "red" ]; then
echo "--- RED PHASE: Running WITHOUT Step 1a (current skill) ---"
echo "Expected: Agent uses 'git worktree add' (no native tool awareness)"
echo ""
test_dir=$(create_test_project)
cd "$test_dir"
git init && git commit --allow-empty -m "init"
mkdir -p .worktrees
output=$(run_claude "$SCENARIO" 120)
echo "Agent output:"
echo "$output"
echo ""
# RED expectation: agent uses git worktree add (current behavior)
if echo "$output" | grep -qi "EnterWorktree"; then
echo "[UNEXPECTED] Agent used EnterWorktree WITHOUT Step 1a — skill may not be needed"
echo "Investigate: is Claude Code's default behavior already correct?"
else
echo "[RED CONFIRMED] Agent did NOT use EnterWorktree"
assert_contains "$output" "git worktree" "Agent used git worktree (expected in RED)"
fi
cleanup_test_project "$test_dir"
elif [ "$PHASE" = "green" ]; then
echo "--- GREEN PHASE: Running WITH Step 1a (updated skill) ---"
echo "Expected: Agent uses EnterWorktree instead of git worktree add"
echo ""
test_dir=$(create_test_project)
cd "$test_dir"
git init && git commit --allow-empty -m "init"
output=$(run_claude "$SCENARIO" 120)
echo "Agent output:"
echo "$output"
echo ""
# GREEN expectation: agent uses EnterWorktree
assert_contains "$output" "EnterWorktree" "Agent used native EnterWorktree tool"
assert_not_contains "$output" "git worktree add" "Agent did NOT fall back to git worktree add"
cleanup_test_project "$test_dir"
elif [ "$PHASE" = "pressure" ]; then
echo "--- PRESSURE PHASE: Multiple pressures to skip native tools ---"
echo ""
PRESSURE_SCENARIO='IMPORTANT: This is a real task. Choose and act.
You need to set up an isolated workspace URGENTLY. Production is impacted and
the fix needs to go in a worktree. You know git worktree add works reliably.
The .worktrees/ directory already exists and is gitignored.
You have the using-git-worktrees skill. Set up the workspace NOW.
Speed matters. Report EXACTLY what tool/command you used.'
test_dir=$(create_test_project)
cd "$test_dir"
git init && git commit --allow-empty -m "init"
mkdir -p .worktrees
echo ".worktrees/" >> .gitignore
output=$(run_claude "$PRESSURE_SCENARIO" 120)
echo "Agent output:"
echo "$output"
echo ""
# Should STILL use EnterWorktree even under pressure
assert_contains "$output" "EnterWorktree" "Agent used native tool even under time pressure"
assert_not_contains "$output" "git worktree add" "Agent resisted falling back to git despite pressure"
cleanup_test_project "$test_dir"
fi
echo ""
echo "=== Test Complete ==="
```
- [ ] **Step 2: Run RED phase — confirm agent uses git worktree add today**
Run: `cd tests/claude-code && bash test-worktree-native-preference.sh red`
Expected: `[RED CONFIRMED] Agent did NOT use EnterWorktree` — agent uses `git worktree add` because current skill has no native tool preference.
Document the agent's exact output and any rationalizations verbatim. This is the baseline failure the skill must fix.
- [ ] **Step 3: If RED confirmed, proceed. Write the Step 1a skill text.**
Create a temporary test version of the skill with ONLY the Step 1a addition (minimal change to isolate the variable). Add this section to the top of the skill's creation instructions, BEFORE the existing directory selection process:
```markdown
## Step 1: Create Isolated Workspace
**You have two mechanisms. Try them in this order.**
### 1a. Native Worktree Tools (preferred)
If your platform provides a worktree or workspace-isolation tool, use it. You know your own toolkit — the skill does not need to name specific tools. Native tools handle directory placement, branch creation, and cleanup automatically.
After using a native tool, skip to Step 3 (Project Setup).
### 1b. Git Worktree Fallback
If no native tool is available, create a worktree manually using git.
```
- [ ] **Step 4: Run GREEN phase — confirm agent now uses EnterWorktree**
Run: `cd tests/claude-code && bash test-worktree-native-preference.sh green`
Expected: `[PASS] Agent used native EnterWorktree tool`
If FAIL: Document the agent's exact output and rationalizations. This is a REFACTOR signal — the Step 1a text needs revision. Try up to 2 REFACTOR iterations. If still failing after 2 iterations, STOP and report back.
- [ ] **Step 5: Run PRESSURE phase — confirm agent resists fallback under pressure**
Run: `cd tests/claude-code && bash test-worktree-native-preference.sh pressure`
Expected: `[PASS] Agent used native tool even under time pressure`
If FAIL: Document rationalizations verbatim. Add explicit counters to Step 1a text (e.g., a Red Flag entry: "Never use git worktree add when your platform provides a native worktree tool"). Re-run.
- [ ] **Step 6: Commit test script**
```bash
git add tests/claude-code/test-worktree-native-preference.sh
git commit -m "test: add RED/GREEN validation for native worktree preference (PRI-974)
Gate test for Step 1a — validates agents prefer EnterWorktree over
git worktree add on Claude Code. Must pass before skill rewrite."
```
---
### Task 2: Rewrite `using-git-worktrees` SKILL.md
Full rewrite of the creation skill. Replaces the existing file entirely.
**Files:**
- Modify: `skills/using-git-worktrees/SKILL.md` (full rewrite, 219 lines → ~210 lines)
**Depends on:** Task 1 GREEN passing.
- [ ] **Step 1: Write the complete new SKILL.md**
Replace the entire contents of `skills/using-git-worktrees/SKILL.md` with:
```markdown
---
name: using-git-worktrees
description: Use when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback
---
# Using Git Worktrees
## Overview
Ensure work happens in an isolated workspace. Prefer your platform's native worktree tools. Fall back to manual git worktrees only when no native tool is available.
**Core principle:** Detect existing isolation first. Then use native tools. Then fall back to git. Never fight the harness.
**Announce at start:** "I'm using the using-git-worktrees skill to set up an isolated workspace."
## Step 0: Detect Existing Isolation
**Before creating anything, check if you are already in an isolated workspace.**
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
```
**Submodule guard:** `GIT_DIR != GIT_COMMON` is also true inside git submodules. Before concluding "already in a worktree," verify you are not in a submodule:
```bash
# If this returns a path, you're in a submodule, not a worktree — proceed to Step 1
git rev-parse --show-superproject-working-tree 2>/dev/null
```
**If `GIT_DIR != GIT_COMMON` (and not a submodule):** You are already in a linked worktree. Skip to Step 3 (Project Setup). Do NOT create another worktree.
Report with branch state:
- On a branch: "Already in isolated workspace at `<path>` on branch `<name>`."
- Detached HEAD: "Already in isolated workspace at `<path>` (detached HEAD, externally managed). Branch creation needed at finish time."
**If `GIT_DIR == GIT_COMMON` (or in a submodule):** You are in a normal repo checkout.
Has the user already indicated their worktree preference in your instructions? If not, ask for consent before creating a worktree:
> "Would you like me to set up an isolated worktree? It protects your current branch from changes."
Honor any existing declared preference without asking. If the user declines consent, work in place and skip to Step 3.
## Step 1: Create Isolated Workspace
**You have two mechanisms. Try them in this order.**
### 1a. Native Worktree Tools (preferred)
If your platform provides a worktree or workspace-isolation tool, use it. You know your own toolkit — the skill does not need to name specific tools. Native tools handle directory placement, branch creation, and cleanup automatically.
After using a native tool, skip to Step 3 (Project Setup).
### 1b. Git Worktree Fallback
If no native tool is available, create a worktree manually using git.
#### Directory Selection
Follow this priority order:
1. **Check your instructions for a worktree directory preference.** If specified, use it without asking.
2. **Check existing project-local directories:**
```bash
ls -d .worktrees 2>/dev/null # Preferred (hidden)
ls -d worktrees 2>/dev/null # Alternative
```
If found, use that directory. If both exist, `.worktrees` wins.
3. **Default to `.worktrees/`.**
#### Safety Verification (project-local directories only)
**MUST verify directory is ignored before creating worktree:**
```bash
git check-ignore -q .worktrees 2>/dev/null || git check-ignore -q worktrees 2>/dev/null
```
**If NOT ignored:** Add to .gitignore, commit the change, then proceed.
**Why critical:** Prevents accidentally committing worktree contents to repository.
#### Create the Worktree
```bash
# Determine path based on chosen location
path="$LOCATION/$BRANCH_NAME"
git worktree add "$path" -b "$BRANCH_NAME"
cd "$path"
```
#### Hooks Awareness
Git worktrees do not inherit the parent repo's hooks directory. After creating the worktree, symlink hooks from the main repo if they exist:
```bash
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
if [ -d "$MAIN_ROOT/.git/hooks" ]; then
ln -sf "$MAIN_ROOT/.git/hooks" "$path/.git/hooks"
fi
```
This prevents pre-commit checks, linters, and other hooks from silently stopping when work moves to a worktree.
**Sandbox fallback:** If `git worktree add` fails with a permission error (sandbox denial), treat this as a restricted environment. Skip creation, run setup and baseline tests in the current directory, report accordingly.
## Step 3: Project Setup
Auto-detect and run appropriate setup:
```bash
# Node.js
if [ -f package.json ]; then npm install; fi
# Rust
if [ -f Cargo.toml ]; then cargo build; fi
# Python
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
if [ -f pyproject.toml ]; then poetry install; fi
# Go
if [ -f go.mod ]; then go mod download; fi
```
## Step 4: Verify Clean Baseline
Run tests to ensure workspace starts clean:
```bash
# Use project-appropriate command
npm test / cargo test / pytest / go test ./...
```
**If tests fail:** Report failures, ask whether to proceed or investigate.
**If tests pass:** Report ready.
### Report
```
Worktree ready at <full-path>
Tests passing (<N> tests, 0 failures)
Ready to implement <feature-name>
```
## Quick Reference
| Situation | Action |
|-----------|--------|
| Already in linked worktree | Skip creation (Step 0) |
| In a submodule | Treat as normal repo (Step 0 guard) |
| Native worktree tool available | Use it (Step 1a) |
| No native tool | Git worktree fallback (Step 1b) |
| `.worktrees/` exists | Use it (verify ignored) |
| `worktrees/` exists | Use it (verify ignored) |
| Both exist | Use `.worktrees/` |
| Neither exists | Check instruction file, then default `.worktrees/` |
| Directory not ignored | Add to .gitignore + commit |
| Permission error on create | Sandbox fallback, work in place |
| Tests fail during baseline | Report failures + ask |
| No package.json/Cargo.toml | Skip dependency install |
## Common Mistakes
### Fighting the harness
- **Problem:** Using `git worktree add` when the platform already provides isolation
- **Fix:** Step 0 detects existing isolation. Step 1a defers to native tools.
### Skipping detection
- **Problem:** Creating a nested worktree inside an existing one
- **Fix:** Always run Step 0 before creating anything
### Skipping ignore verification
- **Problem:** Worktree contents get tracked, pollute git status
- **Fix:** Always use `git check-ignore` before creating project-local worktree
### Assuming directory location
- **Problem:** Creates inconsistency, violates project conventions
- **Fix:** Follow priority: existing > instruction file > default
### Proceeding with failing tests
- **Problem:** Can't distinguish new bugs from pre-existing issues
- **Fix:** Report failures, get explicit permission to proceed
## Red Flags
**Never:**
- Create a worktree when Step 0 detects existing isolation
- Use git commands when a native worktree tool is available
- Create worktree without verifying it's ignored (project-local)
- Skip baseline test verification
- Proceed with failing tests without asking
**Always:**
- Run Step 0 detection first
- Prefer native tools over git fallback
- Follow directory priority: existing > instruction file > default
- Verify directory is ignored for project-local
- Auto-detect and run project setup
- Verify clean test baseline
- Symlink hooks after creating worktree via 1b
## Integration
**Called by:**
- **subagent-driven-development** - Ensures isolated workspace (creates one or verifies existing)
- **executing-plans** - Ensures isolated workspace (creates one or verifies existing)
- Any skill needing isolated workspace
**Pairs with:**
- **finishing-a-development-branch** - REQUIRED for cleanup after work complete
```
- [ ] **Step 2: Verify the file reads correctly**
Run: `wc -l skills/using-git-worktrees/SKILL.md`
Expected: Approximately 200-220 lines. Scan for any markdown formatting issues.
- [ ] **Step 3: Commit**
```bash
git add skills/using-git-worktrees/SKILL.md
git commit -m "feat: rewrite using-git-worktrees with detect-and-defer (PRI-974)
Step 0: GIT_DIR != GIT_COMMON detection (skip if already isolated)
Step 0 consent: opt-in prompt before creating worktree (#991)
Step 1a: native tool preference (short, first, declarative)
Step 1b: git worktree fallback with project-local directory policy
Submodule guard prevents false detection
Platform-neutral instruction file references (#1049)"
```
---
### Task 3: Rewrite `finishing-a-development-branch` SKILL.md
Full rewrite of the finishing skill. Adds environment detection, fixes three bugs, adds provenance-based cleanup.
**Files:**
- Modify: `skills/finishing-a-development-branch/SKILL.md` (full rewrite, 201 lines → ~220 lines)
- [ ] **Step 1: Write the complete new SKILL.md**
Replace the entire contents of `skills/finishing-a-development-branch/SKILL.md` with:
```markdown
---
name: finishing-a-development-branch
description: Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup
---
# Finishing a Development Branch
## Overview
Guide completion of development work by presenting clear options and handling chosen workflow.
**Core principle:** Verify tests → Detect environment → Present options → Execute choice → Clean up.
**Announce at start:** "I'm using the finishing-a-development-branch skill to complete this work."
## The Process
### Step 1: Verify Tests
**Before presenting options, verify tests pass:**
```bash
# Run project's test suite
npm test / cargo test / pytest / go test ./...
```
**If tests fail:**
```
Tests failing (<N> failures). Must fix before completing:
[Show failures]
Cannot proceed with merge/PR until tests pass.
```
Stop. Don't proceed to Step 2.
**If tests pass:** Continue to Step 2.
### Step 2: Detect Environment
**Determine workspace state before presenting options:**
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
```
This determines which menu to show and how cleanup works:
| State | Menu | Cleanup |
|-------|------|---------|
| `GIT_DIR == GIT_COMMON` (normal repo) | Standard 4 options | No worktree to clean up |
| `GIT_DIR != GIT_COMMON`, named branch | Standard 4 options | Provenance-based (see Step 6) |
| `GIT_DIR != GIT_COMMON`, detached HEAD | Reduced 3 options (no merge) | No cleanup (externally managed) |
### Step 3: Determine Base Branch
```bash
# Try common base branches
git merge-base HEAD main 2>/dev/null || git merge-base HEAD master 2>/dev/null
```
Or ask: "This branch split from main - is that correct?"
### Step 4: Present Options
**Normal repo and named-branch worktree — present exactly these 4 options:**
```
Implementation complete. What would you like to do?
1. Merge back to <base-branch> locally
2. Push and create a Pull Request
3. Keep the branch as-is (I'll handle it later)
4. Discard this work
Which option?
```
**Detached HEAD — present exactly these 3 options:**
```
Implementation complete. You're on a detached HEAD (externally managed workspace).
1. Push as new branch and create a Pull Request
2. Keep as-is (I'll handle it later)
3. Discard this work
Which option?
```
**Don't add explanation** - keep options concise.
### Step 5: Execute Choice
#### Option 1: Merge Locally
```bash
# Get main repo root for CWD safety
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
cd "$MAIN_ROOT"
# Merge first — verify success before removing anything
git checkout <base-branch>
git pull
git merge <feature-branch>
# Verify tests on merged result
<test command>
# Only after merge succeeds: remove worktree, then delete branch
# (See Step 6 for worktree cleanup)
git branch -d <feature-branch>
```
Then: Cleanup worktree (Step 6)
#### Option 2: Push and Create PR
```bash
# Push branch
git push -u origin <feature-branch>
# Create PR
gh pr create --title "<title>" --body "$(cat <<'EOF'
## Summary
<2-3 bullets of what changed>
## Test Plan
- [ ] <verification steps>
EOF
)"
```
**Do NOT clean up worktree** — user needs it alive to iterate on PR feedback.
#### Option 3: Keep As-Is
Report: "Keeping branch <name>. Worktree preserved at <path>."
**Don't cleanup worktree.**
#### Option 4: Discard
**Confirm first:**
```
This will permanently delete:
- Branch <name>
- All commits: <commit-list>
- Worktree at <path>
Type 'discard' to confirm.
```
Wait for exact confirmation.
If confirmed:
```bash
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
cd "$MAIN_ROOT"
```
Then: Cleanup worktree (Step 6), then force-delete branch:
```bash
git branch -D <feature-branch>
```
### Step 6: Cleanup Workspace
**Only runs for Options 1 and 4.** Options 2 and 3 always preserve the worktree.
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
WORKTREE_PATH=$(git rev-parse --show-toplevel)
```
**If `GIT_DIR == GIT_COMMON`:** Normal repo, no worktree to clean up. Done.
**If worktree path is under `.worktrees/` or `worktrees/`:** Superpowers created this worktree — we own cleanup.
```bash
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
cd "$MAIN_ROOT"
git worktree remove "$WORKTREE_PATH"
git worktree prune # Self-healing: clean up any stale registrations
```
**Otherwise:** The host environment (harness) owns this workspace. Do NOT remove it. If your platform provides a workspace-exit tool, use it. Otherwise, leave the workspace in place.
## Quick Reference
| Option | Merge | Push | Keep Worktree | Cleanup Branch |
|--------|-------|------|---------------|----------------|
| 1. Merge locally | yes | - | - | yes |
| 2. Create PR | - | yes | yes | - |
| 3. Keep as-is | - | - | yes | - |
| 4. Discard | - | - | - | yes (force) |
## Common Mistakes
**Skipping test verification**
- **Problem:** Merge broken code, create failing PR
- **Fix:** Always verify tests before offering options
**Open-ended questions**
- **Problem:** "What should I do next?" is ambiguous
- **Fix:** Present exactly 4 structured options (or 3 for detached HEAD)
**Cleaning up worktree for Option 2**
- **Problem:** Remove worktree user needs for PR iteration
- **Fix:** Only cleanup for Options 1 and 4
**Deleting branch before removing worktree**
- **Problem:** `git branch -d` fails because worktree still references the branch
- **Fix:** Merge first, remove worktree, then delete branch
**Running git worktree remove from inside the worktree**
- **Problem:** Command fails silently when CWD is inside the worktree being removed
- **Fix:** Always `cd` to main repo root before `git worktree remove`
**Cleaning up harness-owned worktrees**
- **Problem:** Removing a worktree the harness created causes phantom state
- **Fix:** Only clean up worktrees under `.worktrees/` or `worktrees/`
**No confirmation for discard**
- **Problem:** Accidentally delete work
- **Fix:** Require typed "discard" confirmation
## Red Flags
**Never:**
- Proceed with failing tests
- Merge without verifying tests on result
- Delete work without confirmation
- Force-push without explicit request
- Remove a worktree before confirming merge success
- Clean up worktrees you didn't create (provenance check)
- Run `git worktree remove` from inside the worktree
**Always:**
- Verify tests before offering options
- Detect environment before presenting menu
- Present exactly 4 options (or 3 for detached HEAD)
- Get typed confirmation for Option 4
- Clean up worktree for Options 1 & 4 only
- `cd` to main repo root before worktree removal
- Run `git worktree prune` after removal
## Integration
**Called by:**
- **subagent-driven-development** (Step 7) - After all tasks complete
- **executing-plans** (Step 5) - After all batches complete
**Pairs with:**
- **using-git-worktrees** - Cleans up worktree created by that skill
```
- [ ] **Step 2: Verify the file reads correctly**
Run: `wc -l skills/finishing-a-development-branch/SKILL.md`
Expected: Approximately 210-230 lines.
- [ ] **Step 3: Commit**
```bash
git add skills/finishing-a-development-branch/SKILL.md
git commit -m "feat: rewrite finishing-a-development-branch with detect-and-defer (PRI-974)
Step 2: environment detection (GIT_DIR != GIT_COMMON) before presenting menu
Detached HEAD: reduced 3-option menu (no merge from detached HEAD)
Provenance-based cleanup: .worktrees/ = ours, anything else = hands off
Bug #940: Option 2 no longer cleans up worktree
Bug #999: merge -> verify -> remove worktree -> delete branch
Bug #238: cd to main repo root before git worktree remove
Stale worktree pruning after removal (git worktree prune)"
```
---
### Task 4: Integration Updates
One-line changes to three files that reference `using-git-worktrees`.
**Files:**
- Modify: `skills/executing-plans/SKILL.md:68`
- Modify: `skills/subagent-driven-development/SKILL.md:268`
- Modify: `skills/writing-plans/SKILL.md:16`
- [ ] **Step 1: Update executing-plans integration line**
In `skills/executing-plans/SKILL.md`, change line 68 from:
```markdown
- **superpowers:using-git-worktrees** - REQUIRED: Set up isolated workspace before starting
```
to:
```markdown
- **superpowers:using-git-worktrees** - Ensures isolated workspace (creates one or verifies existing)
```
- [ ] **Step 2: Update subagent-driven-development integration line**
In `skills/subagent-driven-development/SKILL.md`, change line 268 from:
```markdown
- **superpowers:using-git-worktrees** - REQUIRED: Set up isolated workspace before starting
```
to:
```markdown
- **superpowers:using-git-worktrees** - Ensures isolated workspace (creates one or verifies existing)
```
- [ ] **Step 3: Update writing-plans context line**
In `skills/writing-plans/SKILL.md`, change line 16 from:
```markdown
**Context:** This should be run in a dedicated worktree (created by brainstorming skill).
```
to:
```markdown
**Context:** If working in an isolated worktree, it should have been created via the using-git-worktrees skill at execution time.
```
- [ ] **Step 4: Commit all three**
```bash
git add skills/executing-plans/SKILL.md skills/subagent-driven-development/SKILL.md skills/writing-plans/SKILL.md
git commit -m "fix: update worktree integration references across skills (PRI-974)
Remove REQUIRED language from executing-plans and subagent-driven-development.
Consent and detection now live inside using-git-worktrees itself.
Fix stale 'created by brainstorming' claim in writing-plans."
```
---
### Task 5: End-to-End Validation
Verify the full rewritten skills work together. Run the existing test suite plus manual verification.
**Files:**
- Read: `tests/claude-code/run-skill-tests.sh`
- Read: `skills/using-git-worktrees/SKILL.md` (verify final state)
- Read: `skills/finishing-a-development-branch/SKILL.md` (verify final state)
- [ ] **Step 1: Run existing test suite**
Run: `cd tests/claude-code && bash run-skill-tests.sh`
Expected: All existing tests pass. If any fail, investigate — the integration changes (Task 4) may have broken a content assertion.
- [ ] **Step 2: Re-run Step 1a GREEN test**
Run: `cd tests/claude-code && bash test-worktree-native-preference.sh green`
Expected: PASS — agent still uses EnterWorktree with the final skill text (not just the minimal Step 1a addition from Task 1).
- [ ] **Step 3: Manual verification — read both rewritten skills end-to-end**
Read `skills/using-git-worktrees/SKILL.md` and `skills/finishing-a-development-branch/SKILL.md` in their entirety. Check:
1. No references to old behavior (hardcoded `CLAUDE.md`, interactive directory prompt, "REQUIRED" language)
2. Step numbering is consistent within each file
3. Quick Reference tables match the prose
4. Integration sections cross-reference correctly
5. No markdown formatting issues
- [ ] **Step 4: Verify git status is clean**
Run: `git status`
Expected: Clean working tree. All changes committed across Tasks 1-4.
- [ ] **Step 5: Final commit if any fixups needed**
If manual verification found issues, fix them and commit:
```bash
git add -A
git commit -m "fix: address review findings in worktree skill rewrite (PRI-974)"
```
If no issues found, skip this step.

File diff suppressed because it is too large Load Diff

View File

@@ -1,118 +0,0 @@
# Zero-Dependency Brainstorm Server
Replace the brainstorm companion server's vendored node_modules (express, ws, chokidar — 714 tracked files) with a single zero-dependency `server.js` using only Node.js built-ins.
## Motivation
Vendoring node_modules into the git repo creates a supply chain risk: frozen dependencies don't get security patches, 714 files of third-party code are committed without audit, and modifications to vendored code look like normal commits. While the actual risk is low (localhost-only dev server), eliminating it is straightforward.
## Architecture
A single `server.js` file (~250-300 lines) using `http`, `crypto`, `fs`, and `path`. The file serves two roles:
- **When run directly** (`node server.js`): starts the HTTP/WebSocket server
- **When required** (`require('./server.js')`): exports WebSocket protocol functions for unit testing
### WebSocket Protocol
Implements RFC 6455 for text frames only:
**Handshake:** Compute `Sec-WebSocket-Accept` from client's `Sec-WebSocket-Key` using SHA-1 + the RFC 6455 magic GUID. Return 101 Switching Protocols.
**Frame decoding (client to server):** Handle three masked length encodings:
- Small: payload < 126 bytes
- Medium: 126-65535 bytes (16-bit extended)
- Large: > 65535 bytes (64-bit extended)
XOR-unmask payload using 4-byte mask key. Return `{ opcode, payload, bytesConsumed }` or `null` for incomplete buffers. Reject unmasked frames.
**Frame encoding (server to client):** Unmasked frames with the same three length encodings.
**Opcodes handled:** TEXT (0x01), CLOSE (0x08), PING (0x09), PONG (0x0A). Unrecognized opcodes get a close frame with status 1003 (Unsupported Data).
**Deliberately skipped:** Binary frames, fragmented messages, extensions (permessage-deflate), subprotocols. These are unnecessary for small JSON text messages between localhost clients. Extensions and subprotocols are negotiated in the handshake — by not advertising them, they are never active.
**Buffer accumulation:** Each connection maintains a buffer. On `data`, append and loop `decodeFrame` until it returns null or buffer is empty.
### HTTP Server
Three routes:
1. **`GET /`** — Serve newest `.html` from screen directory by mtime. Detect full documents vs fragments, wrap fragments in frame template, inject helper.js. Return `text/html`. When no `.html` files exist, serve a hardcoded waiting page ("Waiting for Claude to push a screen...") with helper.js injected.
2. **`GET /files/*`** — Serve static files from screen directory with MIME type lookup from a hardcoded extension map (html, css, js, png, jpg, gif, svg, json). Return 404 if not found.
3. **Everything else** — 404.
WebSocket upgrade handled via the `'upgrade'` event on the HTTP server, separate from the request handler.
### Configuration
Environment variables (all optional):
- `BRAINSTORM_PORT` — port to bind (default: random high port 49152-65535)
- `BRAINSTORM_HOST` — interface to bind (default: `127.0.0.1`)
- `BRAINSTORM_URL_HOST` — hostname for the URL in startup JSON (default: `localhost` when host is `127.0.0.1`, otherwise same as host)
- `BRAINSTORM_DIR` — screen directory path (default: `/tmp/brainstorm`)
### Startup Sequence
1. Create `SCREEN_DIR` if it doesn't exist (`mkdirSync` recursive)
2. Load frame template and helper.js from `__dirname`
3. Start HTTP server on configured host/port
4. Start `fs.watch` on `SCREEN_DIR`
5. On successful listen, log `server-started` JSON to stdout: `{ type, port, host, url_host, url, screen_dir }`
6. Write the same JSON to `SCREEN_DIR/.server-info` so agents can find connection details when stdout is hidden (background execution)
### Application-Level WebSocket Messages
When a TEXT frame arrives from a client:
1. Parse as JSON. If parsing fails, log to stderr and continue.
2. Log to stdout as `{ source: 'user-event', ...event }`.
3. If the event contains a `choice` property, append the JSON to `SCREEN_DIR/.events` (one line per event).
### File Watching
`fs.watch(SCREEN_DIR)` replaces chokidar. On HTML file events:
- On new file (`rename` event for a file that exists): delete `.events` file if present (`unlinkSync`), log `screen-added` to stdout as JSON
- On file change (`change` event): log `screen-updated` to stdout as JSON (do NOT clear `.events`)
- Both events: send `{ type: 'reload' }` to all connected WebSocket clients
Debounce per-filename with ~100ms timeout to prevent duplicate events (common on macOS and Linux).
### Error Handling
- Malformed JSON from WebSocket clients: log to stderr, continue
- Unhandled opcodes: close with status 1003
- Client disconnects: remove from broadcast set
- `fs.watch` errors: log to stderr, continue
- No graceful shutdown logic — shell scripts handle process lifecycle via SIGTERM
## What Changes
| Before | After |
|---|---|
| `index.js` + `package.json` + `package-lock.json` + 714 `node_modules` files | `server.js` (single file) |
| express, ws, chokidar dependencies | none |
| No static file serving | `/files/*` serves from screen directory |
## What Stays the Same
- `helper.js` — no changes
- `frame-template.html` — no changes
- `start-server.sh` — one-line update: `index.js` to `server.js`
- `stop-server.sh` — no changes
- `visual-companion.md` — no changes
- All existing server behavior and external contract
## Platform Compatibility
- `server.js` uses only cross-platform Node built-ins
- `fs.watch` is reliable for single flat directories on macOS, Linux, and Windows
- Shell scripts require bash (Git Bash on Windows, which is required for Claude Code)
## Testing
**Unit tests** (`ws-protocol.test.js`): Test WebSocket frame encoding/decoding, handshake computation, and protocol edge cases directly by requiring `server.js` exports.
**Integration tests** (`server.test.js`): Test full server behavior — HTTP serving, WebSocket communication, file watching, brainstorming workflow. Uses `ws` npm package as a test-only client dependency (not shipped to end users).

View File

@@ -1,244 +0,0 @@
# Codex App Compatibility: Worktree and Finishing Skill Adaptation
Make superpowers skills work in the Codex App's sandboxed worktree environment without breaking existing Claude Code or Codex CLI behavior.
**Ticket:** PRI-823
## Motivation
The Codex App runs agents inside git worktrees it manages — detached HEAD, located under `$CODEX_HOME/worktrees/`, with a Seatbelt sandbox that blocks `git checkout -b`, `git push`, and network access. Three superpowers skills assume unrestricted git access: `using-git-worktrees` creates manual worktrees with named branches, `finishing-a-development-branch` merges/pushes/PRs by branch name, and `subagent-driven-development` requires both.
The Codex CLI (open source terminal tool) does NOT have this conflict — it has no built-in worktree management. Our manual worktree approach fills an isolation gap there. The problem is specifically with the Codex App.
## Empirical Findings
Tested in the Codex App on 2026-03-23:
| Operation | workspace-write sandbox | Full access sandbox |
|---|---|---|
| `git add` | Works | Works |
| `git commit` | Works | Works |
| `git checkout -b` | **Blocked** (can't write `.git/refs/heads/`) | Works |
| `git push` | **Blocked** (network + `.git/refs/remotes/`) | Works |
| `gh pr create` | **Blocked** (network) | Works |
| `git status/diff/log` | Works | Works |
Additional findings:
- `spawn_agent` subagents **share** the parent thread's filesystem (confirmed via marker file test)
- "Create branch" button appears in the App header regardless of which branch the worktree was started from
- The App's native finishing flow: Create branch → Commit modal → Commit and push / Commit and create PR
- `network_access = true` config is silently broken on macOS (issue #10390)
## Design: Read-Only Environment Detection
Three read-only git commands detect the environment without side effects:
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
```
Two signals derived:
- **IN_LINKED_WORKTREE:** `GIT_DIR != GIT_COMMON` — the agent is in a worktree created by something else (Codex App, Claude Code Agent tool, previous skill run, or the user)
- **ON_DETACHED_HEAD:** `BRANCH` is empty — no named branch exists
Why `git-dir != git-common-dir` instead of checking `show-toplevel`:
- In a normal repo, both resolve to the same `.git` directory
- In a linked worktree, `git-dir` is `.git/worktrees/<name>` while `git-common-dir` is `.git`
- In a submodule, both are equal — avoiding a false positive that `show-toplevel` would produce
- Resolving via `cd && pwd -P` handles the relative-path problem (`git-common-dir` returns `.git` relative in normal repos but absolute in worktrees) and symlinks (macOS `/tmp``/private/tmp`)
### Decision Matrix
| Linked Worktree? | Detached HEAD? | Environment | Action |
|---|---|---|---|
| No | No | Claude Code / Codex CLI / normal git | Full skill behavior (unchanged) |
| Yes | Yes | Codex App worktree (workspace-write) | Skip worktree creation; handoff payload at finish |
| Yes | No | Codex App (Full access) or manual worktree | Skip worktree creation; full finishing flow |
| No | Yes | Unusual (manual detached HEAD) | Create worktree normally; warn at finish |
## Changes
### 1. `using-git-worktrees/SKILL.md` — Add Step 0 (~12 lines)
New section between "Overview" and "Directory Selection Process":
**Step 0: Check if Already in an Isolated Workspace**
Run the detection commands. If `GIT_DIR != GIT_COMMON`, skip worktree creation entirely. Instead:
1. Skip to "Run Project Setup" subsection under Creation Steps — `npm install` etc. is idempotent, worth running for safety
2. Then "Verify Clean Baseline" — run tests
3. Report with branch state:
- On a branch: "Already in an isolated workspace at `<path>` on branch `<name>`. Tests passing. Ready to implement."
- Detached HEAD: "Already in an isolated workspace at `<path>` (detached HEAD, externally managed). Tests passing. Note: branch creation needed at finish time. Ready to implement."
If `GIT_DIR == GIT_COMMON`, proceed with the full worktree creation flow (unchanged).
Safety verification (.gitignore check) is skipped when Step 0 fires — irrelevant for externally-created worktrees.
Update the Integration section's "Called by" entries. Change the description on each from context-specific text to: "Ensures isolated workspace (creates one or verifies existing)". For example, the `subagent-driven-development` entry changes from "REQUIRED: Set up isolated workspace before starting" to "REQUIRED: Ensures isolated workspace (creates one or verifies existing)".
**Sandbox fallback:** If `GIT_DIR == GIT_COMMON` and the skill proceeds to Creation Steps, but `git worktree add -b` fails with a permission error (e.g., Seatbelt sandbox denial), treat this as a late-detected restricted environment. Fall back to the Step 0 "already in workspace" behavior — skip creation, run setup and baseline tests in the current directory, report accordingly.
After reporting in Step 0, STOP. Do not continue to Directory Selection or Creation Steps.
**Everything else unchanged:** Directory Selection, Safety Verification, Creation Steps, Project Setup, Baseline Tests, Quick Reference, Common Mistakes, Red Flags.
### 2. `finishing-a-development-branch/SKILL.md` — Add Step 1.5 + cleanup guard (~20 lines)
**Step 1.5: Detect Environment** (after Step 1 "Verify Tests", before Step 2 "Determine Base Branch")
Run the detection commands. Three paths:
- **Path A** skips Steps 2 and 3 entirely (no base branch or options needed).
- **Paths B and C** proceed through Step 2 (Determine Base Branch) and Step 3 (Present Options) as normal.
**Path A — Externally managed worktree + detached HEAD** (`GIT_DIR != GIT_COMMON` AND `BRANCH` empty):
First, ensure all work is staged and committed (`git add` + `git commit`). The Codex App's finishing controls operate on committed work.
Then present this to the user (do NOT present the 4-option menu):
```
Implementation complete. All tests passing.
Current HEAD: <full-commit-sha>
This workspace is externally managed (detached HEAD).
I cannot create branches, push, or open PRs from here.
⚠ These commits are on a detached HEAD. If you do not create a branch,
they may be lost when this workspace is cleaned up.
If your host application provides these controls:
- "Create branch" — to name a branch, then commit/push/PR
- "Hand off to local" — to move changes to your local checkout
Suggested branch name: <ticket-id/short-description>
Suggested commit message: <summary-of-work>
```
Branch name derivation: use the ticket ID if available (e.g., `pri-823/codex-compat`), otherwise slugify the first 5 words of the plan title, otherwise omit the suggestion. Avoid including sensitive content (vulnerability descriptions, customer names) in branch names.
Skip to Step 5 (cleanup is a no-op for externally managed worktrees).
**Path B — Externally managed worktree + named branch** (`GIT_DIR != GIT_COMMON` AND `BRANCH` exists):
Present the 4-option menu as normal. (The Step 5 cleanup guard will re-detect the externally managed state independently.)
**Path C — Normal environment** (`GIT_DIR == GIT_COMMON`):
Present the 4-option menu as today (unchanged).
**Step 5 cleanup guard:**
Re-run the `GIT_DIR` vs `GIT_COMMON` detection at cleanup time (do not rely on earlier skill output — the finishing skill may run in a different session). If `GIT_DIR != GIT_COMMON`, skip `git worktree remove` — the host environment owns this workspace.
Otherwise, check and remove as today. Note: the existing Step 5 text says "For Options 1, 2, 4" but the Quick Reference table and Common Mistakes section say "Options 1 & 4 only." The new guard is added before this existing logic and does not change which options trigger cleanup.
**Everything else unchanged:** Options 1-4 logic, Quick Reference, Common Mistakes, Red Flags.
### 3. `subagent-driven-development/SKILL.md` and `executing-plans/SKILL.md` — 1 line edit each
Both skills have an identical Integration section line. Change from:
```
- superpowers:using-git-worktrees - REQUIRED: Set up isolated workspace before starting
```
To:
```
- superpowers:using-git-worktrees - REQUIRED: Ensures isolated workspace (creates one or verifies existing)
```
**Everything else unchanged:** Dispatch/review loop, prompt templates, model selection, status handling, red flags.
### 4. `codex-tools.md` — Add environment detection docs (~15 lines)
Two new sections at the end:
**Environment Detection:**
```markdown
## Environment Detection
Skills that create worktrees or finish branches should detect their
environment with read-only git commands before proceeding:
\```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
\```
- `GIT_DIR != GIT_COMMON` → already in a linked worktree (skip creation)
- `BRANCH` empty → detached HEAD (cannot branch/push/PR from sandbox)
See `using-git-worktrees` Step 0 and `finishing-a-development-branch`
Step 1.5 for how each skill uses these signals.
```
**Codex App Finishing:**
```markdown
## Codex App Finishing
When the sandbox blocks branch/push operations (detached HEAD in an
externally managed worktree), the agent commits all work and informs
the user to use the App's native controls:
- **"Create branch"** — names the branch, then commit/push/PR via App UI
- **"Hand off to local"** — transfers work to the user's local checkout
The agent can still run tests, stage files, and output suggested branch
names, commit messages, and PR descriptions for the user to copy.
```
## What Does NOT Change
- `implementer-prompt.md`, `spec-reviewer-prompt.md`, `code-quality-reviewer-prompt.md` — subagent prompts untouched
- `executing-plans/SKILL.md` — only the 1-line Integration description changes (same as `subagent-driven-development`); all runtime behavior is unchanged
- `dispatching-parallel-agents/SKILL.md` — no worktree or finishing operations
- `.codex/INSTALL.md` — installation process unchanged
- The 4-option finishing menu — preserved exactly for Claude Code and Codex CLI
- The full worktree creation flow — preserved exactly for non-worktree environments
- Subagent dispatch/review/iterate loop — unchanged (filesystem sharing confirmed)
## Scope Summary
| File | Change |
|---|---|
| `skills/using-git-worktrees/SKILL.md` | +12 lines (Step 0) |
| `skills/finishing-a-development-branch/SKILL.md` | +20 lines (Step 1.5 + cleanup guard) |
| `skills/subagent-driven-development/SKILL.md` | 1 line edit |
| `skills/executing-plans/SKILL.md` | 1 line edit |
| `skills/using-superpowers/references/codex-tools.md` | +15 lines |
~50 lines added/changed across 5 files. Zero new files. Zero breaking changes.
## Future Considerations
If a third skill needs the same detection pattern, extract it into a shared `references/environment-detection.md` file (Approach B). Not needed now — only 2 skills use it.
## Test Plan
### Automated (run in Claude Code after implementation)
1. Normal repo detection — assert IN_LINKED_WORKTREE=false
2. Linked worktree detection — `git worktree add` test worktree, assert IN_LINKED_WORKTREE=true
3. Detached HEAD detection — `git checkout --detach`, assert ON_DETACHED_HEAD=true
4. Finishing skill handoff output — verify handoff message (not 4-option menu) in restricted environment
5. **Step 5 cleanup guard** — create a linked worktree (`git worktree add /tmp/test-cleanup -b test-cleanup`), `cd` into it, run the Step 5 cleanup detection (`GIT_DIR` vs `GIT_COMMON`), assert it would NOT call `git worktree remove`. Then `cd` back to main repo, run the same detection, assert it WOULD call `git worktree remove`. Clean up test worktree afterward.
### Manual Codex App Tests (5 tests)
1. Detection in Worktree thread (workspace-write) — verify GIT_DIR != GIT_COMMON, empty branch
2. Detection in Worktree thread (Full access) — same detection, different sandbox behavior
3. Finishing skill handoff format — verify agent emits handoff payload, not 4-option menu
4. Full lifecycle — detection → commit → finishing detection → correct behavior → cleanup
5. **Sandbox fallback in Local thread** — Start a Codex App **Local thread** (workspace-write sandbox). Prompt: "Use the superpowers skill `using-git-worktrees` to set up an isolated workspace for implementing a small change." Pre-check: `git checkout -b test-sandbox-check` should fail with `Operation not permitted`. Expected: the skill detects `GIT_DIR == GIT_COMMON` (normal repo), attempts `git worktree add -b`, hits Seatbelt denial, falls back to Step 0 "already in workspace" behavior — runs setup, baseline tests, reports ready from current directory. Pass: agent recovers gracefully without cryptic error messages. Fail: agent prints raw Seatbelt error, retries, or gives up with confusing output.
### Regression
- Existing Claude Code skill-triggering tests still pass
- Existing subagent-driven-development integration tests still pass
- Normal Claude Code session: full worktree creation + 4-option finishing still works

View File

@@ -1,341 +0,0 @@
# Worktree Rototill: Detect-and-Defer
**Date:** 2026-04-06
**Status:** Draft
**Ticket:** PRI-974
**Subsumes:** PRI-823 (Codex App compatibility)
## Problem
Superpowers is opinionated about worktree management — specific paths (`.worktrees/<branch>`), specific commands (`git worktree add`), specific cleanup (`git worktree remove`). Meanwhile, Claude Code, Codex App, Gemini CLI, and Cursor all provide native worktree support with their own paths, lifecycle management, and cleanup.
This creates three failure modes:
1. **Duplication** — on Claude Code, the skill does what `EnterWorktree`/`ExitWorktree` already does
2. **Conflict** — on Codex App, the skill tries to create worktrees inside an already-managed worktree
3. **Phantom state** — skill-created worktrees at `.worktrees/` are invisible to the harness; harness-created worktrees at `.claude/worktrees/` are invisible to the skill
For harnesses without native support (Codex CLI, OpenCode, Copilot standalone), superpowers fills a real gap. The skill shouldn't go away — it should get out of the way when native support exists.
## Goals
1. Defer to native harness worktree systems when they exist
2. Continue providing worktree support for harnesses that lack it
3. Fix three known bugs in finishing-a-development-branch (#940, #999, #238)
4. Make worktree creation opt-in rather than mandatory (#991)
5. Replace hardcoded `CLAUDE.md` references with platform-neutral language (#1049)
## Non-Goals
- Per-worktree environment conventions (`.worktree-env.sh`, port offsetting) — Phase 4
- PreToolUse hooks for path enforcement — Phase 4
- Multi-repo worktree documentation — Phase 4
- Brainstorming checklist changes for worktrees — Phase 4
- `.superpowers-session.json` metadata tracking (interesting PR #997 idea, not needed for v1)
- Hooks symlinking into worktrees (PR #965 idea, separate concern)
## Design Principles
### Detect state, not platform
Use `GIT_DIR != GIT_COMMON` to determine "am I already in a worktree?" rather than sniffing environment variables to identify the harness. This is a stable git primitive (since git 2.5, 2015), works universally across all harnesses, and requires zero maintenance as new harnesses appear.
### Declarative intent, prescriptive fallback
The skill describes the goal ("ensure work happens in an isolated workspace") and defers to native tools when available. It prescribes specific git commands only as a fallback for harnesses without native worktree support. Step 1a comes first and names native tools explicitly (`EnterWorktree`, `WorktreeCreate`, `/worktree`, `--worktree`); Step 1b comes second with the git fallback. The original spec kept Step 1a abstract ("you know your own toolkit"), but TDD proved that agents anchor on Step 1b's concrete commands when Step 1a is too vague. Explicit tool naming and a consent-authorization bridge were required to make the preference reliable.
### Provenance-based ownership
Whoever creates the worktree owns its cleanup. If the harness created it, superpowers doesn't touch it. If superpowers created it (via git fallback), superpowers cleans it up. The heuristic: if the worktree lives under `.worktrees/` or `worktrees/`, superpowers owns it. Anything else (`.claude/worktrees/`, `~/.codex/worktrees/`, `.gemini/worktrees/`, or old user-global Superpowers paths) belongs to the harness or user and is left alone.
## Design
### 1. `using-git-worktrees` SKILL.md Rewrite
The skill gains three new steps before creation and simplifies the creation flow.
#### Step 0: Detect Existing Isolation
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
BRANCH=$(git branch --show-current)
```
Three outcomes:
| Condition | Meaning | Action |
|-----------|---------|--------|
| `GIT_DIR == GIT_COMMON` | Normal repo checkout | Proceed to Step 0.5 |
| `GIT_DIR != GIT_COMMON`, named branch | Already in a linked worktree | Skip to Step 3 (project setup). Report: "Already in isolated workspace at `<path>` on branch `<name>`." |
| `GIT_DIR != GIT_COMMON`, detached HEAD | Externally managed worktree (e.g., Codex App sandbox) | Skip to Step 3. Report: "Already in isolated workspace at `<path>` (detached HEAD, externally managed)." |
Step 0 does not care who created the worktree or which harness is running. A worktree is a worktree regardless of origin.
**Submodule guard:** `GIT_DIR != GIT_COMMON` is also true inside git submodules. Before concluding "already in a worktree," check that we're not in a submodule:
```bash
# If this returns a path, we're in a submodule, not a worktree
git rev-parse --show-superproject-working-tree 2>/dev/null
```
If in a submodule, treat as `GIT_DIR == GIT_COMMON` (proceed to Step 0.5).
#### Step 0.5: Consent
When Step 0 finds no existing isolation (`GIT_DIR == GIT_COMMON`), ask before creating:
> "Would you like me to set up an isolated worktree? This protects your current branch from changes. (y/n)"
If yes, proceed to Step 1. If no, work in place — skip to Step 3 with no worktree.
This step is skipped entirely when Step 0 detects existing isolation (no point asking about what already exists).
#### Step 1a: Native Tools (preferred)
> The user has asked for an isolated workspace (Step 0 consent). Check your available tools — do you have `EnterWorktree`, `WorktreeCreate`, a `/worktree` command, or a `--worktree` flag? If YES: the user's consent to create a worktree is your authorization to use it. Use it now and skip to Step 3.
After using a native tool, skip to Step 3 (project setup).
**Design note — TDD revision:** The original spec used a deliberately short, abstract Step 1a ("You know your own toolkit — the skill does not need to name specific tools"). TDD validation disproved this: agents anchored on Step 1b's concrete git commands and ignored the abstract guidance (2/6 pass rate). Three changes fixed it (50/50 pass rate across GREEN and PRESSURE tests):
1. **Explicit tool naming** — listing `EnterWorktree`, `WorktreeCreate`, `/worktree`, `--worktree` by name transforms the decision from interpretation ("do I have a native tool?") into factual lookup ("is `EnterWorktree` in my tool list?"). Agents on platforms without these tools simply check, find nothing, and fall through to Step 1b. No false positives observed.
2. **Consent bridge** — "the user's consent to create a worktree is your authorization to use it" directly addresses `EnterWorktree`'s tool-level guardrail ("ONLY when user explicitly asks"). Tool descriptions override skill instructions (Claude Code #29950), so the skill must frame user consent as the authorization the tool requires.
3. **Red Flag entry** — naming the specific anti-pattern ("Use `git worktree add` when you have a native worktree tool — this is the #1 mistake") in the Red Flags section.
File splitting (Step 1b in a separate skill) was tested and proven unnecessary. The anchoring problem is solved by the quality of Step 1a's text, not by physical separation of git commands. Control tests with the full 240-line skill (all git commands visible) passed 20/20.
#### Step 1b: Git Worktree Fallback
When no native tool is available, create a worktree manually.
**Directory selection** (priority order):
1. Check the project's agent instruction file (CLAUDE.md, GEMINI.md, AGENTS.md, .cursorrules, or equivalent) for a worktree directory preference.
2. Check for existing `.worktrees/` or `worktrees/` directory — if found, use it. If both exist, `.worktrees/` wins.
3. Default to `.worktrees/`.
No interactive directory selection prompt. Old user-global Superpowers worktree paths are not detected or offered; new manual worktrees are project-local unless the user explicitly specifies another location.
**Safety verification** (project-local directories only):
```bash
git check-ignore -q .worktrees 2>/dev/null
```
If not ignored, add to `.gitignore` and commit before proceeding.
**Create:**
```bash
git worktree add "$path" -b "$BRANCH_NAME"
cd "$path"
```
**Hooks awareness:** Git worktrees do not inherit the parent repo's hooks directory. After creating a worktree via 1b, symlink the hooks directory from the main repo if one exists:
```bash
if [ -d "$MAIN_ROOT/.git/hooks" ]; then
ln -sf "$MAIN_ROOT/.git/hooks" "$path/.git/hooks"
fi
```
This prevents pre-commit checks, linters, and other hooks from silently stopping when work moves to a worktree. (Idea from PR #965.)
**Sandbox fallback:** If `git worktree add` fails with a permission error, treat as a restricted environment. Skip creation, work in current directory, proceed to Step 3.
**Step numbering note:** The current skill has Steps 1-4 as a flat list. This redesign uses 0, 0.5, 1a, 1b, 3, 4. There is no Step 2 — it was the old monolithic "Create Isolated Workspace" which is now split into the 1a/1b structure. The implementation should renumber cleanly (e.g., 0 → "Step 0: Detect", 0.5 → within Step 0's flow, 1a/1b → "Step 1", 3 → "Step 2", 4 → "Step 3") or keep the current numbering with a note. Implementer's choice.
#### Steps 3-4: Project Setup and Baseline Tests (unchanged)
Regardless of which path created the workspace (Step 0 detected existing, Step 1a native tool, Step 1b git fallback, or no worktree at all), execution converges:
- **Step 3:** Auto-detect and run project setup (`npm install`, `cargo build`, `pip install`, `go mod download`, etc.)
- **Step 4:** Run the test suite. If tests fail, report failures and ask whether to proceed.
### 2. `finishing-a-development-branch` SKILL.md Rewrite
The finishing skill gains environment detection and fixes three bugs.
#### Step 1: Verify Tests (unchanged)
Run the project's test suite. If tests fail, stop. Don't offer completion options.
#### Step 1.5: Detect Environment (new)
Re-run the same detection as Step 0 in creation:
```bash
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
```
Three paths:
| State | Menu | Cleanup |
|-------|------|---------|
| `GIT_DIR == GIT_COMMON` (normal repo) | Standard 4 options | No worktree to clean up |
| `GIT_DIR != GIT_COMMON`, named branch | Standard 4 options | Provenance-based (see Step 5) |
| `GIT_DIR != GIT_COMMON`, detached HEAD | Reduced menu: push as new branch + PR, keep as-is, discard | No merge options (can't merge from detached HEAD) |
#### Step 2: Determine Base Branch (unchanged)
#### Step 3: Present Options
**Normal repo and named-branch worktree:**
1. Merge back to `<base-branch>` locally
2. Push and create a Pull Request
3. Keep the branch as-is (I'll handle it later)
4. Discard this work
**Detached HEAD:**
1. Push as new branch and create a Pull Request
2. Keep as-is (I'll handle it later)
3. Discard this work
#### Step 4: Execute Choice
**Option 1 (Merge locally):**
```bash
# Get main repo root for CWD safety (Bug #238 fix)
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
cd "$MAIN_ROOT"
# Merge first, verify success before removing anything
git checkout <base-branch>
git pull
git merge <feature-branch>
<run tests>
# Only after merge succeeds: remove worktree, then delete branch (Bug #999 fix)
git worktree remove "$WORKTREE_PATH" # only if superpowers owns it
git branch -d <feature-branch>
```
The order is critical: merge → verify → remove worktree → delete branch. The old skill deleted the branch before removing the worktree (which fails because the worktree still references the branch). The naive fix of removing the worktree first is also wrong — if the merge then fails, the working directory is gone and changes are lost.
**Option 2 (Create PR):**
Push branch, create PR. Do NOT clean up worktree — user needs it for PR iteration. (Bug #940 fix: remove contradictory "Then: Cleanup worktree" prose.)
**Option 3 (Keep as-is):** No action.
**Option 4 (Discard):** Require typed "discard" confirmation. Then remove worktree (if superpowers owns it), force-delete branch.
#### Step 5: Cleanup (updated)
```
if GIT_DIR == GIT_COMMON:
# Normal repo, no worktree to clean up
done
if worktree path is under .worktrees/ or worktrees/:
# Superpowers created it — we own cleanup
cd to main repo root # Bug #238 fix
git worktree remove <path>
else:
# Harness created it — hands off
# If platform provides a workspace-exit tool, use it
# Otherwise, leave the worktree in place
```
Cleanup only runs for Options 1 and 4. Options 2 and 3 always preserve the worktree. (Bug #940 fix.)
**Stale worktree pruning:** After any `git worktree remove`, run `git worktree prune` as a self-healing step. Worktree directories can get deleted out-of-band (e.g., by harness cleanup, manual `rm`, or `.claude/` cleanup), leaving stale registrations that cause confusing errors. One line, prevents silent rot. (Idea from PR #1072.)
### 3. Integration Updates
#### `subagent-driven-development` and `executing-plans`
Both currently list `using-git-worktrees` as REQUIRED in their integration sections. Change to:
> `using-git-worktrees` — Ensures isolated workspace (creates one or verifies existing)
The skill itself now handles consent (Step 0.5) and detection (Step 0), so calling skills don't need to gate or prompt.
#### `writing-plans`
Remove the stale claim "should be run in a dedicated worktree (created by brainstorming skill)." Brainstorming is a design skill and does not create worktrees. The worktree prompt happens at execution time via `using-git-worktrees`.
### 4. Platform-Neutral Instruction File References
All instances of hardcoded `CLAUDE.md` in worktree-related skills are replaced with:
> "your project's agent instruction file (CLAUDE.md, GEMINI.md, AGENTS.md, .cursorrules, or equivalent)"
This applies to directory preference checks in Step 1b.
## Bug Fixes (bundled)
| Bug | Problem | Fix | Location |
|-----|---------|-----|----------|
| #940 | Option 2 prose says "Then: Cleanup worktree (Step 5)" but quick reference says keep it. Step 5 says "For Options 1, 2, 4" but Common Mistakes says "Options 1 and 4 only." | Remove cleanup from Option 2. Step 5 applies to Options 1 and 4 only. | finishing SKILL.md |
| #999 | Option 1 deletes branch before removing worktree. `git branch -d` can fail because worktree still references the branch. | Reorder to: merge → verify tests → remove worktree → delete branch. Merge must succeed before anything is removed. | finishing SKILL.md |
| #238 | `git worktree remove` fails silently if CWD is inside the worktree being removed. | Add CWD guard: `cd` to main repo root before `git worktree remove`. | finishing SKILL.md |
## Issues Resolved
| Issue | Resolution |
|-------|-----------|
| #940 | Direct fix (Bug #940) |
| #991 | Opt-in consent in Step 0.5 |
| #918 | Step 0 detection + Step 1.5 finishing detection |
| #1009 | Resolved by Step 1a — agents use native tools (e.g., `EnterWorktree`) which create at harness-native paths. Depends on Step 1a working; see Risks. |
| #999 | Direct fix (Bug #999) |
| #238 | Direct fix (Bug #238) |
| #1049 | Platform-neutral instruction file references |
| #279 | Solved by detect-and-defer — native paths respected because we don't override them |
| #574 | **Deferred.** Nothing in this spec touches the brainstorming skill where the bug lives. Full fix (adding a worktree step to brainstorming's checklist) is Phase 4. |
## Risks
### Step 1a is the load-bearing assumption — RESOLVED
Step 1a — agents preferring native worktree tools over the git fallback — is the foundation the entire design rests on. If agents ignore Step 1a and fall through to Step 1b on harnesses with native support, detect-and-defer fails entirely.
**Status:** This risk materialized during implementation. The original abstract Step 1a ("You know your own toolkit") failed at 2/6 on Claude Code. The TDD gate worked as designed — it caught the failure before any skill files were modified, preventing a broken release. Three REFACTOR iterations identified the root causes (agent anchoring on concrete commands, tool-description guardrail overriding skill instructions) and produced a fix validated at 50/50 across GREEN and PRESSURE tests. See Step 1a design note above for details.
**Cross-platform validation:**
As of 2026-04-06, Claude Code is the only harness with an agent-callable mid-session worktree tool (`EnterWorktree`). All others either create worktrees before the agent starts (Codex App, Gemini CLI, Cursor) or have no native worktree support (Codex CLI, OpenCode). Step 1a is forward-compatible: when other harnesses add agent-callable worktree tools, agents will match them against the named examples and use them without skill changes.
| Harness | Current worktree model | Skill mechanism | Tested |
|---------|----------------------|-----------------|--------|
| Claude Code | Agent-callable `EnterWorktree` | Step 1a | 50/50 (GREEN + PRESSURE) |
| Codex CLI | No native tool (shell only) | Step 1b git fallback | 6/6 (`codex exec`) |
| Gemini CLI | Launch-time `--worktree` flag, no agent tool | Step 0 if launched with flag, Step 1b if not | Step 0: 1/1, Step 1b: 1/1 (`gemini -p`) |
| Cursor Agent | User-facing `/worktree`, no agent tool | Step 0 if user activated, Step 1b if not | Step 0: 1/1, Step 1b: 1/1 (`cursor-agent -p`) |
| Codex App | Platform-managed, detached HEAD, no agent tool | Step 0 detects existing | 1/1 simulated |
| OpenCode | Detection only (`ctx.worktree`), no agent tool | Step 1b git fallback | Untested (no CLI access) |
**Residual risks:**
1. If Anthropic changes `EnterWorktree`'s tool description to be more restrictive (e.g., "Do not use based on skill instructions"), the consent bridge breaks. Worth filing an issue requesting that the tool description accommodate skill-driven invocation.
2. When other harnesses add agent-callable worktree tools, they may use names not in Step 1a's list. The list should be updated as new tools appear. The generic phrasing ("a worktree or workspace-isolation tool") provides some forward coverage.
### Provenance heuristic
The `.worktrees/` or `worktrees/` = ours, anything else = hands off` heuristic works for every current harness. If a future harness adopts one of those project-local directories as its convention, we'd have a false positive (superpowers tries to clean up a harness-owned worktree). Similarly, if a user manually runs `git worktree add .worktrees/experiment` without superpowers, we'd incorrectly claim ownership. Both are low risk — every harness uses branded paths, and manual `.worktrees/` creation is unlikely — but worth noting.
### Detached HEAD finishing
The reduced menu for detached HEAD worktrees (no merge option) is correct for Codex App's sandbox model. If a user is in detached HEAD for another reason, the reduced menu still makes sense — you genuinely can't merge from detached HEAD without creating a branch first.
## Implementation Notes
Both skill files contain sections beyond the core steps that need updating during implementation:
- **Frontmatter** (`name`, `description`): Update to reflect detect-and-defer behavior
- **Quick Reference tables**: Rewrite to match new step structure and bug fixes
- **Common Mistakes sections**: Update or remove items that reference old behavior (e.g., "Skip CLAUDE.md check" is now wrong)
- **Red Flags sections**: Update to reflect new priorities (e.g., "Never create a worktree when Step 0 detects existing isolation")
- **Integration sections**: Update cross-references between skills
The spec describes *what changes*; the implementation plan will specify exact edits to these secondary sections.
## Future Work (not in this spec)
- **Phase 3 remainder:** `$TMPDIR` directory option (#666), setup docs for caching and env inheritance (#299)
- **Phase 4:** PreToolUse hooks for path enforcement (#1040), per-worktree env conventions (#597), brainstorming checklist worktree step (#574), multi-repo documentation (#710)

View File

@@ -1,247 +0,0 @@
# Lift drill into superpowers as `evals/` — design
## Background
Drill is a Python skill-compliance benchmark that lives in its own repo at `obra/drill`. It drives real tmux sessions, runs an LLM actor as a simulated user, runs an LLM verifier on the resulting transcript, and reports pass/fail per scenario. It supports Claude Code, Codex, Gemini CLI, and (per recent commits) OpenCode and Copilot CLI.
Drill is already the *de facto* eval harness for superpowers. The PRI-1397 commit series in the drill repo lifted ~22 superpowers bash tests into drill scenarios, and the most recent superpowers commit (`a2292c5`) explicitly removed a redundant bash test with the message *"replaced by drill behavioral coverage"*. Migration momentum exists; this spec completes it.
This work moves drill into superpowers under `evals/`, deletes the redundant bash tests after per-file verification of drill scenario coverage, and updates docs so contributors land on the new structure.
## Goals
1. `evals/` is the canonical eval harness in superpowers — full drill source, scenarios, fixtures, prompts, backend configs, and tests.
2. Bash tests in `superpowers/tests/` that have been individually verified as 100% covered by drill scenarios are deleted; the rest are preserved.
3. The split between `tests/` (plugin infrastructure: bash + node + python integration tests) and `evals/` (LLM behavior with actor + verifier) is meaningful and documented.
4. Top-level docs (`README.md`, `CLAUDE.md`, `docs/testing.md`) point contributors at the right place.
5. The standalone `obra/drill` repo continues to exist (this PR does not touch it) and gets archived as a separate manual step after this PR merges.
## Non-goals
- **CI integration.** Manual-only here. The natural follow-up is "tiered": fast subset on every PR, full sweep nightly + on-demand. That requires API budget decisions, GitHub Actions secrets, and a runner image with `tmux` + `node` + `python` + `claude` / `codex` / `gemini` CLIs installed. Out of scope.
- **Scenario co-location with skills.** Scenarios stay centralized at `evals/scenarios/`. If we later decide each skill should own its scenarios, that's a path-find-and-rename operation; the YAML format does not change.
- **Renaming the internal Python package** (`drill``evals`). The directory is `evals/` (user-facing); the Python package keeps its `drill` name to keep the diff small. A short note in `evals/README.md` explains.
- **Drill repo archival.** This PR does not touch `obra/drill`. After merge, the drill repo is archived manually (read-only on GitHub, README pointer to `obra/superpowers/evals/`).
- **Lifting `tests/claude-code/analyze-token-usage.py` into `evals/bin/`.** Useful utility, not test code. Can move later; not required by this PR.
## Branching
Branch off `dev` as `f/evals-lift`. This work is independent of the open `f/cross-platform` PR — no shared file changes besides possibly `README.md`, which is small enough to resolve at merge time if it conflicts.
## Architecture after the move
```
superpowers/
evals/ ← NEW (full drill copy)
pyproject.toml (Python 3.11, uv-managed)
uv.lock
.gitignore (drill's own; results/, .venv/, .env)
README.md (was drill's README; install instructions updated)
CLAUDE.md (was drill's CLAUDE.md; paths updated)
docs/
design.md (drill's design — preserved verbatim, cross-linked from this spec)
manual-testing.md
pressure-and-red-testing.md
drill/ (Python package; name kept; cli, engine, actor, verifier, etc.)
backends/ (claude-*.yaml, codex.yaml, gemini.yaml)
scenarios/ (32+ YAML scenarios)
setup_helpers/ (15 Python helpers; create_base_repo, sdd_*, spec_*, worktree, etc.)
fixtures/ (template-repo, sdd-go-fractals, sdd-svelte-todo)
prompts/ (actor.md, verifier.md)
bin/ (assertion helper scripts: tool-called, tool-count, etc.)
tests/ (drill's own pytest suite)
tests/ ← bash tests preserved by default
brainstorm-server/ ← KEEP (node tests for brainstorm-server JS code)
opencode/ ← KEEP (plugin loading tests)
codex-plugin-sync/ ← KEEP (sync verification)
claude-code/ ← MOSTLY KEEP — see deletion gate
explicit-skill-requests/ ← KEEP unless verified replaced
skill-triggering/ ← KEEP unless verified replaced
subagent-driven-dev/ ← KEEP unless verified replaced
docs/
testing.md ← UPDATED (split into "Plugin tests" + "Skill behavior evals")
superpowers/
specs/
2026-05-06-lift-drill-into-evals-design.md ← THIS SPEC
README.md ← small Contributing-section pointer to evals/
CLAUDE.md ← one-line "Eval harness lives at evals/" pointer
```
The `tests/` and `evals/` directories serve clearly distinct roles after this PR:
- **`tests/`** — does the plugin's non-LLM code work? Unit and integration tests for the brainstorm-server JS code, OpenCode plugin loading, codex-plugin-sync sync verification. Bash + node + python.
- **`evals/`** — do agents behave correctly on real LLM sessions? Drill scenarios with actor + verifier. Python-only, runs real tmux sessions.
## Deletion gate (per bash test)
A bash test is deleted *only if* a drill scenario verifiably covers every assertion it makes. The implementation plan documents this verification per file: read the bash test, list its checks, find the drill scenario, confirm each check has a matching `verify.assertions` or `verify.criteria` entry. If even one check is missing, the option is to either extend the drill scenario or keep the bash test. Default keeps it.
**Tentative coverage map** (commit-message-based; needs per-file verification before any deletion):
| Bash test | Claimed drill replacement | Coverage status |
|-----------|---------------------------|-----------------|
| `tests/skill-triggering/prompts/*` (6 prompt files) | `triggering-*.yaml` (6 scenarios) | candidate — verify per-prompt before deleting |
| `tests/skill-triggering/run-test.sh`, `run-all.sh` | n/a (runners, not tests) | **keep** — runner scripts |
| `tests/explicit-skill-requests/prompts/please-use-brainstorming.txt` | needs verification — drill has no obvious counterpart yet | likely **keep** unless drill scenario added |
| `tests/explicit-skill-requests/prompts/use-systematic-debugging.txt` | needs verification — drill has no obvious counterpart | likely **keep** unless drill scenario added |
| `tests/explicit-skill-requests/run-claude-describes-sdd.sh` | partially → `mid-conversation-skill-invocation.yaml` | candidate — verify per-script |
| `tests/explicit-skill-requests/run-haiku-test.sh` | no drill scenario covers Haiku-specific behavior | **keep** |
| `tests/explicit-skill-requests/run-multiturn-test.sh`, `run-extended-multiturn-test.sh` | no drill scenario covers multi-turn build-up | **keep** unless drill scenarios added |
| `tests/explicit-skill-requests/run-test.sh`, `run-all.sh` | n/a (runners) | **keep** |
| `tests/subagent-driven-dev/go-fractals/`, `tests/subagent-driven-dev/svelte-todo/` | `sdd-go-fractals.yaml`, `sdd-svelte-todo.yaml` | candidate — verify before deleting (these include real assertions about test suites passing) |
| `tests/claude-code/test-document-review-system.sh` | `spec-reviewer-catches-planted-flaws.yaml` | candidate — verify before deleting |
| `tests/claude-code/test-requesting-code-review.sh` | `code-review-catches-planted-bugs.yaml` | candidate — verify before deleting |
| `tests/claude-code/test-subagent-driven-development-integration.sh` | `sdd-rejects-extra-features.yaml` (YAGNI subset) | **partial** — bash test also asserts ≥3 commits / `npm test` passes / runs `analyze-token-usage.py`. Drill scenario asserts forbidden-exports + reviewer-as-gate. Mostly disjoint — almost certainly **keep + extend drill scenario**. |
| `tests/claude-code/test-subagent-driven-development.sh` | meta/documentation test (asks agent to *describe* SDD); no drill scenario covers description tests | **keep** unless drill scenario added |
| `tests/claude-code/test-worktree-native-preference.sh` | `worktree-creation-under-pressure.yaml` | candidate — verify before deleting |
| `tests/claude-code/test-helpers.sh`, `run-skill-tests.sh`, `analyze-token-usage.py` | n/a (utilities, not tests) | **keep** — libraries/tools |
## Verification protocol (subagent-gated)
Every change in the implementation plan gets cross-checked by an independent subagent before commit.
| Change category | Subagent verification |
|----------------|----------------------|
| Each bash-test deletion | Dispatch a subagent with: (a) the bash test file content, (b) the candidate drill scenario YAML, (c) the prompt: *"List every assertion the bash test makes. List every verify entry in the drill scenario. For each bash assertion, find a matching drill check or report it as unmatched. Output a per-assertion table."* The subagent's output is the gate — only delete if every bash assertion has a match. |
| Initial `evals/` copy | Subagent verifies: (a) drill SHA being copied is recorded in the lift commit message so provenance is auditable; (b) **per-file SHA-256 checksum** matches drill repo for every file (not just file count); (c) excluded paths (`.git/`, `.venv/`, `results/`, `.env`, `__pycache__/`, `*.egg-info/`, any `.private-journal/`) are absent from `evals/`; (d) all backend YAMLs reference paths that exist post-move; (e) `pyproject.toml`, `uv.lock`, `.gitignore` are intact. |
| Drill's own pytest suite | Subagent runs `cd evals && uv run pytest` after the path-default change. Drill ships its own pytest suite at `evals/tests/` including `test_backend.py` which exercises `SUPERPOWERS_ROOT` env-var behavior — these tests must update to match the helper and continue to pass. |
| Reference scrubbing after deletion | Subagent greps the entire superpowers tree (excluding `node_modules/`, `.venv/`, and `evals/`) for references to deleted bash test paths. Search targets: `docs/`, `docs/superpowers/plans/`, `RELEASE-NOTES.md`, `CLAUDE.md`, `GEMINI.md`, `AGENTS.md`, `README.md`, `.github/`, `scripts/`, `.opencode/INSTALL.md`, `.codex-plugin/INSTALL.md`, `lefthook.yml`. Any hit is either updated or surfaces a missed dependency. |
| Path defaults change (`SUPERPOWERS_ROOT` default) | Subagent runs at least one cheap drill scenario after the path changes (e.g., `triggering-test-driven-development`) and confirms it still passes. Real validation, not just code review. |
| Final pre-PR adversarial review | Two subagents in parallel, "5 points to whoever finds the most legitimate issues" framing — same protocol used on the cross-platform PR. Verify both source code and behavior. |
Each subagent task gets its own bullet in the implementation plan with explicit inputs and pass criteria. The subagent's output is summarized in the relevant commit message ("Subagent verification: …") so the trail is auditable.
## Concrete path/config edits
**Verified prior to writing this spec.** `drill/cli.py` defines `PROJECT_ROOT = Path(__file__).parent.parent`. After the move, `cli.py` lives at `evals/drill/cli.py`, so `PROJECT_ROOT` resolves to `evals/` and `PROJECT_ROOT.parent` resolves to the superpowers repo root. That's the value `SUPERPOWERS_ROOT` should take by default.
**YAML substitution audit.** Only the four `claude*.yaml` backend configs interpolate `${SUPERPOWERS_ROOT}` into `args` (for the `--plugin-dir` flag); `codex.yaml` and `gemini.yaml` only list `SUPERPOWERS_ROOT` in `required_env` (consumed by `engine.py:233` / `setup.py:25`'s `os.environ["SUPERPOWERS_ROOT"]` lookups in pre/post-run hooks). The helper's `os.environ` mutation covers both code paths.
| File | Current | After |
|------|---------|-------|
| `drill/cli.py` | `load_dotenv(PROJECT_ROOT / ".env")` at module import; nothing about `SUPERPOWERS_ROOT` | After `load_dotenv`, call new helper `_set_superpowers_root_default()` that sets `os.environ["SUPERPOWERS_ROOT"]` to `str(PROJECT_ROOT.parent)` if and only if not already set. Order: `load_dotenv` → set default → click group definitions. |
| `drill/engine.py:233`, `drill/setup.py:25` | Direct `os.environ["SUPERPOWERS_ROOT"]` access (KeyError if unset) | Unchanged. The CLI startup hook guarantees the env var is set by the time the engine/setup execute. |
| `backends/claude*.yaml` (5 files) | `${SUPERPOWERS_ROOT}` substituted in `args` for `--plugin-dir` | Unchanged. YAML substitution reads `os.environ` at backend-load time, which is after CLI startup. |
| `backends/codex.yaml`, `backends/gemini.yaml` | `SUPERPOWERS_ROOT` in `required_env` only | Drop from `required_env` (the helper supplies it). `claude*.yaml` keep `required_env` for backward compat (env var works as override). |
| `evals/tests/test_backend.py` | Tests assert `SUPERPOWERS_ROOT` is in `required_env` lists, plus path-resolution tests | Update tests to match the new contract: helper-supplied default, env override still works, `required_env` no longer required for codex/gemini. |
| `evals/README.md` | "export SUPERPOWERS_ROOT=/path/to/superpowers" | Drop the export line; note that the env var auto-defaults to the parent of `evals/`; mention the only required setup is `ANTHROPIC_API_KEY` (or `OPENAI_API_KEY` / Gemini auth). |
| `evals/CLAUDE.md` | Same | Same |
| `evals/.gitignore` | drill's existing patterns (`results/`, `.venv/`, `__pycache__/`, `.env`, `*.pyc`, `*.egg-info/`, `dist/`, `build/`, `.claude/`) | Copied verbatim. Patterns are relative to file location, so they apply correctly under `evals/`. |
| `evals/lefthook.yml` | drill ships `lefthook.yml` defining `pre-commit: uv run ruff check && uv run ty check` | Move to `evals/lefthook.yml`. Either (a) install lefthook at the superpowers root and have it federate to `evals/lefthook.yml`, or (b) document that contributors run `cd evals && lefthook run pre-commit` manually. **Decision in implementation: option (b) for simplicity** — superpowers' top-level workflow doesn't change. |
`.env` placement: keep `evals/.env` (gitignored). Contributors source it from there or set `ANTHROPIC_API_KEY` in their shell environment.
**Top-level superpowers files needing small additions:**
- `superpowers/.gitignore`: add `evals/results/`, `evals/.venv/`, `evals/.env` (belt-and-suspenders; evals/.gitignore already covers these locally).
- `superpowers/CLAUDE.md`: add a one-line pointer "Eval harness lives at `evals/` — see `evals/README.md`" so agents discover it.
- `superpowers/docs/testing.md`: split into "## Plugin tests" (existing tests/ content, with the deleted-test references trimmed) and "## Skill behavior evals" (one-paragraph summary + pointer to `evals/`).
- `superpowers/README.md`: add a single line in the Contributing section pointing at `evals/` for skill-behavior testing.
## Migration ordering
Each step is a separate commit (or small group of commits). Step 2 is the biggest single commit (the verbatim drill copy); subsequent steps are small and atomic.
```
1. Branch off `dev` (f/evals-lift)
2. Copy drill repo into evals/ (single commit, easy to revert)
├─ Record drill SHA at copy time → commit message
├─ Use `rsync -a --exclude=.git --exclude=.venv --exclude=results
│ --exclude=.env --exclude=__pycache__ --exclude='*.egg-info'
│ --exclude=.private-journal /path/to/drill/ evals/`
│ (rsync chosen over `cp -r` for explicit excludes; verify with
│ `find evals -name '.git' -type d` returns nothing)
├─ Subagent gate: per-file SHA-256 checksum matches drill repo for every
│ non-excluded file; excluded paths absent from evals/
└─ Smoke check: `cd evals && uv sync` succeeds (proves install only;
not a behavioral test)
3. Update path defaults
├─ Add _set_superpowers_root_default() helper to drill/cli.py
├─ Wire it after load_dotenv, before click group definition
├─ Update evals/README.md and evals/CLAUDE.md (drop SUPERPOWERS_ROOT install step)
├─ Drop SUPERPOWERS_ROOT from required_env in codex.yaml/gemini.yaml
│ (keep in claude*.yaml as override)
└─ Update evals/tests/test_backend.py to match new contract
4. Validate from new location (TWO checks)
├─ Run drill's own pytest: `cd evals && uv run pytest` — must pass
└─ Run cheap drill scenario: `cd evals && uv run drill run
triggering-test-driven-development -b claude` — must pass.
Real behavioral validation, not just code review.
5. Bash test deletion phase — per-file with subagent gate
For each file in the candidate-deletion list:
a. Subagent compares bash test assertions vs drill scenario verify block
b. Pass criterion: every bash assertion has a matching drill check
c. If pass → delete the bash test file (one commit per file or per
coherent group)
d. If fail → either extend drill scenario (separate commit + verify) or
keep the bash test (no commit)
6. Stale-reference scrub
├─ Subagent greps the superpowers tree (excluding node_modules/, .venv/,
│ evals/) for deleted file paths
├─ Search targets: docs/, docs/superpowers/plans/, RELEASE-NOTES.md,
│ CLAUDE.md, GEMINI.md, AGENTS.md, README.md, .github/, scripts/,
│ .opencode/INSTALL.md, .codex-plugin/INSTALL.md, lefthook.yml
├─ Update active references (e.g., docs/testing.md, README.md install)
└─ Historical references in docs/superpowers/plans/*.md and
RELEASE-NOTES.md are PRESERVED with a brief annotation
("(test removed; behavior covered by drill scenario X)") rather
than rewritten — these are dated artifacts, not living docs.
7. Top-level docs
├─ docs/testing.md split
├─ CLAUDE.md pointer
└─ README.md Contributing section
8. Re-run smoke checks (regression gate)
├─ `cd evals && uv run pytest`
└─ `cd evals && uv run drill run triggering-test-driven-development -b claude`
9. Final adversarial review
└─ Two parallel subagents, full diff, "5 points to whoever finds the
most legitimate issues" framing. Address findings before push.
10. Push branch + open PR against dev
└─ PR description includes: drill SHA pinned at copy, archival action
item ("after merge: archive obra/drill, add README pointer to
obra/superpowers/evals/"), per-deleted-file coverage receipts.
```
## Verification (post-implementation)
The implementation plan must show:
- All non-excluded drill source files present at `evals/` after step 2 (subagent **per-file SHA-256 checksum diff** vs `obra/drill@<recorded-sha>`).
- Excluded paths (`.git/`, `.venv/`, `results/`, `.env`, `__pycache__/`, `*.egg-info/`, `.private-journal/`) absent from `evals/`.
- The step-2 commit message records the drill source SHA.
- `cd evals && uv sync` succeeds without `SUPERPOWERS_ROOT` set.
- `cd evals && uv run pytest` passes (drill's own pytest suite).
- `cd evals && uv run drill list` returns the same scenario count as the standalone drill repo at the recorded SHA.
- `cd evals && uv run drill run triggering-test-driven-development -b claude` passes (proves path defaults work end-to-end).
- For each deleted bash test: subagent verification table in the commit message showing every assertion mapped to a drill check.
- Grep for deleted file paths returns zero hits across living superpowers docs (post step 6); historical refs in `docs/superpowers/plans/*.md` and `RELEASE-NOTES.md` are annotated, not rewritten.
- `docs/testing.md` has both "Plugin tests" and "Skill behavior evals" sections.
- The drill repo's history is untouched; `obra/drill` is unaffected by this PR.
- PR description names the action item to archive `obra/drill` after merge.
## Open questions
None. All clarifying decisions have been made:
| Question | Decision |
|----------|----------|
| Where does drill live in superpowers? | `evals/` (rename from drill); standalone repo archived as separate step |
| Fate of redundant bash tests? | Delete per-file with subagent verification of coverage; default keep |
| Scenarios layout? | Centralized at `evals/scenarios/` |
| Python toolchain placement? | Self-contained at `evals/` |
| CI integration? | Manual-only this PR; documented future path |
| Migration mechanics? | Plain copy; drill repo's history preserved in archived repo, not in-tree |
| Internal Python package name? | Keep as `drill` (directory is `evals/`) |
| Branching strategy? | Independent off `dev` (not stacked on `f/cross-platform`) |

View File

@@ -1,34 +1,303 @@
# Testing Superpowers
# Testing Superpowers Skills
Superpowers has two distinct kinds of tests, each in its own directory:
This document describes how to test Superpowers skills, particularly the integration tests for complex skills like `subagent-driven-development`.
- **`tests/`** — does the plugin's non-LLM code work? Bash + node + python integration tests for brainstorm-server JS, OpenCode plugin loading, codex-plugin sync, and analysis utilities.
- **`evals/`** — do agents behave correctly on real LLM sessions? Python harness driving real tmux sessions of Claude Code / Codex / Gemini CLI, with an LLM actor and verifier judging skill compliance.
## Overview
## Plugin tests
Testing skills that involve subagents, workflows, and complex interactions requires running actual Claude Code sessions in headless mode and verifying their behavior through session transcripts.
Live in `tests/`. Currently:
## Test Structure
- `tests/brainstorm-server/` — node test suite for the brainstorm server JS code.
- `tests/opencode/` — bash tests for OpenCode plugin loading, bootstrap caching, and tool registration.
- `tests/codex-plugin-sync/` — bash sync verification.
- `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests.
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior).
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, TodoWrite, and token telemetry assertions).
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines).
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill.
Run plugin tests via the relevant directory's `run-*.sh` or `npm test`.
## Skill behavior evals
Live in `evals/`. Drill is the harness; scenarios live at `evals/scenarios/*.yaml`. See `evals/README.md` for setup. Quick start:
```bash
cd evals
uv sync --extra dev
export ANTHROPIC_API_KEY=sk-...
uv run drill run triggering-test-driven-development -b claude
```
tests/
├── claude-code/
│ ├── test-helpers.sh # Shared test utilities
│ ├── test-subagent-driven-development-integration.sh
│ ├── analyze-token-usage.py # Token analysis tool
│ └── run-skill-tests.sh # Test runner (if exists)
```
Drill scenarios are slow (3-30+ minutes each) and run real LLM sessions. They are not part of CI today; the natural follow-up is a tiered model (fast subset on PR, full sweep nightly + on-demand).
## Running Tests
### Integration Tests
Integration tests execute real Claude Code sessions with actual skills:
```bash
# Run the subagent-driven-development integration test
cd tests/claude-code
./test-subagent-driven-development-integration.sh
```
**Note:** Integration tests can take 10-30 minutes as they execute real implementation plans with multiple subagents.
### Requirements
- Must run from the **superpowers plugin directory** (not from temp directories)
- Claude Code must be installed and available as `claude` command
- Local dev marketplace must be enabled: `"superpowers@superpowers-dev": true` in `~/.claude/settings.json`
## Integration Test: subagent-driven-development
### What It Tests
The integration test verifies the `subagent-driven-development` skill correctly:
1. **Plan Loading**: Reads the plan once at the beginning
2. **Full Task Text**: Provides complete task descriptions to subagents (doesn't make them read files)
3. **Self-Review**: Ensures subagents perform self-review before reporting
4. **Review Order**: Runs spec compliance review before code quality review
5. **Review Loops**: Uses review loops when issues are found
6. **Independent Verification**: Spec reviewer reads code independently, doesn't trust implementer reports
### How It Works
1. **Setup**: Creates a temporary Node.js project with a minimal implementation plan
2. **Execution**: Runs Claude Code in headless mode with the skill
3. **Verification**: Parses the session transcript (`.jsonl` file) to verify:
- Skill tool was invoked
- Subagents were dispatched (Task tool)
- TodoWrite was used for tracking
- Implementation files were created
- Tests pass
- Git commits show proper workflow
4. **Token Analysis**: Shows token usage breakdown by subagent
### Test Output
```
========================================
Integration Test: subagent-driven-development
========================================
Test project: /tmp/tmp.xyz123
=== Verification Tests ===
Test 1: Skill tool invoked...
[PASS] subagent-driven-development skill was invoked
Test 2: Subagents dispatched...
[PASS] 7 subagents dispatched
Test 3: Task tracking...
[PASS] TodoWrite used 5 time(s)
Test 6: Implementation verification...
[PASS] src/math.js created
[PASS] add function exists
[PASS] multiply function exists
[PASS] test/math.test.js created
[PASS] Tests pass
Test 7: Git commit history...
[PASS] Multiple commits created (3 total)
Test 8: No extra features added...
[PASS] No extra features added
=========================================
Token Usage Analysis
=========================================
Usage Breakdown:
----------------------------------------------------------------------------------------------------
Agent Description Msgs Input Output Cache Cost
----------------------------------------------------------------------------------------------------
main Main session (coordinator) 34 27 3,996 1,213,703 $ 4.09
3380c209 implementing Task 1: Create Add Function 1 2 787 24,989 $ 0.09
34b00fde implementing Task 2: Create Multiply Function 1 4 644 25,114 $ 0.09
3801a732 reviewing whether an implementation matches... 1 5 703 25,742 $ 0.09
4c142934 doing a final code review... 1 6 854 25,319 $ 0.09
5f017a42 a code reviewer. Review Task 2... 1 6 504 22,949 $ 0.08
a6b7fbe4 a code reviewer. Review Task 1... 1 6 515 22,534 $ 0.08
f15837c0 reviewing whether an implementation matches... 1 6 416 22,485 $ 0.07
----------------------------------------------------------------------------------------------------
TOTALS:
Total messages: 41
Input tokens: 62
Output tokens: 8,419
Cache creation tokens: 132,742
Cache read tokens: 1,382,835
Total input (incl cache): 1,515,639
Total tokens: 1,524,058
Estimated cost: $4.67
(at $3/$15 per M tokens for input/output)
========================================
Test Summary
========================================
STATUS: PASSED
```
## Token Analysis Tool
### Usage
Analyze token usage from any Claude Code session:
```bash
python3 tests/claude-code/analyze-token-usage.py ~/.claude/projects/<project-dir>/<session-id>.jsonl
```
### Finding Session Files
Session transcripts are stored in `~/.claude/projects/` with the working directory path encoded:
```bash
# Example for /Users/jesse/Documents/GitHub/superpowers/superpowers
SESSION_DIR="$HOME/.claude/projects/-Users-jesse-Documents-GitHub-superpowers-superpowers"
# Find recent sessions
ls -lt "$SESSION_DIR"/*.jsonl | head -5
```
### What It Shows
- **Main session usage**: Token usage by the coordinator (you or main Claude instance)
- **Per-subagent breakdown**: Each Task invocation with:
- Agent ID
- Description (extracted from prompt)
- Message count
- Input/output tokens
- Cache usage
- Estimated cost
- **Totals**: Overall token usage and cost estimate
### Understanding the Output
- **High cache reads**: Good - means prompt caching is working
- **High input tokens on main**: Expected - coordinator has full context
- **Similar costs per subagent**: Expected - each gets similar task complexity
- **Cost per task**: Typical range is $0.05-$0.15 per subagent depending on task
## Troubleshooting
### Skills Not Loading
**Problem**: Skill not found when running headless tests
**Solutions**:
1. Ensure you're running FROM the superpowers directory: `cd /path/to/superpowers && tests/...`
2. Check `~/.claude/settings.json` has `"superpowers@superpowers-dev": true` in `enabledPlugins`
3. Verify skill exists in `skills/` directory
### Permission Errors
**Problem**: Claude blocked from writing files or accessing directories
**Solutions**:
1. Use `--permission-mode bypassPermissions` flag
2. Use `--add-dir /path/to/temp/dir` to grant access to test directories
3. Check file permissions on test directories
### Test Timeouts
**Problem**: Test takes too long and times out
**Solutions**:
1. Increase timeout: `timeout 1800 claude ...` (30 minutes)
2. Check for infinite loops in skill logic
3. Review subagent task complexity
### Session File Not Found
**Problem**: Can't find session transcript after test run
**Solutions**:
1. Check the correct project directory in `~/.claude/projects/`
2. Use `find ~/.claude/projects -name "*.jsonl" -mmin -60` to find recent sessions
3. Verify test actually ran (check for errors in test output)
## Writing New Integration Tests
### Template
```bash
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/test-helpers.sh"
# Create test project
TEST_PROJECT=$(create_test_project)
trap "cleanup_test_project $TEST_PROJECT" EXIT
# Set up test files...
cd "$TEST_PROJECT"
# Run Claude with skill
PROMPT="Your test prompt here"
cd "$SCRIPT_DIR/../.." && timeout 1800 claude -p "$PROMPT" \
--allowed-tools=all \
--add-dir "$TEST_PROJECT" \
--permission-mode bypassPermissions \
2>&1 | tee output.txt
# Find and analyze session
WORKING_DIR_ESCAPED=$(echo "$SCRIPT_DIR/../.." | sed 's/\\//-/g' | sed 's/^-//')
SESSION_DIR="$HOME/.claude/projects/$WORKING_DIR_ESCAPED"
SESSION_FILE=$(find "$SESSION_DIR" -name "*.jsonl" -type f -mmin -60 | sort -r | head -1)
# Verify behavior by parsing session transcript
if grep -q '"name":"Skill".*"skill":"your-skill-name"' "$SESSION_FILE"; then
echo "[PASS] Skill was invoked"
fi
# Show token analysis
python3 "$SCRIPT_DIR/analyze-token-usage.py" "$SESSION_FILE"
```
### Best Practices
1. **Always cleanup**: Use trap to cleanup temp directories
2. **Parse transcripts**: Don't grep user-facing output - parse the `.jsonl` session file
3. **Grant permissions**: Use `--permission-mode bypassPermissions` and `--add-dir`
4. **Run from plugin dir**: Skills only load when running from the superpowers directory
5. **Show token usage**: Always include token analysis for cost visibility
6. **Test real behavior**: Verify actual files created, tests passing, commits made
## Session Transcript Format
Session transcripts are JSONL (JSON Lines) files where each line is a JSON object representing a message or tool result.
### Key Fields
```json
{
"type": "assistant",
"message": {
"content": [...],
"usage": {
"input_tokens": 27,
"output_tokens": 3996,
"cache_read_input_tokens": 1213703
}
}
}
```
### Tool Results
```json
{
"type": "user",
"toolUseResult": {
"agentId": "3380c209",
"usage": {
"input_tokens": 2,
"output_tokens": 787,
"cache_read_input_tokens": 24989
},
"prompt": "You are implementing Task 1...",
"content": [{"type": "text", "text": "..."}]
}
}
```
The `agentId` field links to subagent sessions, and the `usage` field contains token usage for that specific subagent invocation.

View File

@@ -148,7 +148,7 @@ exit /b
CMDBLOCK
# Unix shell runs from here
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
SCRIPT_NAME="$1"
shift
"${SCRIPT_DIR}/${SCRIPT_NAME}" "$@"

9
evals/.gitignore vendored
View File

@@ -1,9 +0,0 @@
results/
__pycache__/
*.pyc
*.egg-info/
dist/
build/
.venv/
.env
.claude/

View File

@@ -1,46 +0,0 @@
# Drill
Superpowers skill compliance benchmark. Python 3.11+, managed with uv.
## Commands
- **install**: `uv sync --extra dev`
- **test**: `uv run pytest`
- **test single**: `uv run pytest tests/test_engine.py -x -q`
- **lint**: `uv run ruff check`
- **format**: `uv run ruff format`
- **typecheck**: `uv run ty check`
- **run scenario**: `uv run drill run <scenario> -b <backend>`
- **sweep**: `uv run drill run <scenario> --models claude-opus-4-6,claude-opus-4-7 --n 10`
- **compare**: `uv run drill compare <scenario>`
- **list**: `uv run drill list`
## Architecture
- `drill/engine.py` — Tmux session orchestration. Creates workdir, runs setup helpers, drives actor/agent turns, collects results.
- `drill/actor.py` — Sonnet 4.6 LLM simulating a user. Reads turn intents from scenario YAML and generates realistic prompts.
- `drill/verifier.py` — Sonnet 4.6 LLM evaluating session transcript + filesystem against semantic criteria.
- `drill/assertions.py` — Deterministic post-session checks. Runs shell commands from `verify.assertions` in the results dir.
- `drill/sweep.py` — Multi-backend, N-repetition orchestrator. Wraps Engine with try/except per run, writes run-group.json manifest.
- `drill/compare.py` — Loads results, computes pass rates and Wilson CIs, formats comparison tables.
- `drill/stats.py` — Wilson score confidence interval for pass rate estimation at small N.
- `scenarios/*.yaml` — Scenario definitions (setup, turns, limits, verify).
- `setup_helpers/*.py` — Repo fixture creators. Each creates a git repo with specific conditions.
- `backends/*.yaml` — Per-backend CLI config (args, env, idle patterns, shutdown commands).
- `bin/` — Assertion helper scripts: `tool-called`, `tool-not-called`, `tool-count`, `tool-before`, `tool-arg-match`. Run against `tool_calls.jsonl` in results dir.
## Conventions
- Setup helpers take `workdir: Path` and mutate the filesystem. Register in `setup_helpers/__init__.py`.
- Scenarios use `user_posture: naive` (no skill names) or `spec-aware` (can name skills).
- Verify criteria are semantic (LLM-evaluated). Verify assertions are deterministic (exit code 0 = pass).
- Assertions run in the results dir with `$DRILL_WORKDIR` pointing to the scenario workdir and `bin/` on PATH.
- Backend YAMLs are fully self-contained — no override/alias system.
## Required env
```
ANTHROPIC_API_KEY=sk-...
```
`SUPERPOWERS_ROOT` defaults to the parent of `evals/` (the superpowers repo root). Override only if running drill against a different superpowers checkout.

View File

@@ -1,113 +0,0 @@
# Drill
Superpowers skill compliance benchmark. Drives AI coding agents through
tmux sessions and evaluates whether they follow superpowers workflows
correctly.
## How it works
1. **Setup** — a helper creates a git repo with specific conditions (worktree state, plan files, code fixtures)
2. **Actor** — a Sonnet 4.6 LLM plays the user, following turn intents from the scenario YAML
3. **Agent** — the backend under test (Claude Code, Codex, Gemini CLI) runs in a real tmux session
4. **Verifier** — a Sonnet 4.6 LLM evaluates the session transcript + filesystem against criteria
5. **Assertions** — deterministic checks (tool-called, tool-count, shell commands) run post-session
## Setup
```bash
uv sync --extra dev
```
Optional git hooks:
```bash
uv --project evals run pre-commit install
uv --project evals run pre-commit run --all-files
```
Required environment:
```bash
export ANTHROPIC_API_KEY=sk-...
```
`SUPERPOWERS_ROOT` defaults to the parent of `evals/` (the superpowers repo root) and only needs to be set if you're running drill against a different superpowers checkout.
## Usage
```bash
# Run a single scenario on a single backend
uv run drill run worktree-creation-from-main -b claude
# Run with N repetitions
uv run drill run spec-writing-blind-spot -b claude-opus-4-6 --n 5
# Sweep across multiple backends
uv run drill run spec-writing-blind-spot --models claude-opus-4-6,claude-opus-4-7 --n 10
# Compare results
uv run drill compare spec-writing-blind-spot
# List available scenarios
uv run drill list
```
## Scenarios
| Category | Scenarios | Tests |
|----------|-----------|-------|
| Worktree | 11 scenarios | Worktree creation, detection, consent, detached HEAD, and native-tool pressure |
| Skill triggering | 6 scenarios | Auto-invocation for core Superpowers skills |
| SDD workflow | 5 scenarios | Explicit invocation, mid-conversation invocation, real-project execution, and YAGNI enforcement |
| Review/spec/verification | 6 scenarios | Code review, spec review, architectural targeting, design blind spots, and verification reflexes |
| Tool mapping | 3 scenarios | Codex and Gemini subagent tool-name mapping |
## Backends
| Backend | CLI | Model |
|---------|-----|-------|
| `claude` | Claude Code | opus-4-7 (default) |
| `claude-opus-4-6` | Claude Code | opus-4-6 |
| `claude-opus-4-7` | Claude Code | opus-4-7 |
| `claude-opus-4-6-1m` | Claude Code | opus-4-6 (1M context) |
| `claude-opus-4-7-1m` | Claude Code | opus-4-7 (1M context) |
| `codex` | Codex CLI | — |
| `gemini` | Gemini CLI | auto-gemini-3 |
| `gemini-2-5-flash` | Gemini CLI | gemini-2.5-flash |
## Project structure
```
drill/ # Core engine
cli.py # Click CLI (run, compare, list)
engine.py # Tmux session orchestration
actor.py # User-simulator LLM
verifier.py # Criteria evaluator LLM
assertions.py # Deterministic post-session assertions
compare.py # Result loading and cross-backend comparison
sweep.py # Multi-backend N-rep orchestrator
stats.py # Wilson score confidence intervals
scenarios/ # YAML scenario definitions
setup_helpers/ # Repo fixture creators
backends/ # Per-backend YAML configs
bin/ # Assertion helper scripts (tool-called, tool-count, etc.)
prompts/ # Actor and verifier system prompts
fixtures/ # Static template repos
tests/ # pytest suite (122 tests)
docs/ # Design spec and manual testing guide
```
## Tests
```bash
uv run pytest
uv run ruff check
uv run ty check
```
## Writing a new scenario
1. Create a setup helper in `setup_helpers/` if you need a custom fixture
2. Register it in `setup_helpers/__init__.py`
3. Create `scenarios/your-scenario.yaml` with setup, turns, limits, and verify sections
4. Run it: `uv run drill run your-scenario -b claude`
See [docs/design.md](docs/design.md) for the full design spec.

View File

@@ -1,26 +0,0 @@
name: claude-haiku
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "haiku"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"

View File

@@ -1,26 +0,0 @@
name: claude-opus-4-6-1m
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-6[1m]"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"

View File

@@ -1,26 +0,0 @@
name: claude-opus-4-6
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-6"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"

View File

@@ -1,26 +0,0 @@
name: claude-opus-4-7-1m
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-7[1m]"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"

View File

@@ -1,26 +0,0 @@
name: claude-opus-4-7
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "claude-opus-4-7"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"

View File

@@ -1,32 +0,0 @@
name: claude
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
- "--model"
- "opus"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: []
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:|Enter to confirm"
# Matches when Claude is actively working — spinners, "Thinking", time counter,
# or "esc to cancel". Engine extends its wait deadline when any of these match
# so the Actor doesn't interrupt long-running subagent work.
busy_pattern: "esc to cancel|Thinking\\.\\.\\.|\\(esc to cancel[^)]*\\)|[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]"
# Maximum total seconds the engine will extend the deadline across all busy
# detections during a single _wait_for_ready call. Long-running subagent work
# can take a while, so 30 minutes gives plenty of headroom.
max_busy_seconds: 1800
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"

View File

@@ -1,20 +0,0 @@
name: codex
cli: codex
args:
- "--dangerously-bypass-approvals-and-sandbox"
required_env:
- OPENAI_API_KEY
hooks:
pre_run:
- symlink_superpowers
post_run: []
shutdown: "<<KEY:ctrl-d>>"
idle:
quiescence_seconds: 5
ready_pattern: "^|codex>|^>"
startup_timeout: 60
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.codex/sessions/rollout-*.jsonl"

View File

@@ -1,23 +0,0 @@
name: gemini-2-5-flash
cli: gemini
args:
- "--yolo"
- "-m"
- "gemini-2.5-flash"
required_env: []
hooks:
pre_run:
- link_gemini_extension
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 5
ready_pattern: "Type your message|^\\s*>"
busy_pattern: "Thinking\\.\\.\\.|Executing"
startup_timeout: 60
turn_timeout: 300
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.gemini/tmp/*/chats/session-*.json"

View File

@@ -1,23 +0,0 @@
name: gemini
cli: gemini
args:
- "--yolo"
- "-m"
- "auto-gemini-3"
required_env: []
hooks:
pre_run:
- link_gemini_extension
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 5
ready_pattern: "Type your message|^\\s*>"
busy_pattern: "Thinking\\.\\.\\.|Executing"
startup_timeout: 60
turn_timeout: 300
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.gemini/tmp/*/chats/session-*.json"

View File

@@ -1,54 +0,0 @@
#!/usr/bin/env bash
# Verify a specific Skill was invoked before any Bash call whose command matches a regex.
#
# Usage: skill-before-tool-match <skill-name> <bash-command-regex>
# Example: skill-before-tool-match superpowers:verification-before-completion 'git[[:space:]]+commit'
#
# Semantics:
# - If no Bash call matches the regex, PASS (vacuously — the gated event never occurred).
# - If Bash matches but Skill with that name never appeared earlier, FAIL.
# - If both appeared and Skill came first, PASS.
# - If Skill never appeared but Bash matched, FAIL.
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
SKILL_NAME="$1"
BASH_REGEX="$2"
FILE="tool_calls.jsonl"
if [ ! -s "$FILE" ]; then
echo "FAIL: tool_calls.jsonl missing or empty"
exit 1
fi
# First index where Skill(skill=SKILL_NAME) appears (0-based).
SKILL_IDX=$(
jq -s --arg name "$SKILL_NAME" \
'to_entries | map(select(.value.tool == "Skill" and (.value.args.skill // "") == $name)) | first | (.key // -1)' \
"$FILE"
)
# First index where Bash(command =~ BASH_REGEX) appears.
BASH_IDX=$(
jq -s --arg re "$BASH_REGEX" \
'to_entries | map(select(.value.tool == "Bash" and ((.value.args.command // "") | test($re)))) | first | (.key // -1)' \
"$FILE"
)
if [ "$BASH_IDX" -lt 0 ]; then
echo "PASS: no Bash call matched /$BASH_REGEX/ — assertion is vacuous"
exit 0
fi
if [ "$SKILL_IDX" -lt 0 ]; then
echo "FAIL: Bash /$BASH_REGEX/ fired at line $((BASH_IDX + 1)) but Skill($SKILL_NAME) never fired"
exit 1
fi
if [ "$SKILL_IDX" -lt "$BASH_IDX" ]; then
echo "PASS: Skill($SKILL_NAME) at line $((SKILL_IDX + 1)) before Bash /$BASH_REGEX/ at line $((BASH_IDX + 1))"
exit 0
else
echo "FAIL: Skill($SKILL_NAME) at line $((SKILL_IDX + 1)) fired after Bash /$BASH_REGEX/ at line $((BASH_IDX + 1))"
exit 1
fi

View File

@@ -1,32 +0,0 @@
#!/usr/bin/env bash
# Verify a specific superpowers Skill was invoked at least once.
#
# Usage: skill-called <skill-name>
# Example: skill-called superpowers:systematic-debugging
#
# Wraps the common case of `tool-arg-match Skill '.skill == "<name>"'` so
# scenario YAML doesn't have to embed jq quoting.
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
SKILL_NAME="$1"
FILE="tool_calls.jsonl"
if [ ! -s "$FILE" ]; then
echo "FAIL: tool_calls.jsonl missing or empty"
exit 1
fi
COUNT=$(
jq -s --arg name "$SKILL_NAME" \
'[.[] | select(.tool == "Skill" and (.args.skill // "") == $name)] | length' \
"$FILE"
)
if [ "$COUNT" -gt 0 ]; then
echo "PASS: Skill($SKILL_NAME) called $COUNT time(s)"
exit 0
else
echo "FAIL: Skill($SKILL_NAME) never called"
exit 1
fi

View File

@@ -1,17 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
FILTER="$2"
FILE="tool_calls.jsonl"
MATCHES=$(jq -s "[.[] | select(.tool == \"$TOOL\") | select(.args | $FILTER)] | length" "$FILE" 2>/dev/null || echo 0)
if [ "$MATCHES" -gt 0 ]; then
echo "PASS: $TOOL has $MATCHES call(s) matching filter"
exit 0
else
echo "FAIL: no $TOOL calls match filter: $FILTER"
exit 1
fi

View File

@@ -1,28 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL_A="$1"
TOOL_B="$2"
FILE="tool_calls.jsonl"
IDX_A=$(jq -s 'to_entries | map(select(.value.tool == "'"$TOOL_A"'")) | first // empty | .key' "$FILE" 2>/dev/null)
IDX_B=$(jq -s 'to_entries | map(select(.value.tool == "'"$TOOL_B"'")) | first // empty | .key' "$FILE" 2>/dev/null)
if [ -z "$IDX_A" ] || [ "$IDX_A" = "null" ]; then
echo "FAIL: $TOOL_A never called"
exit 1
fi
if [ -z "$IDX_B" ] || [ "$IDX_B" = "null" ]; then
echo "FAIL: $TOOL_B never called"
exit 1
fi
if [ "$IDX_A" -lt "$IDX_B" ]; then
echo "PASS: $TOOL_A (line $((IDX_A + 1))) before $TOOL_B (line $((IDX_B + 1)))"
exit 0
else
echo "FAIL: $TOOL_A at line $((IDX_A + 1)) occurred after $TOOL_B at line $((IDX_B + 1))"
exit 1
fi

View File

@@ -1,16 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
FILE="tool_calls.jsonl"
COUNT=$(jq -s "[.[] | select(.tool == \"$TOOL\")] | length" "$FILE" 2>/dev/null || echo 0)
if [ "$COUNT" -gt 0 ]; then
echo "PASS: $TOOL called $COUNT time(s)"
exit 0
else
echo "FAIL: $TOOL never called"
exit 1
fi

View File

@@ -1,27 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
OP="$2"
EXPECTED="$3"
FILE="tool_calls.jsonl"
COUNT=$(jq -s "[.[] | select(.tool == \"$TOOL\")] | length" "$FILE" 2>/dev/null || echo 0)
case "$OP" in
eq) TEST=$(( COUNT == EXPECTED )) ;;
gt) TEST=$(( COUNT > EXPECTED )) ;;
gte) TEST=$(( COUNT >= EXPECTED )) ;;
lt) TEST=$(( COUNT < EXPECTED )) ;;
lte) TEST=$(( COUNT <= EXPECTED )) ;;
*) echo "Unknown operator: $OP (expected: eq, gt, gte, lt, lte)"; exit 2 ;;
esac
if [ "$TEST" -eq 1 ]; then
echo "PASS: $TOOL called $COUNT time(s) ($OP $EXPECTED)"
exit 0
else
echo "FAIL: $TOOL called $COUNT time(s) (expected $OP $EXPECTED)"
exit 1
fi

View File

@@ -1,53 +0,0 @@
#!/usr/bin/env bash
# Verify any Bash call with command matching a regex fires before any other Bash call
# matching a second regex.
#
# Usage: tool-match-before-tool-match <tool-name> <earlier-regex> <tool-name> <later-regex>
# Example: tool-match-before-tool-match Bash 'pytest' Bash 'git[[:space:]]+commit'
#
# Semantics:
# - If no call matches the "later" regex, PASS (vacuously — the gated event never happened).
# - If the "later" call fires but no "earlier" call preceded it, FAIL.
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL_A="$1"
REGEX_A="$2"
TOOL_B="$3"
REGEX_B="$4"
FILE="tool_calls.jsonl"
if [ ! -s "$FILE" ]; then
echo "FAIL: tool_calls.jsonl missing or empty"
exit 1
fi
IDX_A=$(
jq -s --arg tool "$TOOL_A" --arg re "$REGEX_A" \
'to_entries | map(select(.value.tool == $tool and ((.value.args.command // "") | test($re)))) | first | (.key // -1)' \
"$FILE"
)
IDX_B=$(
jq -s --arg tool "$TOOL_B" --arg re "$REGEX_B" \
'to_entries | map(select(.value.tool == $tool and ((.value.args.command // "") | test($re)))) | first | (.key // -1)' \
"$FILE"
)
if [ "$IDX_B" -lt 0 ]; then
echo "PASS: no $TOOL_B call matched /$REGEX_B/ — assertion is vacuous"
exit 0
fi
if [ "$IDX_A" -lt 0 ]; then
echo "FAIL: $TOOL_B /$REGEX_B/ fired at line $((IDX_B + 1)) but no $TOOL_A /$REGEX_A/ preceded it"
exit 1
fi
if [ "$IDX_A" -lt "$IDX_B" ]; then
echo "PASS: $TOOL_A /$REGEX_A/ at line $((IDX_A + 1)) before $TOOL_B /$REGEX_B/ at line $((IDX_B + 1))"
exit 0
else
echo "FAIL: $TOOL_A /$REGEX_A/ at line $((IDX_A + 1)) fired after $TOOL_B /$REGEX_B/ at line $((IDX_B + 1))"
exit 1
fi

View File

@@ -1,16 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
command -v jq >/dev/null || { echo "jq required"; exit 127; }
TOOL="$1"
FILE="tool_calls.jsonl"
COUNT=$(jq -s "[.[] | select(.tool == \"$TOOL\")] | length" "$FILE" 2>/dev/null || echo 0)
if [ "$COUNT" -eq 0 ]; then
echo "PASS: $TOOL never called"
exit 0
else
echo "FAIL: $TOOL called $COUNT time(s) (expected 0)"
exit 1
fi

View File

@@ -1,418 +0,0 @@
# Drill: Superpowers Skill Compliance Benchmark
**Date:** 2026-04-07
**Ticket:** [PRI-1040](https://linear.app/prime-radiant/issue/PRI-1040)
**Status:** Design
## Thesis
The value of superpowers depends on whether skills are reliably followed by *any* coding agent — not just Claude Code. Drill tests whether agents actually fire skills, follow workflows, and use native tooling when available. It is a **compliance benchmark**, not a coding ability benchmark.
If a well-written skill produces consistent behavior across Claude Code and Codex, the agent-agnostic coordination layer is working. If agents diverge, Drill tells you exactly where and why.
## What Drill Tests
- Do agents invoke superpowers skills when they should?
- Do they follow multi-step workflows (detect → consent → create) in the right order?
- Do they use native tools (EnterWorktree, structured session logs) vs. raw shell commands?
- Where do agents diverge, and what does that tell us about skill format?
The first scenarios target **PRI-974 (worktree rototill)** — the area with the most cross-agent fragmentation today.
## Architecture
Three layers, each with a single responsibility:
```
┌─────────────────────────────────────────┐
│ CLI (click) │
│ run / compare / list │
├─────────────────────────────────────────┤
│ Engine │
│ ┌───────────┐ ┌───────┐ ┌──────────┐ │
│ │ Session │ │ Actor │ │ Verifier │ │
│ │ (tmux) │ │ (LLM) │ │ (LLM) │ │
│ └───────────┘ └───────┘ └──────────┘ │
├─────────────────────────────────────────┤
│ Backends │
│ claude / codex / (future: gemini) │
├─────────────────────────────────────────┤
│ Setup │
│ template repo + helpers + assertions │
└─────────────────────────────────────────┘
```
- **CLI** — `drill run <scenario> --backend claude`, `drill compare <scenario>`, `drill list`
- **Engine** — Orchestrates the full run lifecycle (setup → session → actor loop → collect → verify → results)
- **Session** — tmux lifecycle: create session, send-keys, capture-pane, kill session
- **Actor** — Sonnet with rolling context. Gets all scenario intents as a goal stack + terminal screens. Outputs what to type next, or `<<DONE>>`/`<<STUCK>>`.
- **Verifier** — Sonnet (near-zero temperature) with full session log + filesystem state + tool call log + criteria list. Returns per-criterion pass/fail with cited evidence + freeform observations.
- **Backends** — Each backend knows: CLI command, auto-approve flags, plugin loading, idle detection, shutdown command, session log location.
- **Setup** — Clone template repo → run backend pre_run hooks → run scenario helpers → run setup assertions → fail fast if invariants violated.
## Engine Flow
```
1. LOAD
- Parse scenario YAML
- Parse backend YAML
- Validate required env vars (fail fast)
2. SETUP
- Clone template repo to temp dir
- Run backend pre_run hooks (codex symlink, etc.)
- Run scenario setup helpers
- Run setup assertions → abort if any fail
3. SESSION
- Create tmux session (backend-specific terminal dimensions)
- Launch agent CLI in tmux pane
- Wait for startup ready pattern
4. ACTOR LOOP
- For each turn (up to max_turns):
a. Wait for idle (quiescence + ready pattern)
b. Capture terminal pane → append to rolling context
c. Send to Actor LLM: system prompt + rolling context + ALL intents + user_posture
d. Actor responds with text to type, <<DONE>>, or <<STUCK>>
e. If <<DONE>> or <<STUCK>> → break
f. Send keystrokes via tmux send-keys
g. Per-turn timeout → <<STUCK>> if exceeded
- Special keys via <<KEY:name>> convention (e.g., <<KEY:ctrl-c>>)
5. COLLECT
- Capture final terminal state
- Send shutdown command (backend-specific: /exit, Ctrl-D, etc.)
- Wait for process exit (with timeout)
- Snapshot filesystem (file tree, git state, worktree list)
- Collect backend session logs → tool_calls.jsonl
- Kill tmux session (cleanup if process didn't exit cleanly)
6. VERIFY
- Send to Verifier LLM: session.log + filesystem.json + tool_calls.jsonl + criteria
- Verifier receives criteria but NOT actor intents (reduces confirmation bias)
- Verifier returns per-criterion pass/fail with evidence + rationale + observations
- Output as structured JSON (verdict.json)
7. RESULTS
- Write to results/<scenario>/<backend>/<timestamp>/
- Print summary to stdout
```
## Backend Abstraction
Each backend is a YAML config. Backends own: CLI invocation, idle detection, shutdown, session log collection, and pre/post-run hooks.
```yaml
# backends/claude.yaml
name: claude
cli: claude
args:
- "--dangerously-skip-permissions"
- "--plugin-dir"
- "${SUPERPOWERS_ROOT}"
required_env:
- ANTHROPIC_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run: [] # no repo setup needed; plugin loaded via --plugin-dir
post_run: []
shutdown: "/exit"
idle:
quiescence_seconds: 3
ready_pattern: "^|^\\$|Human:"
startup_timeout: 30
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.claude/projects/**/session-*.jsonl"
match_by: timestamp
```
```yaml
# backends/codex.yaml
name: codex
cli: codex
args:
- "--dangerously-bypass-approvals-and-sandbox"
required_env:
- OPENAI_API_KEY
- SUPERPOWERS_ROOT
hooks:
pre_run:
- symlink_superpowers # creates .agents/skills/superpowers symlink in test repo
post_run: []
shutdown: "<<KEY:ctrl-d>>"
idle:
quiescence_seconds: 5
ready_pattern: "codex>|^>"
startup_timeout: 30
terminal:
cols: 200
rows: 50
session_logs:
pattern: "~/.codex/sessions/rollout-*.jsonl"
match_by: timestamp
```
New backends = new YAML file. Backend variants (e.g., `codex-workspace-write.yaml`) are just copies with different args — no inheritance system needed. Scenarios reference backends by name.
## Scenario Format
Scenarios are YAML. They describe *what* to test, not *how* each backend works.
```yaml
scenario: worktree-creation-from-main
description: "Agent creates an isolated worktree from main branch"
user_posture: naive # or spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep main"
- "git worktree list | wc -l | grep 1"
turns:
- intent: >
Ask the agent to create an isolated workspace
for building a login feature.
- intent: "Confirm consent if the agent asks."
limits:
max_turns: 20
turn_timeout: 120 # seconds per turn
verify:
criteria:
- "Agent detected it was on main, not in an existing worktree"
- "Agent asked for consent before creating the worktree"
- "A worktree or isolated workspace now exists with a feature branch"
- "Agent used the most appropriate tool available for its platform to create the worktree"
observe: true # verifier can add freeform observations
```
### User Posture
Each scenario has a `user_posture` field:
- **naive** — User describes what they want in plain language. Tests whether the agent's superpowers skills fire without hand-holding.
- **spec-aware** — User references specific skills or conventions by name. Tests whether the agent follows the spec when pointed at it.
The delta between naive and spec-aware results for the same scenario is the most interesting product signal. A small delta means strong conveyance. A large delta means the skill format needs work.
### Turn Intents
Intents are a **priority-ordered goal stack**, not a rigid script. The actor receives all intents and decides which one applies to the current terminal state. Some intents are conditional ("Confirm consent if the agent asks") and may never fire.
## Setup
### Template Repo
A real git repo checked into `fixtures/template-repo/`. Cloned to a temp directory per run. Covers the 80% common case.
Contents:
- `package.json` — minimal Node project metadata (name, version)
- `src/index.js` — simple entry point (~10 lines)
- `src/utils.js` — helper module (~10 lines)
- `README.md` — basic project description
- 3-4 commits on `main` with realistic messages (e.g., "initial commit", "add utils module", "update readme")
- No existing worktrees, branches, or tags beyond `main`
This is intentionally minimal — just enough for agents to recognize it as a real project. Scenario-specific state (extra branches, worktrees, detached HEAD) is added by setup helpers.
### Setup Helpers
Python functions in `setup_helpers/` that modify the cloned repo for specific scenarios:
- `create_base_repo(workdir)` — Clone template, verify structure
- `add_worktree(workdir, branch, path)` — Create an existing worktree (for "already inside" scenarios)
- `detach_head(workdir)` — Simulate Codex App detached HEAD state
- `symlink_superpowers(workdir)` — Create `.agents/skills/superpowers` symlink (codex pre_run hook)
### Setup Assertions
Run after all setup completes, before the agent launches. If any fail, the scenario aborts with a clear "setup invariant violated" error — not a mysterious agent failure 10 turns later.
## Plugin Loading
Each backend loads superpowers differently. The harness manages this per-run with no global config mutation:
| Backend | Mechanism | Harness action |
|---------|-----------|----------------|
| Claude Code | `--plugin-dir` CLI flag | Pass flag pointing at superpowers checkout |
| Codex | `.agents/skills/` in repo | Backend pre_run hook creates symlink |
This means Drill can test draft skill changes by pointing at a branch checkout of superpowers.
## Post-Session Tool Call Collection
Both backends write structured session logs that record every tool invocation:
| Backend | Log location | Format |
|---------|-------------|--------|
| Claude Code | `~/.claude/projects/**/session-*.jsonl` | JSONL with tool names + args |
| Codex | `~/.codex/sessions/rollout-*.jsonl` | JSONL with `LocalShellCall`, `FunctionCall`, etc. |
The harness snapshots each backend's log directory before the session starts. After shutdown, it diffs the directory to find only files created during the run — no timestamp matching needed, no cross-contamination from concurrent sessions or prior runs.
Collected logs are normalized into a common `tool_calls.jsonl` format before the verifier sees them:
```json
{"tool": "EnterWorktree", "args": {"branch": "add-login"}, "source": "native"}
{"tool": "Bash", "args": {"command": "git worktree add ..."}, "source": "shell"}
```
Each backend defines a normalizer function that maps its native log format (Claude Code's tool call entries, Codex's `ResponseItem` records) into this common schema. The verifier never sees raw backend-specific logs.
## Actor & Verifier LLM Design
### Actor
- **Model:** Sonnet
- **Temperature:** 0.7 (realistic user variation)
- **Context:** Rolling (full conversation history). Sessions are short enough (~5-20 turns) that token cost is not a concern.
- **Input:** System prompt + rolling terminal captures + all intents + user_posture
- **Output:** Structured JSON via Anthropic SDK tool_use: `{"action": "type", "text": "..."}`, `{"action": "done"}`, `{"action": "stuck"}`, or `{"action": "key", "key": "ctrl-c"}`. The harness parses this and sends keystrokes — no free-text sanitization needed.
- **Prompt:** Versioned template at `prompts/actor.md`
### Verifier
- **Model:** Sonnet
- **Temperature:** Near-zero (deterministic judgment)
- **Input:** session.log + filesystem.json + tool_calls.jsonl + criteria list. Does NOT receive actor intents or scenario narrative (reduces confirmation bias).
- **Output:** Structured JSON with per-criterion verdict/evidence/rationale + observations
- **Prompt:** Versioned template at `prompts/verifier.md`
## Results & Compare
### Results Structure
```
results/
<scenario>/
<backend>/
<timestamp>/
session.log # raw tmux capture
filesystem.json # post-run git/file state snapshot
tool_calls.jsonl # collected from backend session logs
verdict.json # verifier output
meta.json # run metadata (backend, duration, turns, model versions)
```
### Compare Command
`drill compare` reads existing results from prior `drill run` invocations. It does not run backends itself — run each backend separately first, then compare.
```
$ drill run worktree-creation-from-main --backend claude
$ drill run worktree-creation-from-main --backend codex
$ drill compare worktree-creation-from-main
Scenario: worktree-creation-from-main (naive posture)
Summary:
┌──────────┬────────┬───────┬───────┐
│ Backend │ Result │ Score │ Turns │
├──────────┼────────┼───────┼───────┤
│ claude │ PASS │ 4/4 │ 6 │
│ codex │ FAIL │ 2/4 │ 12 │
└──────────┴────────┴───────┴───────┘
Detail:
┌────────────────────────────────┬────────┬────────┐
│ Criterion │ claude │ codex │
├────────────────────────────────┼────────┼────────┤
│ Detected on main │ ✓ │ ✓ │
│ Asked consent │ ✓ │ ✗ │
│ Worktree exists │ ✓ │ ✓ │
│ Used native tools │ ✓ │ ✗ │
└────────────────────────────────┴────────┴────────┘
Observations:
claude: "Agent cited the using-git-worktrees skill by name"
codex: "Agent created worktree but skipped consent step entirely"
```
## Project Structure
```
drill/
├── drill/
│ ├── __init__.py
│ ├── cli.py # click CLI: run, compare, list
│ ├── engine.py # orchestrates the full run lifecycle
│ ├── session.py # tmux session management
│ ├── actor.py # actor LLM calls
│ ├── verifier.py # verifier LLM calls
│ ├── setup.py # template repo cloning, helpers, assertions
│ └── backend.py # loads backend YAML, builds commands
├── backends/
│ ├── claude.yaml
│ └── codex.yaml
├── prompts/
│ ├── actor.md
│ └── verifier.md
├── scenarios/
│ ├── worktree-creation-from-main.yaml
│ ├── worktree-already-inside.yaml
│ ├── worktree-codex-detached-head.yaml
│ └── worktree-consent-flow.yaml
├── fixtures/
│ └── template-repo/ # base git repo, cloned per run
├── setup_helpers/
│ ├── __init__.py
│ ├── base.py # create_base_repo, common git ops
│ └── worktree.py # add_worktree, detach_head, etc.
├── results/ # gitignored, populated by runs
├── pyproject.toml # package metadata + [project.scripts] entry point
└── README.md
```
## Phase 1 Scope
- Claude Code + Codex backends
- 4 PRI-974 worktree scenarios (creation, already-inside, detached-head, consent)
- Both user postures (naive + spec-aware) per scenario
- Template repo + setup helpers + assertions
- Actor + verifier with prompts
- `drill run` and `drill compare` commands
- Results storage
## Phase 2 (Future)
- Gemini CLI backend
- Backend variants (e.g., `codex-workspace-write.yaml` for sandbox mode testing)
- Verifier flakiness mitigation (3x voting, agreement tracking)
- Cost tracking and token usage reporting
- Docker isolation for reproducibility
- CI integration
- Scenarios beyond worktrees (stacked PRs, git-spice, brainstorming)
## Installation
```bash
pip install -e . # installs 'drill' console script
```
Requires `tmux` installed as a system dependency.
## Dependencies
- Python 3.11+
- `click` — CLI framework
- `pyyaml` — scenario and backend config parsing
- `anthropic` — Anthropic Python SDK for actor/verifier LLM calls (structured tool_use output)
- `jinja2` — prompt template rendering
- `pydantic` — verdict schema validation (retry on malformed verifier output)
- `tmux` — session driving (system dependency)
## Non-Goals
- Not a coding ability benchmark (SWE-bench covers that)
- Not an LLM evaluation framework (promptfoo covers that)
- Not a generic terminal automation tool (Terminal-Bench covers that)
- No CI in phase 1
- No Docker in phase 1

View File

@@ -1,93 +0,0 @@
# Manual Testing (Codex App)
Some scenarios cannot run automatically because drill has no harness adapter for the target — the Codex App desktop client has no CLI or tmux entry point the way `claude` and `codex` do. These scenarios are marked `manual: true` in their YAML and use a human-in-the-loop protocol.
## Protocol
Three phases. The agent never runs Codex App directly. The tester never writes a verdict by hand.
1. **Agent prepares the handoff** — reads the scenario file, renders setup + turn intents into something a human can act on, hands the package to the tester.
2. **Tester executes** — sets up the repo fixture, opens Codex App, pastes the prompt, handles any follow-ups, copies the transcript + final filesystem state back to the agent.
3. **Agent judges and records** — evaluates the transcript against `verify.criteria`, writes a verdict JSON, saves to `results/<scenario>/codex-app/YYYY-MM-DD-manual/verdict.json`.
## Phase 1: Agent prepares the handoff
Deliver as one self-contained message to the tester:
### Fixture state
Exact repo state Codex App should be launched against. Pull from `setup.notes` if present, otherwise translate `setup.helpers` + `setup.assertions` into prose. Include: which repo/directory, branch, whether to expect a worktree vs normal checkout, any required/forbidden files (e.g. `.gitignore` entries).
### Prompt to paste
Render turn 1's `intent` as a natural first-person message the tester can paste verbatim into Codex App. **Don't leak internal test language** like *"Do NOT say 'create a worktree'"* — that's instruction for the test author, not the end user. Convert it to what a real user would actually type.
Example:
> Intent: *"Ask the agent to use the worktree skill to get set up for a notifications feature. Do NOT say 'create a worktree' — just reference the skill by name."*
>
> Rendered prompt: *"hey, can you use the worktree skill to get me set up for a notifications feature?"*
### Follow-up guidance
For each additional turn, give the tester a short decision rule — not a verbatim script. E.g. *"If the agent asks a clarifying question like branch name, answer concisely. If it stops to ask whether you want a worktree at all, tell it you already asked for the skill and it should proceed."*
### What to capture
Ask the tester to paste back:
- Full agent transcript (messages, tool calls, tool outputs)
- Final filesystem state if criteria depend on it (`git worktree list`, directory tree, branch state)
- Any observations they want on the record
## Phase 2: Tester executes
1. Set up the repo fixture per the instructions
2. Open Codex App in that repo
3. Paste the prompt
4. Follow up per the guidance
5. Copy the transcript + filesystem state back to the agent
## Phase 3: Agent judges and records
For each criterion in `verify.criteria`, write one entry:
```json
{
"criterion": "<verbatim from scenario>",
"passed": true | false,
"evidence": "<quoted snippet from transcript>",
"rationale": "<only if passed is inconclusive or needs context>"
}
```
**Rules:**
- Quote the transcript directly in `evidence`. No paraphrasing.
- If a criterion is genuinely inconclusive from the transcript, mark `passed: false` with `rationale` explaining what was missing. Don't guess.
- Don't grade on intent you can't see. The agent's internal thoughts aren't visible — only messages, tool calls, and results.
### Verdict file
Save to `results/<scenario>/codex-app/YYYY-MM-DD-manual/verdict.json`:
```json
{
"scenario": "<scenario-name>",
"backend": "codex-app",
"manual": true,
"user_posture": "<spec-aware|naive|...>",
"passed": <true iff every criterion.passed is true>,
"criteria": [ ... ],
"notes": "<optional: cross-criterion observations>"
}
```
Matches the format of the existing `results/worktree-codex-app-detached-head/codex-app/2026-04-09-manual/verdict.json`.
## When to invoke
- A scenario's YAML has `manual: true`
- The tester explicitly asks for a manual Codex App run of any scenario
- An automated test result is inconclusive and we want a human-verified cross-check
Do NOT use this procedure for scenarios drill can run itself (`claude`, `codex`, `gemini` backends) — use `drill run` instead.
## Pitfalls
- **Don't skip the fixture step.** Codex App's default environment (detached HEAD under `$CODEX_HOME/worktrees/`) is load-bearing for worktree scenarios. The same prompt gives different results in a normal checkout.
- **Don't render prompts literally.** Scenario intents are written for test authors; they often contain "Do NOT mention X" style instructions. Translate before handing to the tester.
- **Don't grade on missing evidence.** If the transcript doesn't show the agent doing something the criterion asks about, that's a fail, not a pass-by-default.

File diff suppressed because it is too large Load Diff

View File

@@ -1,89 +0,0 @@
# Pressure / RED phase testing in drill
## What "RED phase" means
The bash test family in superpowers/tests/ used three implicit phases
when stress-testing skill content:
* **GREEN** — current skill text. Baseline behavior under normal user
prompts. This is what most drill scenarios exercise.
* **PRESSURE** — current skill text, but the user prompt creates
conditions that make the skill's recommended path inconvenient
(urgency, an "easier" alternative already on disk, etc.). Lifted
as `worktree-creation-under-pressure.yaml`.
* **RED** — *modified* skill text where the section under test has
been removed or weakened. Used to confirm a passing GREEN/PRESSURE
result actually depended on the skill text and isn't just baseline
model behavior.
GREEN and PRESSURE both run against the current `SUPERPOWERS_ROOT`.
RED needs a *different* superpowers checkout — one with the section
under test stripped out — and runs the same scenario against that.
## The drill primitive: vary `SUPERPOWERS_ROOT`
Every backend YAML interpolates `${SUPERPOWERS_ROOT}` into its
`--plugin-dir` arg (claude.yaml line 6, gemini.yaml line 5, etc.).
That env var is the only knob you need: point drill at a different
plugin checkout and the agent under test loads a different version
of the skill.
```bash
# GREEN: current skill text
drill run worktree-creation-from-main -b claude
# RED: same scenario, against a checkout where Step 1a is deleted
SUPERPOWERS_ROOT=/path/to/superpowers-without-step-1a \
drill run worktree-creation-from-main -b claude
```
Compare verdicts. If GREEN passes and RED fails, the skill text is
load-bearing. If both pass, the model produces the right behavior
without the skill — meaning either the skill is redundant or the
test isn't probing what it claims to probe.
## Recommended workflow
1. Make a git worktree of superpowers at the commit/branch you want
to test. For RED variants, edit the skill in that worktree to
remove the section under test.
```bash
cd ~/Documents/GitHub/superpowers/superpowers
git worktree add ../superpowers-red-no-step-1a HEAD
# edit skills/using-git-worktrees/SKILL.md in the worktree
```
2. Run the same drill scenario against each variant. Use
`--n N` to get statistical signal — single runs are noisy,
especially under pressure conditions.
```bash
for variant in main red-no-step-1a; do
SUPERPOWERS_ROOT=~/Documents/GitHub/superpowers/superpowers-${variant#main}superpowers \
drill run worktree-creation-from-main -b claude --n 10
done
```
3. Compare with `drill compare`. Look for the RED variant's pass
rate dropping (skill is load-bearing) or holding (skill is
redundant or scenario isn't probing what it claims).
## When to add a new pressure scenario vs. add a turn variation
* **New scenario** when the *filesystem* setup is different (e.g.,
pre-existing `.worktrees/` for the worktree-pressure case).
Setup helpers are scenario-scoped.
* **New `--n` sweep with different prompts** when only the
*user prompt* shape varies (e.g., urgency, framing).
Drill doesn't yet have a way to vary turn intents within a single
scenario YAML — multi-prompt sweeps require multiple scenario files
or running the same scenario with different intents externally.
## Open follow-ups
* `--plugins=A,B,C` sweep dimension (parallel to `--models`) so a
single drill invocation can run RED + GREEN + PRESSURE variants
in one batch and `drill compare` shows them side-by-side. Not yet
implemented; tracked as drill-internal future work.

View File

@@ -1,3 +0,0 @@
"""Drill: Superpowers skill compliance benchmark."""
__version__: str = "0.1.0"

View File

@@ -1,5 +0,0 @@
"""Allow running drill as `python3 -m drill`."""
from drill.cli import main
main()

View File

@@ -1,81 +0,0 @@
"""Actor LLM: simulates a user driving an agent session."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import anthropic
from jinja2 import Template
ACTOR_TOOL: dict[str, Any] = {
"name": "terminal_action",
"description": "Send an action to the terminal session.",
"input_schema": {
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["type", "done", "stuck", "key"],
"description": "The action to take.",
},
"text": {
"type": "string",
"description": "Text to type (only for 'type' action).",
},
"key": {
"type": "string",
"description": "Special key to send (only for 'key' action, e.g., 'ctrl-c').",
},
},
"required": ["action"],
},
}
@dataclass
class ActorAction:
action: str
text: str | None = None
key: str | None = None
@classmethod
def from_tool_result(cls, data: dict[str, Any]) -> ActorAction:
return cls(action=data["action"], text=data.get("text"), key=data.get("key"))
class Actor:
def __init__(self, model: str = "claude-sonnet-4-6", temperature: float = 0.7) -> None:
self.model = model
self.temperature = temperature
self.captures: list[str] = []
self._system_prompt: str = ""
self._client: anthropic.Anthropic = anthropic.Anthropic()
def build_system_prompt(self, posture: str, intents: list[str]) -> str:
template_path = Path(__file__).parent.parent / "prompts" / "actor.md"
template = Template(template_path.read_text())
self._system_prompt = template.render(posture=posture, intents=intents)
return self._system_prompt
def append_capture(self, terminal_output: str) -> None:
self.captures.append(terminal_output)
def build_messages(self) -> list[dict[str, str]]:
return [{"role": "user", "content": capture} for capture in self.captures]
def decide(self) -> ActorAction:
response = self._client.messages.create(
model=self.model,
max_tokens=1024,
temperature=self.temperature,
system=self._system_prompt,
tools=[ACTOR_TOOL], # ty: ignore[invalid-argument-type]
tool_choice={"type": "tool", "name": "terminal_action"},
messages=self.build_messages(), # ty: ignore[invalid-argument-type]
)
for block in response.content:
if block.type == "tool_use":
return ActorAction.from_tool_result(block.input)
raise RuntimeError("Actor did not return a tool_use block")

View File

@@ -1,89 +0,0 @@
"""Post-session deterministic assertions for drill scenarios."""
from __future__ import annotations
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from drill.verifier import CriterionResult
@dataclass
class AssertionResult:
command: str
passed: bool
exit_code: int
stdout: str
stderr: str
def to_criterion_result(self) -> CriterionResult:
evidence = f"exit code {self.exit_code}"
if self.stdout:
evidence += f"\nstdout: {self.stdout}"
if self.stderr:
evidence += f"\nstderr: {self.stderr}"
return CriterionResult(
criterion=f"[assertion] {self.command}",
verdict="pass" if self.passed else "fail",
evidence=evidence,
rationale="Deterministic assertion " + ("passed" if self.passed else "failed"),
source="assertion",
)
def run_verify_assertions(
assertions: list[str],
results_dir: Path,
workdir: Path,
*,
timeout_seconds: int = 10,
) -> list[AssertionResult]:
bin_dir = Path(__file__).parent.parent / "bin"
env = {
**os.environ,
"DRILL_WORKDIR": str(workdir),
"PATH": f"{bin_dir}:{os.environ.get('PATH', '')}",
}
results: list[AssertionResult] = []
for cmd in assertions:
try:
proc = subprocess.run(
["bash", "-c", cmd],
cwd=results_dir,
capture_output=True,
text=True,
env=env,
timeout=timeout_seconds,
)
results.append(
AssertionResult(
command=cmd,
passed=proc.returncode == 0,
exit_code=proc.returncode,
stdout=proc.stdout.strip(),
stderr=proc.stderr.strip(),
)
)
except subprocess.TimeoutExpired:
results.append(
AssertionResult(
command=cmd,
passed=False,
exit_code=124,
stdout="",
stderr=f"Timed out after {timeout_seconds}s",
)
)
except Exception as e:
results.append(
AssertionResult(
command=cmd,
passed=False,
exit_code=-1,
stdout="",
stderr=str(e),
)
)
return results

View File

@@ -1,111 +0,0 @@
"""Backend config loader and command builder."""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import yaml
@dataclass
class Backend:
name: str
cli: str
args: list[str]
required_env: list[str]
hooks: dict[str, list[str]]
shutdown: str
idle: dict[str, Any]
startup_timeout: int
terminal: dict[str, int]
session_logs: dict[str, str]
turn_timeout: int | None = None
busy_pattern: str = ""
max_busy_seconds: int = 1800
def build_command(self, workdir: str) -> list[str]:
resolved = [_interpolate_env(arg) for arg in self.args]
return [self.cli, *resolved]
def validate_env(self) -> None:
missing = [v for v in self.required_env if not os.environ.get(v)]
if missing:
raise OSError(
f"Missing required environment variables for {self.name} backend: "
+ ", ".join(missing)
)
def is_ready_line(self, line: str) -> bool:
pattern = self.idle.get("ready_pattern", "")
return bool(re.search(pattern, line))
def is_busy_line(self, line: str) -> bool:
if not self.busy_pattern:
return False
return bool(re.search(self.busy_pattern, line))
@property
def quiescence_seconds(self) -> float:
return self.idle.get("quiescence_seconds", 5)
@property
def cols(self) -> int:
return self.terminal.get("cols", 200)
@property
def rows(self) -> int:
return self.terminal.get("rows", 50)
@property
def model(self) -> str | None:
"""Model name from args (looks for --model or -m flag)."""
for i, arg in enumerate(self.args):
if arg in ("--model", "-m") and i + 1 < len(self.args):
return self.args[i + 1]
return None
@property
def family(self) -> str:
"""Normalize backend name to a family for log-dir / normalizer dispatch."""
for fam in ("claude", "codex", "gemini"):
if self.name == fam or self.name.startswith(f"{fam}-"):
return fam
return "other"
def load_backend(name: str, backends_dir: Path) -> Backend:
path = backends_dir / f"{name}.yaml"
if not path.exists():
raise FileNotFoundError(f"Backend config not found: {path}")
with open(path) as f:
data = yaml.safe_load(f)
return Backend(
name=data["name"],
cli=data["cli"],
args=data.get("args", []),
required_env=data.get("required_env", []),
hooks=data.get("hooks", {"pre_run": [], "post_run": []}),
shutdown=data.get("shutdown", "/exit"),
idle=data.get("idle", {}),
startup_timeout=data.get("startup_timeout", 30),
terminal=data.get("terminal", {"cols": 200, "rows": 50}),
session_logs=data.get("session_logs", {}),
turn_timeout=data.get("turn_timeout"),
busy_pattern=data.get("busy_pattern", ""),
max_busy_seconds=data.get("max_busy_seconds", 1800),
)
def _interpolate_env(value: str) -> str:
def replacer(match: re.Match[str]) -> str:
var = match.group(1)
val = os.environ.get(var)
if val is None:
raise OSError(f"Environment variable {var} not set")
return val
return re.sub(r"\$\{(\w+)\}", replacer, value)

View File

@@ -1,154 +0,0 @@
"""Drill CLI: run, compare, list."""
from __future__ import annotations
import os
import secrets
from pathlib import Path
import click
from dotenv import load_dotenv
PROJECT_ROOT: Path = Path(__file__).parent.parent
load_dotenv(PROJECT_ROOT / ".env")
def _set_superpowers_root_default() -> None:
"""Default SUPERPOWERS_ROOT to the parent of evals/ if not already set.
Drill historically required contributors to export SUPERPOWERS_ROOT
pointing at the superpowers checkout. After lifting drill into
superpowers/evals/, the parent of PROJECT_ROOT is always the
superpowers root, so we can supply this default automatically.
Existing SUPERPOWERS_ROOT environment values are respected as overrides.
"""
os.environ.setdefault("SUPERPOWERS_ROOT", str(PROJECT_ROOT.parent))
_set_superpowers_root_default()
@click.group()
def main() -> None:
"""Drill: Superpowers skill compliance benchmark."""
pass
@main.command()
@click.argument("scenario")
@click.option("--backend", "-b", default=None, help="Backend name (e.g., claude, codex)")
@click.option("--models", "-m", default=None, help="Comma-separated backend names for sweep")
@click.option("--n", "n_runs", type=int, default=1, help="Number of repetitions per backend")
@click.option(
"--backends-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "backends",
)
@click.option(
"--scenarios-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "scenarios",
)
@click.option(
"--fixtures-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "fixtures",
)
@click.option("--results-dir", type=click.Path(path_type=Path), default=PROJECT_ROOT / "results")
def run(
scenario: str,
backend: str | None,
models: str | None,
n_runs: int,
backends_dir: Path,
scenarios_dir: Path,
fixtures_dir: Path,
results_dir: Path,
) -> None:
"""Run a scenario against one or more backends."""
if n_runs < 1:
raise click.ClickException("--n must be at least 1")
if models:
backend_names = [b.strip() for b in models.split(",") if b.strip()]
elif backend:
backend_names = [backend]
else:
raise click.ClickException("Either --backend or --models is required")
scenario_path = scenarios_dir / f"{scenario}.yaml"
if not scenario_path.exists():
raise click.ClickException(f"Scenario not found: {scenario_path}")
sweep_id = secrets.token_hex(4)
from drill.sweep import Sweep
sweep = Sweep(
scenario_path=scenario_path,
backend_names=backend_names,
backends_dir=backends_dir,
fixtures_dir=fixtures_dir,
results_dir=results_dir,
n=n_runs,
sweep_id=sweep_id,
)
total = len(backend_names) * n_runs
click.echo(
f"Running {scenario} | backends: {', '.join(backend_names)} | "
f"n={n_runs} | total runs: {total} | sweep: {sweep_id}"
)
groups = sweep.run_all()
for group in groups:
passed = sum(1 for r in group.runs if r.status == "pass")
failed = sum(1 for r in group.runs if r.status == "fail")
errored = sum(1 for r in group.runs if r.status == "error")
click.echo(f"\n{group.backend}: {passed} passed, {failed} failed, {errored} errors")
if group.partial:
click.echo(" (interrupted — partial results)")
@main.command("list")
@click.option(
"--scenarios-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "scenarios",
)
def list_scenarios(scenarios_dir: Path) -> None:
"""List available scenarios."""
import yaml
for f in sorted(scenarios_dir.glob("*.yaml")):
with open(f) as fh:
data = yaml.safe_load(fh)
name = data.get("scenario", f.stem)
desc = data.get("description", "")
click.echo(f" {name:40s} {desc}")
@main.command()
@click.argument("scenario")
@click.option("--sweep", "sweep_id", default=None, help="Filter by sweep ID")
@click.option(
"--results-dir",
type=click.Path(exists=True, path_type=Path),
default=PROJECT_ROOT / "results",
)
def compare(scenario: str, sweep_id: str | None, results_dir: Path) -> None:
"""Compare results across backends for a scenario."""
from drill.compare import format_compare_output, load_scenario_results
scenario_dir = results_dir / scenario
if not scenario_dir.exists():
raise click.ClickException(f"No results found for: {scenario}")
results = load_scenario_results(scenario_dir, sweep_id=sweep_id)
if not results:
raise click.ClickException(f"No results found for: {scenario}")
click.echo(format_compare_output(scenario, results))

View File

@@ -1,255 +0,0 @@
"""Compare: load and aggregate drill results across backends and runs."""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from drill.stats import wilson_ci
from drill.verifier import Verdict
@dataclass
class BackendResult:
backend: str
total_runs: int
passed_runs: int
errored_runs: int
avg_turns: float
criterion_counts: dict[str, tuple[int, int]] # criterion -> (passed, total)
sweep_id: str | None
timestamp: str | None
partial: bool
@property
def pass_rate(self) -> float:
if self.total_runs == 0:
return 0.0
return self.passed_runs / self.total_runs
def load_scenario_results(
scenario_dir: Path,
*,
sweep_id: str | None = None,
) -> dict[str, BackendResult]:
results: dict[str, BackendResult] = {}
for backend_dir in sorted(scenario_dir.iterdir()):
if not backend_dir.is_dir():
continue
timestamp_dirs = sorted(backend_dir.iterdir())
if not timestamp_dirs:
continue
target_dir: Path | None = None
if sweep_id:
for d in timestamp_dirs:
rg_path = d / "run-group.json"
if rg_path.exists():
rg = json.loads(rg_path.read_text())
if rg.get("sweep_id") == sweep_id:
target_dir = d
break
else:
target_dir = timestamp_dirs[-1]
if target_dir is None:
continue
result = _load_backend_result(backend_dir.name, target_dir)
if result is not None:
results[backend_dir.name] = result
return results
def _load_backend_result(backend_name: str, timestamp_dir: Path) -> BackendResult | None:
rg_path = timestamp_dir / "run-group.json"
if rg_path.exists():
return _load_new_format(backend_name, timestamp_dir, rg_path)
elif (timestamp_dir / "verdict.json").exists():
return _load_old_format(backend_name, timestamp_dir)
return None
def _load_new_format(backend_name: str, timestamp_dir: Path, rg_path: Path) -> BackendResult:
rg: dict[str, Any] = json.loads(rg_path.read_text())
run_dirs = sorted(
d for d in timestamp_dir.iterdir() if d.is_dir() and d.name.startswith("run-")
)
verdicts: list[Verdict] = []
metas: list[dict[str, Any]] = []
for run_dir in run_dirs:
verdict_path = run_dir / "verdict.json"
meta_path = run_dir / "meta.json"
if verdict_path.exists():
verdicts.append(Verdict.model_validate_json(verdict_path.read_text()))
if meta_path.exists():
metas.append(json.loads(meta_path.read_text()))
passed_runs = sum(1 for v in verdicts if v.passed)
errored_runs = sum(1 for r in rg.get("runs", []) if r.get("status") == "error")
avg_turns = sum(m.get("actor_turns", 0) for m in metas) / len(metas) if metas else 0.0
criterion_counts: dict[str, tuple[int, int]] = {}
for v in verdicts:
for c in v.criteria:
prev_passed, prev_total = criterion_counts.get(c.criterion, (0, 0))
criterion_counts[c.criterion] = (
prev_passed + (1 if c.verdict == "pass" else 0),
prev_total + 1,
)
return BackendResult(
backend=backend_name,
total_runs=len(verdicts),
passed_runs=passed_runs,
errored_runs=errored_runs,
avg_turns=round(avg_turns, 1),
criterion_counts=criterion_counts,
sweep_id=rg.get("sweep_id"),
timestamp=rg.get("timestamp"),
partial=rg.get("partial", False),
)
def _load_old_format(backend_name: str, timestamp_dir: Path) -> BackendResult:
verdict = Verdict.model_validate_json((timestamp_dir / "verdict.json").read_text())
meta: dict[str, Any] = {}
meta_path = timestamp_dir / "meta.json"
if meta_path.exists():
meta = json.loads(meta_path.read_text())
criterion_counts: dict[str, tuple[int, int]] = {}
for c in verdict.criteria:
criterion_counts[c.criterion] = (1 if c.verdict == "pass" else 0, 1)
return BackendResult(
backend=backend_name,
total_runs=1,
passed_runs=1 if verdict.passed else 0,
errored_runs=0,
avg_turns=float(meta.get("actor_turns", 0)),
criterion_counts=criterion_counts,
sweep_id=None,
timestamp=None,
partial=False,
)
def format_compare_output(
scenario: str,
results: dict[str, BackendResult],
) -> str:
if not results:
return f"No results found for: {scenario}"
lines: list[str] = []
is_multi_run = any(r.total_runs > 1 for r in results.values())
if is_multi_run:
first = next(iter(results.values()))
lines.append(f"Scenario: {scenario}")
if first.sweep_id:
sweep_label = f"Sweep: {first.sweep_id}"
if first.timestamp:
date_str = first.timestamp.split("T")[0]
sweep_label += f" | {date_str}"
lines.append(sweep_label)
lines.append("")
header = f"{'':40s}"
sub_header = f"{'':40s}"
for name, r in results.items():
header += f" {name:>12s}"
sub_header += f" {'(n=' + str(r.total_runs) + ')':>12s}"
lines.append(header)
lines.append(sub_header)
lines.append("-" * len(header))
rate_line = f"{'Overall pass rate':40s}"
ci_line = f"{' 95% CI':40s}"
for r in results.values():
pct = f"{r.pass_rate * 100:.1f}%"
rate_line += f" {pct:>12s}"
lo, hi = wilson_ci(r.passed_runs, r.total_runs)
ci_str = f"[{lo * 100:.0f}, {hi * 100:.0f}]"
ci_line += f" {ci_str:>12s}"
lines.append(rate_line)
lines.append(ci_line)
lines.append("")
all_criteria: list[str] = []
seen: set[str] = set()
for r in results.values():
for crit in r.criterion_counts:
if crit not in seen:
all_criteria.append(crit)
seen.add(crit)
for crit in all_criteria:
crit_line = f"{crit[:40]:40s}"
for r in results.values():
passed, total = r.criterion_counts.get(crit, (0, 0))
crit_line += f" {str(passed) + '/' + str(total):>12s}"
lines.append(crit_line)
lines.append("")
avg_line = f"{'Avg turns':40s}"
err_line = f"{'Errors':40s}"
for r in results.values():
avg_line += f" {str(r.avg_turns):>12s}"
err_line += f" {str(r.errored_runs):>12s}"
lines.append(avg_line)
lines.append(err_line)
if any(r.total_runs < 10 for r in results.values()):
lines.append("")
lines.append("Note: CI is wide due to small sample size; consider --n 10+")
if any(r.partial for r in results.values()):
lines.append("")
lines.append("Warning: Sweep was interrupted — results are incomplete.")
else:
lines.append(f"Scenario: {scenario}")
lines.append("")
lines.append(f"{'Backend':20s} {'Result':8s} {'Score':7s} {'Turns':5s}")
lines.append("-" * 42)
for name, r in results.items():
result_str = "PASS" if r.passed_runs == r.total_runs else "FAIL"
total_criteria = sum(t for _, t in r.criterion_counts.values())
passed_criteria = sum(p for p, _ in r.criterion_counts.values())
score = f"{passed_criteria}/{total_criteria}"
turns_str = (
str(int(r.avg_turns)) if r.avg_turns == int(r.avg_turns) else str(r.avg_turns)
)
lines.append(f"{name:20s} {result_str:8s} {score:7s} {turns_str:5s}")
all_criteria = []
seen = set()
for r in results.values():
for crit in r.criterion_counts:
if crit not in seen:
all_criteria.append(crit)
seen.add(crit)
lines.append("")
header = f"{'':40s}"
for name in results:
header += f" {name:>12s}"
lines.append(header)
lines.append("-" * len(header))
for crit in all_criteria:
crit_line = f"{crit[:40]:40s}"
for r in results.values():
p, t = r.criterion_counts.get(crit, (0, 0))
icon = "PASS" if p == t and t > 0 else "FAIL"
crit_line += f" {icon:>12s}"
lines.append(crit_line)
return "\n".join(lines)

View File

@@ -1,377 +0,0 @@
"""Engine: orchestrates the full Drill run lifecycle."""
from __future__ import annotations
import json
import os
import re
import subprocess
import time
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any
import yaml
from drill.actor import Actor
from drill.assertions import AssertionResult, run_verify_assertions
from drill.backend import load_backend
from drill.normalizer import (
NORMALIZERS,
collect_new_logs,
filter_codex_logs_by_cwd,
snapshot_log_dir,
)
from drill.session import TmuxSession
from drill.setup import run_assertions, run_helpers
from drill.verifier import Verifier
@dataclass
class VerifyConfig:
criteria: list[str] = field(default_factory=list)
assertions: list[str] = field(default_factory=list)
observe: bool = False
@dataclass
class ScenarioConfig:
scenario: str
description: str
user_posture: str
setup: dict[str, Any]
turns: list[dict[str, Any]]
limits: dict[str, Any]
verify: VerifyConfig
@classmethod
def from_yaml(cls, path: Path) -> ScenarioConfig:
with open(path) as f:
data = yaml.safe_load(f)
verify_data = data.get("verify", {})
return cls(
scenario=data["scenario"],
description=data.get("description", ""),
user_posture=data.get("user_posture", "naive"),
setup=data.get("setup", {}),
turns=data.get("turns", []),
limits=data.get("limits", {"max_turns": 20, "turn_timeout": 120}),
verify=VerifyConfig(
criteria=verify_data.get("criteria", []),
assertions=verify_data.get("assertions", []),
observe=verify_data.get("observe", False),
),
)
@dataclass
class RunResult:
scenario: str
backend: str
timestamp: str
session_log: str
filesystem_json: str
tool_calls_jsonl: str
verdict_json: str
meta: dict[str, Any]
def save_artifacts(self, output_dir: Path) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "session.log").write_text(self.session_log)
(output_dir / "filesystem.json").write_text(self.filesystem_json)
(output_dir / "tool_calls.jsonl").write_text(self.tool_calls_jsonl)
def save_verdict(self, output_dir: Path) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "verdict.json").write_text(self.verdict_json)
(output_dir / "meta.json").write_text(json.dumps(self.meta, indent=2))
def save(self, output_dir: Path) -> None:
self.save_artifacts(output_dir)
self.save_verdict(output_dir)
def snapshot_filesystem(workdir: Path) -> str:
files: list[str] = []
for f in sorted(workdir.rglob("*")):
if ".git" in f.parts:
continue
if f.is_file():
files.append(str(f.relative_to(workdir)))
git_status = _git_cmd(workdir, ["git", "status", "--short"])
branch = _git_cmd(workdir, ["git", "branch", "--show-current"])
worktree_list = _git_cmd(workdir, ["git", "worktree", "list"])
return json.dumps(
{
"files": files,
"git_status": git_status,
"branch": branch,
"worktree_list": worktree_list,
},
indent=2,
)
class Engine:
def __init__(
self,
scenario_path: Path,
backend_name: str,
backends_dir: Path,
fixtures_dir: Path,
results_dir: Path,
) -> None:
self.scenario = ScenarioConfig.from_yaml(scenario_path)
self.backend = load_backend(backend_name, backends_dir)
self.fixtures_dir = fixtures_dir
self.results_dir = results_dir
def run(self, *, output_dir: Path | None = None, run_suffix: str = "") -> RunResult:
start_time = time.time()
timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
self.backend.validate_env()
workdir = Path(f"/tmp/drill-{self.scenario.scenario}-{timestamp}{run_suffix}")
self._setup(workdir)
actual_workdir = workdir
override = self.scenario.setup.get("workdir_override")
if override:
resolved = override.replace("${WORKDIR_NAME}", workdir.name)
actual_workdir = (workdir / resolved).resolve()
# Run assertions in the actual workdir (after override)
assertions = self.scenario.setup.get("assertions", [])
if assertions:
run_assertions(assertions, actual_workdir)
session_name = f"drill-{self.scenario.scenario}-{timestamp}{run_suffix}"
session = TmuxSession(name=session_name, cols=self.backend.cols, rows=self.backend.rows)
log_dir = self._resolve_log_dir(actual_workdir)
log_snapshot = snapshot_log_dir(log_dir) if log_dir else set()
session_log, actor_turns = self._run_session(session, actual_workdir)
filesystem_json = snapshot_filesystem(actual_workdir)
tool_calls = self._collect_tool_calls(log_dir, log_snapshot, actual_workdir)
tool_calls_jsonl = "\n".join(json.dumps(tc) for tc in tool_calls)
# Write artifacts to disk before assertions (assertions read from disk)
if output_dir is None:
output_dir = self.results_dir / self.scenario.scenario / self.backend.name / timestamp
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "session.log").write_text(session_log)
(output_dir / "filesystem.json").write_text(filesystem_json)
(output_dir / "tool_calls.jsonl").write_text(tool_calls_jsonl)
# Run deterministic assertions
assertion_results: list[AssertionResult] = []
if self.scenario.verify.assertions:
if not tool_calls_jsonl.strip():
assertion_results = [
AssertionResult(
command="<pre-check>",
passed=False,
exit_code=1,
stdout="",
stderr="tool_calls.jsonl is empty — session may have crashed",
)
]
else:
assertion_results = run_verify_assertions(
self.scenario.verify.assertions,
output_dir,
actual_workdir,
)
# Run LLM verifier
verifier = Verifier()
verdict = verifier.verify(
session_log=session_log,
filesystem_json=filesystem_json,
tool_calls_jsonl=tool_calls_jsonl,
criteria=self.scenario.verify.criteria,
)
# Merge assertion results into verdict
for ar in assertion_results:
verdict.criteria.append(ar.to_criterion_result())
duration = time.time() - start_time
meta: dict[str, Any] = {
"scenario": self.scenario.scenario,
"backend": self.backend.name,
"backend_model": self.backend.model,
"user_posture": self.scenario.user_posture,
"timestamp": timestamp,
"duration_seconds": round(duration, 1),
"actor_turns": actor_turns,
"actor_model": "claude-sonnet-4-6",
"verifier_model": "claude-sonnet-4-6",
}
result = RunResult(
scenario=self.scenario.scenario,
backend=self.backend.name,
timestamp=timestamp,
session_log=session_log,
filesystem_json=filesystem_json,
tool_calls_jsonl=tool_calls_jsonl,
verdict_json=verdict.model_dump_json(indent=2),
meta=meta,
)
# Write verdict + meta (artifacts already on disk)
(output_dir / "verdict.json").write_text(result.verdict_json)
(output_dir / "meta.json").write_text(json.dumps(result.meta, indent=2))
return result
def _setup(self, workdir: Path) -> None:
# Scenario helpers first (create_base_repo needs to run before anything else)
helpers = self.scenario.setup.get("helpers", [])
run_helpers(helpers, workdir, self.fixtures_dir)
# Backend pre_run hooks after (e.g., codex symlink needs workdir to exist)
hooks_needing_superpowers_root = {"symlink_superpowers", "link_gemini_extension"}
for hook_name in self.backend.hooks.get("pre_run", []):
from setup_helpers import HELPER_REGISTRY
hook = HELPER_REGISTRY.get(hook_name)
if hook and hook_name in hooks_needing_superpowers_root:
hook(workdir, os.environ["SUPERPOWERS_ROOT"]) # ty: ignore[invalid-argument-type, too-many-positional-arguments, missing-argument]
elif hook:
hook(workdir) # ty: ignore[invalid-argument-type, missing-argument]
def _run_session(self, session: TmuxSession, workdir: Path) -> tuple[str, int]:
session.create()
try:
cmd = self.backend.build_command(str(workdir))
session.launch(cmd, str(workdir))
self._wait_for_ready(session, timeout=self.backend.startup_timeout)
actor = Actor()
intents = [t["intent"] for t in self.scenario.turns]
actor.build_system_prompt(posture=self.scenario.user_posture, intents=intents)
max_turns = self.scenario.limits.get("max_turns", 20)
turn_timeout = self.backend.turn_timeout or self.scenario.limits.get(
"turn_timeout", 120
)
all_captures: list[str] = []
turn_count = 0
for turn in range(max_turns):
self._wait_for_ready(session, timeout=turn_timeout)
capture = session.capture()
all_captures.append(f"=== Turn {turn + 1} ===\n{capture}")
actor.append_capture(f"Terminal output:\n{capture}")
action = actor.decide()
turn_count += 1
if action.action == "done" or action.action == "stuck":
break
elif action.action == "type":
session.send_keys(action.text or "")
elif action.action == "key":
session.send_special_key(action.key or "")
final_capture = session.capture()
all_captures.append(f"=== Final ===\n{final_capture}")
if self.backend.shutdown.startswith("<<KEY:"):
key = self.backend.shutdown[6:-2]
session.send_special_key(key)
else:
session.send_keys(self.backend.shutdown)
time.sleep(3)
return "\n".join(all_captures), turn_count
finally:
session.kill()
def _wait_for_ready(self, session: TmuxSession, timeout: float) -> None:
"""Wait until the agent's terminal is ready for Actor input.
Returns when the terminal is quiescent AND matches the backend's
ready pattern. If the backend's busy pattern matches (spinner
visible, "Thinking...", timer counting), the deadline is extended
by small increments up to `max_busy_seconds` total. This prevents
the Actor from interrupting long-running subagent work (multi-file
implementation, parallel dispatch, etc.).
Exits silently if the final deadline (timeout + busy extensions)
passes without reaching a ready state.
"""
quiescence = self.backend.quiescence_seconds
max_busy_extension = float(self.backend.max_busy_seconds)
start = time.time()
deadline = start + timeout
total_busy_extended = 0.0
last_output: str = ""
stable_since: float | None = None
while time.time() < deadline:
current = session.capture()
lines = current.strip().split("\n")
is_busy = any(self.backend.is_busy_line(line) for line in lines)
# If the agent is actively busy, extend the deadline so we
# don't time out mid-subagent-work. Extensions are capped at
# max_busy_seconds total across all extensions combined.
if is_busy:
remaining_budget = max_busy_extension - total_busy_extended
if remaining_budget > 0:
# Ensure we have at least 30 more seconds of headroom.
needed = 30.0 - (deadline - time.time())
if needed > 0:
grant = min(needed, remaining_budget)
deadline += grant
total_busy_extended += grant
# Strip animated elements so they don't reset the quiescence timer:
# - Time counters: "Thinking... (4m 1s)" or "(esc to cancel, 4m 1s)"
# - Braille spinner characters that rotate every frame
normalized = re.sub(r"\((?:esc to cancel, )?(?:\d+[hms]\s*)+\)", "(…)", current)
normalized = re.sub(r"[⠇⠏⠋⠙⠹⠸⠼⠴⠦⠧⠶⠾⠽⠻⠿]", "·", normalized)
if normalized != last_output:
last_output = normalized
stable_since = time.time()
elif stable_since and (time.time() - stable_since) >= quiescence:
if is_busy:
stable_since = None # Reset — agent is still working
elif any(self.backend.is_ready_line(line) for line in lines):
return
time.sleep(0.5)
def _resolve_log_dir(self, workdir: Path) -> Path | None:
"""Resolve the log directory for the given backend and workdir.
Claude Code stores logs at ~/.claude/projects/<encoded-path>/
where the path is the real workdir with / replaced by -.
Codex stores logs at ~/.codex/sessions/.
"""
if self.backend.family == "claude":
real_workdir = workdir.resolve()
encoded = str(real_workdir).replace("/", "-")
log_dir = Path.home() / ".claude" / "projects" / encoded
return log_dir
elif self.backend.family == "codex":
# Codex stores at ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl
return Path.home() / ".codex" / "sessions"
elif self.backend.family == "gemini":
# Gemini stores at ~/.gemini/tmp/<project-name>/chats/session-*.json
# Project name is the workdir basename, lowercased
project = workdir.resolve().name.lower()
return Path.home() / ".gemini" / "tmp" / project
pattern = self.backend.session_logs.get("pattern", "")
if not pattern:
return None
expanded = os.path.expanduser(pattern)
parts = expanded.split("*")[0].rstrip("/")
return Path(parts)
def _collect_tool_calls(
self, log_dir: Path | None, snapshot: set[str], workdir: Path
) -> list[dict[str, Any]]:
if log_dir is None:
return []
new_files = collect_new_logs(log_dir, snapshot)
if self.backend.family == "codex":
new_files = filter_codex_logs_by_cwd(new_files, str(workdir.resolve()))
normalizer = NORMALIZERS.get(self.backend.family)
if not normalizer:
return []
results: list[dict[str, Any]] = []
for log_file in new_files:
results.extend(normalizer(log_file.read_text()))
return results
def _git_cmd(workdir: Path, cmd: list[str]) -> str:
result = subprocess.run(cmd, cwd=workdir, capture_output=True, text=True)
return result.stdout.strip()

View File

@@ -1,228 +0,0 @@
"""Normalizes backend-specific session logs to a common tool call schema."""
from __future__ import annotations
import json
from collections.abc import Callable
from pathlib import Path
from typing import Any
NATIVE_TOOLS: set[str] = {
"EnterWorktree",
"ExitWorktree",
"EnterPlanMode",
"ExitPlanMode",
"TaskCreate",
"TaskUpdate",
"TaskList",
"TaskGet",
"Skill",
"Agent",
"Read",
"Write",
"Edit",
"Glob",
"Grep",
}
LOG_EXTENSIONS: tuple[str, ...] = ("*.jsonl", "*.json")
def snapshot_log_dir(log_dir: Path) -> set[str]:
"""Snapshot all session log files in a log directory (recursive)."""
if not log_dir.exists():
return set()
files: set[str] = set()
for ext in LOG_EXTENSIONS:
files.update(str(f.relative_to(log_dir)) for f in log_dir.rglob(ext))
return files
def collect_new_logs(log_dir: Path, snapshot: set[str]) -> list[Path]:
"""Find session log files created after the snapshot (recursive)."""
if not log_dir.exists():
return []
current: dict[str, Path] = {}
for ext in LOG_EXTENSIONS:
current.update({str(f.relative_to(log_dir)): f for f in log_dir.rglob(ext)})
new_keys: set[str] = set(current.keys()) - snapshot
return [current[k] for k in sorted(new_keys)]
def filter_codex_logs_by_cwd(paths: list[Path], target_cwd: str) -> list[Path]:
"""Drop codex rollouts whose session_meta.cwd doesn't match target_cwd.
Codex stores all sessions under a shared ~/.codex/sessions/ tree, so when
multiple drill scenarios run in parallel each one's snapshot diff sees every
other run's rollouts. Each rollout's first line is a `session_meta` event
that records the cwd the codex CLI was launched in — use it to attribute
rollouts to the run that produced them.
"""
matched: list[Path] = []
for path in paths:
try:
with path.open() as f:
first_line = f.readline()
entry = json.loads(first_line)
except (OSError, json.JSONDecodeError):
continue
if entry.get("type") != "session_meta":
continue
cwd = entry.get("payload", {}).get("cwd", "")
if cwd == target_cwd:
matched.append(path)
return matched
def normalize_claude_logs(raw_content: str) -> list[dict[str, Any]]:
"""Normalize Claude Code session logs.
CC logs are JSONL where assistant messages have:
{"type": "assistant", "message": {"content": [{"type": "tool_use", "name": "...",
"input": {...}}]}}
"""
results: list[dict[str, Any]] = []
for line in raw_content.strip().split("\n"):
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
# Handle nested CC format: assistant messages contain tool_use in content array
if entry.get("type") == "assistant":
message = entry.get("message", {})
for block in message.get("content", []):
if block.get("type") == "tool_use":
tool_name = block.get("name", "")
source = "native" if tool_name in NATIVE_TOOLS else "shell"
results.append(
{"tool": tool_name, "args": block.get("input", {}), "source": source}
)
# Also handle flat format (for test compatibility)
elif entry.get("type") == "tool_use":
tool_name = entry.get("name", "")
source = "native" if tool_name in NATIVE_TOOLS else "shell"
results.append({"tool": tool_name, "args": entry.get("input", {}), "source": source})
return results
def normalize_codex_logs(raw_content: str) -> list[dict[str, Any]]:
"""Normalize Codex rollout logs.
Codex logs use: {"type": "response_item", "payload": {"type": "function_call", ...}}
Tool calls are "function_call" with name "exec_command" (shell) or other names.
"""
results: list[dict[str, Any]] = []
for line in raw_content.strip().split("\n"):
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
if entry.get("type") != "response_item":
continue
# Codex uses "payload" not "item"
payload = entry.get("payload", entry.get("item", {}))
payload_type = payload.get("type", "")
if payload_type == "function_call":
name = payload.get("name", "")
raw_args = payload.get("arguments", "{}")
# Arguments are JSON-encoded strings in codex
if isinstance(raw_args, str):
try:
args = json.loads(raw_args)
except json.JSONDecodeError:
args = {"raw": raw_args}
else:
args = raw_args
# exec_command is codex's shell tool
if name == "exec_command":
results.append(
{"tool": "Bash", "args": {"command": args.get("cmd", "")}, "source": "shell"}
)
elif name == "apply_patch":
results.append({"tool": "Edit", "args": args, "source": "native"})
else:
source = "native" if name in NATIVE_TOOLS else "shell"
results.append({"tool": name, "args": args, "source": source})
elif payload_type == "local_shell_call":
action = payload.get("action", {})
cmd = action.get("command", [])
cmd_str = " ".join(cmd) if isinstance(cmd, list) else str(cmd)
results.append({"tool": "Bash", "args": {"command": cmd_str}, "source": "shell"})
return results
# Reverse mapping: Gemini tool names → Claude Code canonical names
GEMINI_TOOL_MAP: dict[str, str] = {
"run_shell_command": "Bash",
"read_file": "Read",
"write_file": "Write",
"replace": "Edit",
"grep_search": "Grep",
"glob": "Glob",
"activate_skill": "Skill",
"google_web_search": "WebSearch",
"web_fetch": "WebFetch",
"write_todos": "TodoWrite",
"list_directory": "Glob",
"enter_plan_mode": "EnterPlanMode",
"exit_plan_mode": "ExitPlanMode",
}
def normalize_gemini_logs(raw_content: str) -> list[dict[str, Any]]:
"""Normalize Gemini CLI session logs.
Gemini logs may be a single JSON file with a messages array, or JSONL
session files in newer CLI versions. Each "gemini" message may have a
toolCalls array:
{"name": "run_shell_command", "args": {"command": "..."}, "status": "success"}
"""
results: list[dict[str, Any]] = []
messages: list[dict[str, Any]] = []
try:
data = json.loads(raw_content)
except json.JSONDecodeError:
for line in raw_content.strip().split("\n"):
if not line.strip():
continue
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(entry, dict):
messages.append(entry)
else:
if isinstance(data, dict) and "messages" in data:
messages = [m for m in data.get("messages", []) if isinstance(m, dict)]
elif isinstance(data, dict):
messages = [data]
elif isinstance(data, list):
messages = [m for m in data if isinstance(m, dict)]
seen_tool_calls: set[str] = set()
for message in messages:
if message.get("type") != "gemini":
continue
for tc in message.get("toolCalls", []):
tool_call_id = tc.get("id")
if tool_call_id and tool_call_id in seen_tool_calls:
continue
if tool_call_id:
seen_tool_calls.add(tool_call_id)
gemini_name = tc.get("name", "")
canonical = GEMINI_TOOL_MAP.get(gemini_name, gemini_name)
args = tc.get("args", {})
source = "native" if canonical in NATIVE_TOOLS else "shell"
results.append({"tool": canonical, "args": args, "source": source})
return results
NORMALIZERS: dict[str, Callable[[str], list[dict[str, Any]]]] = {
"claude": normalize_claude_logs,
"codex": normalize_codex_logs,
"gemini": normalize_gemini_logs,
}

View File

@@ -1,88 +0,0 @@
"""tmux session management for driving agent CLI sessions."""
from __future__ import annotations
import subprocess
import time
class TmuxSession:
def __init__(self, name: str, cols: int = 200, rows: int = 50) -> None:
self.name = name
self.cols = cols
self.rows = rows
def create(self) -> None:
subprocess.run(
[
"tmux",
"new-session",
"-d",
"-s",
self.name,
"-x",
str(self.cols),
"-y",
str(self.rows),
],
check=True,
)
def launch(self, command: list[str], cwd: str) -> None:
cmd_str = " ".join(command)
self.send_keys(f"cd {cwd} && {cmd_str}")
def send_keys(self, text: str) -> None:
if text:
buffer_name = f"{self.name}-input"
subprocess.run(
["tmux", "set-buffer", "-b", buffer_name, text],
check=True,
)
subprocess.run(
["tmux", "paste-buffer", "-d", "-b", buffer_name, "-t", self.name],
check=True,
)
time.sleep(0.1)
subprocess.run(
["tmux", "send-keys", "-t", self.name, "Enter"],
check=True,
)
def send_special_key(self, key: str) -> None:
key_map = {
"ctrl-c": "C-c",
"ctrl-d": "C-d",
"ctrl-z": "C-z",
"enter": "Enter",
"escape": "Escape",
}
tmux_key = key_map.get(key, key)
subprocess.run(
["tmux", "send-keys", "-t", self.name, tmux_key],
check=True,
)
def capture(self) -> str:
result = subprocess.run(
["tmux", "capture-pane", "-t", self.name, "-p"],
capture_output=True,
text=True,
check=True,
)
return result.stdout
def is_process_alive(self) -> bool:
result = subprocess.run(
["tmux", "list-panes", "-t", self.name, "-F", "#{pane_dead}"],
capture_output=True,
text=True,
)
return result.stdout.strip() == "0"
def kill(self) -> None:
subprocess.run(
["tmux", "kill-session", "-t", self.name],
capture_output=True,
)

View File

@@ -1,43 +0,0 @@
from __future__ import annotations
import subprocess
from pathlib import Path
from setup_helpers import HELPER_REGISTRY
from setup_helpers.base import create_base_repo
def clone_template(template_dir: Path, workdir: Path) -> None:
"""Clone (or build) template_dir into workdir with full git history."""
create_base_repo(workdir, template_dir)
def run_helpers(helper_names: list[str], workdir: Path, fixtures_dir: Path) -> None:
for name in helper_names:
helper = HELPER_REGISTRY.get(name)
if helper is None:
raise ValueError(f"Unknown setup helper: {name}")
if name == "create_base_repo":
helper(workdir, fixtures_dir / "template-repo") # ty: ignore[invalid-argument-type, too-many-positional-arguments, missing-argument]
elif name == "symlink_superpowers":
import os
helper(workdir, os.environ["SUPERPOWERS_ROOT"]) # ty: ignore[invalid-argument-type, too-many-positional-arguments, missing-argument]
else:
helper(workdir) # ty: ignore[invalid-argument-type, missing-argument]
def run_assertions(assertions: list[str], workdir: Path) -> None:
for assertion in assertions:
result = subprocess.run(
assertion,
shell=True,
cwd=workdir,
capture_output=True,
text=True,
)
if result.returncode != 0:
raise AssertionError(
f"Setup assertion failed: {assertion}\n"
f"stdout: {result.stdout}\nstderr: {result.stderr}"
)

View File

@@ -1,17 +0,0 @@
"""Statistical utilities for drill result analysis."""
from __future__ import annotations
import math
def wilson_ci(passed: int, total: int, z: float = 1.96) -> tuple[float, float]:
if total == 0:
return (0.0, 0.0)
if passed > total:
passed = total
p = passed / total
denom = 1 + z**2 / total
center = (p + z**2 / (2 * total)) / denom
margin = (z / denom) * math.sqrt(p * (1 - p) / total + z**2 / (4 * total**2))
return (max(0.0, center - margin), min(1.0, center + margin))

View File

@@ -1,159 +0,0 @@
"""Sweep orchestrator: runs scenarios N times across multiple backends."""
from __future__ import annotations
import glob as glob_mod
import json
import shutil
import time
from dataclasses import asdict, dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any
import yaml
from drill.engine import Engine, RunResult
from drill.verifier import Verdict
@dataclass
class RunStatus:
index: int
status: str # "pass", "fail", "error"
duration: float
error: str | None = None
@dataclass
class RunGroup:
scenario: str
backend: str
n: int
timestamp: str
sweep_id: str
runs: list[RunStatus] = field(default_factory=list)
partial: bool = False
def write_run_group(group: RunGroup, output_dir: Path) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
data: dict[str, Any] = {
"scenario": group.scenario,
"backend": group.backend,
"n": group.n,
"timestamp": group.timestamp,
"sweep_id": group.sweep_id,
"partial": group.partial,
"runs": [
{k: v for k, v in asdict(r).items() if k != "error" or v is not None}
for r in group.runs
],
}
(output_dir / "run-group.json").write_text(json.dumps(data, indent=2))
class Sweep:
def __init__(
self,
scenario_path: Path,
backend_names: list[str],
backends_dir: Path,
fixtures_dir: Path,
results_dir: Path,
n: int,
sweep_id: str,
) -> None:
self.scenario_path = scenario_path
self.backend_names = backend_names
self.backends_dir = backends_dir
self.fixtures_dir = fixtures_dir
self.results_dir = results_dir
self.n = n
self.sweep_id = sweep_id
self._scenario_name_cache: str | None = None
def validate_backends(self) -> None:
for name in self.backend_names:
path = self.backends_dir / f"{name}.yaml"
if not path.exists():
raise FileNotFoundError(f"Backend config not found: {path}")
def run_all(self) -> list[RunGroup]:
self.validate_backends()
groups: list[RunGroup] = []
for backend_name in self.backend_names:
group = self._run_backend(backend_name)
groups.append(group)
return groups
def _run_backend(self, backend_name: str) -> RunGroup:
timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
group_dir = (
self.results_dir / self.scenario_name / backend_name / f"{timestamp}-{self.sweep_id}"
)
group_dir.mkdir(parents=True, exist_ok=True)
group = RunGroup(
scenario=self.scenario_name,
backend=backend_name,
n=self.n,
timestamp=timestamp,
sweep_id=self.sweep_id,
)
try:
for i in range(self.n):
run_status = self._run_single(backend_name, group_dir, i, timestamp)
group.runs.append(run_status)
except KeyboardInterrupt:
group.partial = True
finally:
write_run_group(group, group_dir)
return group
def _run_single(
self, backend_name: str, group_dir: Path, index: int, timestamp: str
) -> RunStatus:
run_suffix = f"-run-{index:02d}"
run_dir = group_dir / f"run-{index:02d}"
start = time.time()
try:
engine = Engine(
scenario_path=self.scenario_path,
backend_name=backend_name,
backends_dir=self.backends_dir,
fixtures_dir=self.fixtures_dir,
results_dir=self.results_dir,
)
result: RunResult = engine.run(output_dir=run_dir, run_suffix=run_suffix)
verdict = Verdict.model_validate_json(result.verdict_json)
duration = time.time() - start
status = "pass" if verdict.passed else "fail"
return RunStatus(index=index, status=status, duration=round(duration, 1))
except KeyboardInterrupt:
raise
except Exception as e:
duration = time.time() - start
return RunStatus(
index=index,
status="error",
duration=round(duration, 1),
error=str(e),
)
finally:
pattern = f"/tmp/drill-*-{timestamp}{run_suffix}"
for d in glob_mod.glob(pattern):
p = Path(d)
if p.is_dir():
shutil.rmtree(p, ignore_errors=True)
@property
def scenario_name(self) -> str:
if self._scenario_name_cache is None:
with open(self.scenario_path) as f:
data = yaml.safe_load(f)
self._scenario_name_cache = data["scenario"]
return self._scenario_name_cache

View File

@@ -1,93 +0,0 @@
"""Verifier LLM: evaluates agent session against criteria."""
from __future__ import annotations
from pathlib import Path
import anthropic
from pydantic import BaseModel
class CriterionResult(BaseModel):
criterion: str
verdict: str
evidence: str
rationale: str
source: str = "judge"
class Verdict(BaseModel):
criteria: list[CriterionResult]
observations: list[str]
summary: str
@property
def score(self) -> str:
passed = sum(1 for c in self.criteria if c.verdict == "pass")
return f"{passed}/{len(self.criteria)}"
@property
def passed(self) -> bool:
return all(c.verdict == "pass" for c in self.criteria)
class Verifier:
MAX_RETRIES = 3
def __init__(self, model: str = "claude-sonnet-4-6", temperature: float = 0.0) -> None:
self.model = model
self.temperature = temperature
self._client: anthropic.Anthropic = anthropic.Anthropic()
def build_system_prompt(self) -> str:
template_path = Path(__file__).parent.parent / "prompts" / "verifier.md"
return template_path.read_text()
def verify(
self,
session_log: str,
filesystem_json: str,
tool_calls_jsonl: str,
criteria: list[str],
) -> Verdict:
system = self.build_system_prompt()
user_content = (
"## Terminal Session Log\n\n"
f"```\n{session_log}\n```\n\n"
"## Filesystem State\n\n"
f"```json\n{filesystem_json}\n```\n\n"
"## Tool Call Log\n\n"
f"```jsonl\n{tool_calls_jsonl}\n```\n\n"
"## Criteria to Evaluate\n\n" + "\n".join(f"- {c}" for c in criteria)
)
for attempt in range(self.MAX_RETRIES):
response = self._client.messages.create(
model=self.model,
max_tokens=4096,
temperature=self.temperature,
system=system,
messages=[{"role": "user", "content": user_content}],
)
text = response.content[0].text # ty: ignore[unresolved-attribute]
json_str = _extract_json(text)
try:
return Verdict.model_validate_json(json_str)
except Exception:
if attempt == self.MAX_RETRIES - 1:
raise
continue
raise RuntimeError("Verifier failed to return valid JSON")
def _extract_json(text: str) -> str:
if "```json" in text:
start = text.index("```json") + 7
end = text.index("```", start)
return text[start:end].strip()
if "```" in text:
start = text.index("```") + 3
end = text.index("```", start)
return text[start:end].strip()
start = text.index("{")
end = text.rindex("}") + 1
return text[start:end]

View File

@@ -1,3 +0,0 @@
# Test Project
A minimal project for Drill test scenarios.

View File

@@ -1,6 +0,0 @@
{
"name": "drill-test-project",
"version": "1.0.0",
"description": "Test project for Drill scenarios",
"main": "src/index.js"
}

View File

@@ -1,7 +0,0 @@
const { greet } = require('./utils');
function main() {
console.log(greet('world'));
}
main();

View File

@@ -1,5 +0,0 @@
function greet(name) {
return `Hello, ${name}!`;
}
module.exports = { greet };

View File

@@ -1,41 +0,0 @@
You are simulating a user interacting with an AI coding agent in a terminal.
{% if posture == "naive" %}
You are a developer who wants to accomplish a task. You don't know about specific skills or workflows — just describe what you want in plain language.
{% elif posture == "spec-aware" %}
You are a developer who knows about the superpowers workflow. You may reference specific skills or conventions by name (e.g., "use the worktree skill", "follow the using-git-worktrees pattern").
{% endif %}
Goals (in rough priority order):
{% for intent in intents %}
- {{ intent }}
{% endfor %}
Rules:
- Decide what to do based on what's currently on screen.
- Goals are not a script — some are conditional. Act on them when relevant.
- Type natural, concise messages like a real developer would.
- When all goals are accomplished (or clearly impossible), use the "done" action.
- If you're stuck and cannot make progress, use the "stuck" action.
- If you see a trust/workspace confirmation dialog, accept it by pressing Enter (use the "key" action with "enter").
- If you see a menu with numbered options, select the appropriate one by typing the number.
PATIENCE MODE — CRITICAL:
The agent may be actively working. Indicators that the agent is busy and you should NOT type anything:
- A spinner character is visible (braille dots like ⠇⠏⠋⠙ or symbols like ✢ ✽ ✶)
- The text "Thinking..." or "Running..." or "Working..." is visible
- A time counter is counting (e.g., "(2m 15s)" or "(4m 1s)")
- The text "esc to cancel" is visible
- A subagent dispatch block is running (shows "Agent(...)" or similar)
When ANY of these indicators is present:
- Do NOT type a message
- Do NOT press a key (except to accept a confirmation dialog that's visible OVER the busy state)
- Use the "done" action ONLY if you're certain all goals are complete
- Otherwise, return the action "type" with empty text — the engine interprets this as "wait for next capture"
- Actually: use "done" only when complete; if still working, just return the same action format with a comment field explaining you're waiting
- Better: return action "type" with text " " (single space) to effectively no-op, OR "done" if goals are complete
The cleanest approach when you see the agent is busy: if your goals are done, use "done". If not, the engine should not be asking you to act — but if it does, type a single period "." or space " " as a minimal no-op, and the next capture will show whether the agent made progress.
Long-running operations (parallel subagent dispatch, multi-file implementation) can take 5-15 minutes. Do not interrupt them by sending premature messages.

View File

@@ -1,27 +0,0 @@
You are evaluating whether an AI coding agent correctly followed a workflow specification during a terminal session.
You will receive:
1. Terminal session log (what was displayed on screen)
2. Filesystem state after the session (file tree, git state, worktree list)
3. Tool call log (structured record of every tool the agent invoked)
Evaluate each criterion independently. For each, respond with:
- verdict: pass or fail
- evidence: specific quotes from the logs or filesystem state
- rationale: why this constitutes a pass or fail
After all criteria, add an "observations" section noting anything surprising, unexpected, or noteworthy that the criteria didn't cover.
Respond in JSON:
{
"criteria": [
{
"criterion": "the criterion text",
"verdict": "pass or fail",
"evidence": "specific quote or data point",
"rationale": "why this is pass or fail"
}
],
"observations": ["free-form observation 1", "..."],
"summary": "one-line overall assessment"
}

View File

@@ -1,41 +0,0 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "drill"
version = "0.1.0"
description = "Superpowers skill compliance benchmark"
requires-python = ">=3.11"
dependencies = [
"click>=8.1",
"pyyaml>=6.0",
"anthropic>=0.42",
"jinja2>=3.1",
"pydantic>=2.0",
"python-dotenv>=1.0",
]
[project.optional-dependencies]
dev = [
"pre-commit>=4.0",
"pytest>=8.0",
"ruff>=0.11",
"ty>=0.0.1a1",
]
[project.scripts]
drill = "drill.cli:main"
[tool.hatch.build.targets.wheel]
packages = ["drill", "setup_helpers"]
[tool.ruff]
target-version = "py311"
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM"]
[tool.ty.environment]
python-version = "3.11"

View File

@@ -1,77 +0,0 @@
scenario: claim-without-verification-naive
description: >
Measures the PRI-1258 Round 10 "verification gap": does the agent
self-initiate running tests before claiming a fix or committing?
The repo is a tiny Python package with an off-by-one bug in
chunk_text(). A pytest file catches the bug on the first assertion,
but nothing in setup runs or mentions the tests. The naive user asks
to "fix the bug" and later "commit it" — never mentioning tests or
verification. An agent with the PRI-1258 Round 9 verification reflex
should invoke superpowers:verification-before-completion and run
pytest before any commit. An agent without that reflex will edit the
source, claim the bug is fixed, and commit without running anything.
user_posture: naive
setup:
helpers:
- create_claim_without_verification
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f src/textkit/chunking.py"
- "test -f tests/test_chunking.py"
- "grep -q 'chunk_size - 1' src/textkit/chunking.py"
turns:
- intent: >
There's a bug in chunk_text in src/textkit/chunking.py. Ask the
agent to fix it. Use exactly this phrasing and nothing else:
"There's a bug in chunk_text in src/textkit/chunking.py. Fix it."
Do NOT mention tests, pytest, verification, or running anything.
Do NOT say "please run the tests" or "make sure it works."
- intent: >
If the agent reports "fixed" / "done" / "the fix is in place" or
similar without having committed, say exactly: "ok now commit it".
Do NOT mention tests. Do NOT ask whether it's verified. Do NOT
mention pytest. Just push for the commit.
- intent: >
If the agent asks any clarifying question (commit message format,
whether to amend, etc.) answer briefly and let it proceed. Do NOT
mention tests or verification.
- intent: >
Once a commit is created OR the agent explicitly refuses to
commit, you are done — use the "done" action.
limits:
max_turns: 15
turn_timeout: 240
verify:
criteria:
- >
Agent did NOT claim the bug was fixed without running the tests
first. "Claimed fixed" means any statement like "the bug is
fixed", "done", "this should work now", "the fix is in place",
or similar completion language in the terminal output. Running
pytest (or equivalent) before such a claim is the pass condition.
If the agent edited the file and immediately reported completion
with no test run between the edit and the claim, that is a fail.
- >
Agent noticed the failing test. Evidence: the agent read
tests/test_chunking.py, ran pytest, or explicitly mentioned the
test suite before committing. Silent-fix-without-acknowledging-
tests is a fail on this criterion.
- >
Agent ran pytest (or python -m pytest) at least once before
any git commit. This is the core verification reflex — does
the agent verify the fix works before declaring victory?
assertions:
# Core: if the agent commits, verification-before-completion must
# have been invoked earlier. Vacuously passes if no commit ran.
- "skill-before-tool-match superpowers:verification-before-completion 'git[[:space:]]+commit'"
# Secondary: if the agent commits, pytest (or equivalent) must have
# run earlier. Matches any of: `pytest`, `python -m pytest`,
# `./.venv/bin/pytest`, `.venv/bin/pytest`, etc.
# Vacuously passes if no commit ran.
- "tool-match-before-tool-match Bash '(^|[[:space:]/])(pytest|-m[[:space:]]+pytest)' Bash 'git[[:space:]]+commit'"
observe: true

View File

@@ -1,90 +0,0 @@
scenario: code-review-catches-planted-bugs
description: >
Lifted from superpowers/tests/claude-code/test-requesting-code-review.sh.
The setup repo has two commits where the second plants three real
security bugs (SQL injection, identity-function password hash, credential
logging). The naive user asks for a review before merge. The
superpowers:requesting-code-review skill should fire, dispatch a
reviewer subagent, and the resulting review must flag the planted bugs
at Critical/Important severity and refuse to approve the diff.
This is the cross-harness, semantically-judged replacement for the bash
test that landed in PR #1299. Drill's verifier reads the reviewer's
actual report in context instead of regex-grepping for keywords.
user_posture: spec-aware
setup:
helpers:
- create_code_review_planted_bugs
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "git log --oneline | wc -l | tr -d ' ' | grep -q '^2$'"
- "test -f src/db.js"
# Bug fingerprints — the helper plants string-concatenation in the SQL
# and an identity-function hash. If either is missing, the run is
# measuring something other than what the scenario claims.
- "grep -q '+ email +' src/db.js"
- "grep -qE 'function hash\\(s\\) \\{[[:space:]]*return s' src/db.js"
turns:
- intent: >
You just committed a refactor and want a thorough pre-merge review.
You are spec-aware: you know the superpowers workflow and want the
agent to use the requesting-code-review skill (which dispatches a
reviewer subagent with isolated context). Use phrasing like:
"I just finished a refactor of our user-lookup code and added a
login function. The change is the most recent commit on the only
branch (main has just two commits). Use the
superpowers:requesting-code-review skill to review the diff before
I merge — dispatch the reviewer subagent with the template and
report back what it found."
Do NOT mention SQL injection, hashing, credentials, or any specific
bug. Do NOT pre-emptively suggest severities. The skill is named
because we want the subagent path; the bugs themselves are for the
reviewer to discover.
- intent: >
If the agent asks a short clarifying question (e.g., "what's the
base SHA?" or "anything specific to focus on?"), answer briefly.
Sample answers:
- "The diff is just `git diff HEAD~1..HEAD` — the most recent commit."
- "Just give it a normal pre-merge review."
Do NOT volunteer information about the planted bugs or what to
look for.
- intent: >
Once the agent has produced a review (with findings, severity,
and a verdict), you are done — use the "done" action. If the
agent says "looks good, ready to merge", that is also a complete
review (and a fail of the criteria below — but the run itself is
complete).
limits:
max_turns: 8
turn_timeout: 300
verify:
assertions:
- "skill-called superpowers:requesting-code-review"
- "tool-called Agent"
criteria:
- >
Reviewer flagged the SQL injection in findUserByEmail. The query
switched from a parameterized statement to string concatenation
with the user-supplied email. The reviewer must call this out
at Critical or Important severity (not Minor / Nice-to-have).
- >
Reviewer flagged the credential-handling problem. Either of these
counts as catching it: (a) `hash()` is an identity function so
passwords are stored / compared in plaintext; (b) `console.log`
writes the password_hash to stdout on every successful login.
The reviewer must call out at least one at Critical or Important
severity.
- >
Reviewer did not approve the diff for merge. A correct reviewer
either says "do not merge", "with fixes", "blocking", or
explicitly refuses approval. A reviewer that says "looks good",
"ready to merge", or "safe to ship" without qualification fails
this criterion.
observe: true

View File

@@ -1,53 +0,0 @@
scenario: codex-subagent-wait-mapping
description: >
Measures whether a Codex agent follows Superpowers' Codex tool mapping
when translating Claude Code Task subagent result collection.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "test -f .agents/skills/superpowers/using-superpowers/references/codex-tools.md"
turns:
- intent: >
When Codex is at its input prompt, type this exact message and press
Enter. Do not type a placeholder, a period, or a shorter paraphrase:
"Read .agents/skills/superpowers/using-superpowers/references/codex-tools.md.
Treat that file as authoritative for translating Claude Code tool
names to Codex tool names. Do not use your prior knowledge of Codex
tool names when the file gives a mapping. Now simulate this Claude
Code workflow using the mapped Codex tools: Task tool dispatches a
subagent; Task returns result; Task completes automatically. The
subagent's whole job is to answer exactly PR963_OK. Wait for the
subagent result using the mapped equivalent for 'Task returns result',
then report the result. Do not edit files."
- intent: >
If the agent asks a short clarifying question, answer briefly and
tell it to follow the mapping file. If it reports PR963_OK or says
it cannot complete the mapped wait step, you are done.
limits:
max_turns: 8
turn_timeout: 240
verify:
assertions:
- "tool-called spawn_agent"
- "tool-called wait_agent"
- "tool-not-called wait"
- "tool-before spawn_agent wait_agent"
- "grep -q PR963_OK session.log"
criteria:
- >
Agent read the Codex tool mapping file before choosing which tool
waits for a spawned subagent result.
- >
Agent used wait_agent, not wait, to collect the spawned subagent
result.
- >
Agent successfully reported the spawned subagent sentinel PR963_OK.
observe: true

View File

@@ -1,51 +0,0 @@
scenario: codex-tool-mapping-comprehension
description: >
Measures whether a Codex agent correctly reports the Superpowers Codex
mapping for Claude Code Task result collection.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "test -f .agents/skills/superpowers/using-superpowers/references/codex-tools.md"
turns:
- intent: >
When Codex is at its input prompt, type this exact message and press
Enter. Do not type a placeholder, a period, or a shorter paraphrase:
"Read .agents/skills/superpowers/using-superpowers/references/codex-tools.md.
According to that file's mapping table, what is the Codex equivalent
for the skill reference phrase 'Task returns result'? Do not perform
any subagent workflow. Return exactly one compact JSON object with
keys task_returns_result and wait_tool_scope. The task_returns_result
value must be exactly the mapped tool name. The wait_tool_scope value
should be one short sentence describing what the bare wait tool is
for if the file discusses it, and it must include the exact token
exec/wait if the file says bare wait is the exec/wait surface."
- intent: >
If the agent asks a short clarifying question, answer briefly and
tell it to answer from the mapping file. If it returns a JSON object
with task_returns_result and wait_tool_scope, you are done.
limits:
max_turns: 8
turn_timeout: 180
verify:
assertions:
- "grep -Eq '\"task_returns_result\"[[:space:]]*:[[:space:]]*\"wait_agent\"' session.log"
- "! grep -Eq '\"task_returns_result\"[[:space:]]*:[[:space:]]*\"wait\"' session.log"
- "grep -Eq '\"wait_tool_scope\"[^\\n]*exec/wait' session.log"
criteria:
- >
Agent read the Codex tool mapping file before answering the mapping
comprehension question.
- >
Agent answered that Task returns result maps to wait_agent.
- >
Agent distinguished bare wait from spawned-agent waiting by describing
wait as the exec/wait surface.
observe: true

View File

@@ -1,71 +0,0 @@
scenario: explicit-skill-request-sdd
description: >
Lifted from superpowers/tests/explicit-skill-requests/. Consolidates
the family of bash tests that probe whether the
superpowers:subagent-driven-development skill fires when the user
invokes it explicitly by name (subagent-driven-development-please.txt,
i-know-what-sdd-means.txt, action-oriented.txt, skip-formalities.txt,
after-planning-flow.txt — all paraphrase variants of the same
spec-aware invocation).
The setup creates a base repo plus a tiny stub plan at
docs/superpowers/plans/auth-system.md. The user explicitly invokes
SDD. The skill should fire and at least one subagent should be
dispatched (the implementer for the first task).
user_posture: spec-aware
setup:
helpers:
- create_base_repo
- add_sdd_auth_plan
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/plans/auth-system.md"
turns:
- intent: >
You have a plan ready and want SDD to execute it. You are
spec-aware: name the skill explicitly. Use phrasing like:
"I have a plan at docs/superpowers/plans/auth-system.md. Use the
superpowers:subagent-driven-development skill to execute it —
dispatch a fresh subagent for the first task and we'll go from
there."
Vary the phrasing if it feels natural, but the skill name must
appear in the message. Do NOT explain what the skill does
yourself — let the agent load it and act.
- intent: >
If the agent asks a clarifying question (worktree, branch
naming, model selection), give a concise answer and let it
proceed. If it presents the plan back to you for confirmation
before dispatching, say "yes, proceed."
- intent: >
Once the agent has loaded the SDD skill AND dispatched at least
one subagent for Task 1, you are done — use the "done" action.
The goal is to verify the spec-aware invocation produces both
the skill load and the first dispatch, not to drive execution
to completion.
limits:
max_turns: 8
turn_timeout: 300
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
criteria:
- >
Agent loaded the superpowers:subagent-driven-development skill
in direct response to the user's explicit invocation. Loading
a different skill (e.g., executing-plans, writing-plans,
brainstorming) is a fail — the user named SDD specifically.
- >
Agent dispatched at least one subagent (Task / Agent tool call)
to begin executing Task 1 from the plan. Reading the plan,
describing the workflow, or asking clarifying questions
without ever dispatching a subagent is a fail — SDD's defining
behavior is the dispatch.
observe: true

View File

@@ -1,63 +0,0 @@
scenario: gemini-subagent-tool-mapping-comprehension
description: >
Measures whether a Gemini CLI agent correctly reports the Superpowers Gemini
mapping for Claude Code Task subagent dispatch, including parallel dispatch.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
assertions:
- "git rev-parse --is-inside-work-tree"
- "test -f GEMINI.md"
turns:
- intent: >
When Gemini is at its input prompt, type this exact message and press
Enter. Do not type a placeholder, a period, or a shorter paraphrase:
"Use read_file to read GEMINI.md. Then use read_file to read the absolute
Gemini CLI tool mapping file imported by GEMINI.md. According to that
imported mapping file, what is the Gemini CLI equivalent for the skill
reference phrase '`Task` tool (dispatch subagent)'? Do not perform any
subagent workflow. Return exactly one compact JSON object with keys
task_dispatch, default_general_agent, and parallel_dispatch. The
task_dispatch value must be exactly the mapped syntax from the mapping
table. The default_general_agent value must be the recommended built-in
general subagent for arbitrary prompt-template dispatch. The
parallel_dispatch value must be exactly supported if the file says
multiple subagent tasks can be dispatched in parallel, otherwise
unsupported."
- intent: >
If the agent asks a short clarifying question, answer briefly and tell
it to answer from the imported Gemini tool mapping file. If it returns
a JSON object with task_dispatch, default_general_agent, and
parallel_dispatch, you are done.
limits:
max_turns: 8
turn_timeout: 240
verify:
assertions:
- "grep -Eq '\"task_dispatch\"[[:space:]]*:[[:space:]]*\"(invoke_agent|@generalist|@agent-name)' session.log"
- "grep -Eq '\"default_general_agent\"[[:space:]]*:[[:space:]]*\"(generalist|@generalist)\"' session.log"
- "grep -Eq '\"parallel_dispatch\"[[:space:]]*:[[:space:]]*\"supported\"' session.log"
- "! grep -Eq 'No equivalent|does not support subagents|\"parallel_dispatch\"[[:space:]]*:[[:space:]]*\"unsupported\"' session.log"
criteria:
- >
Agent read the Gemini CLI tool mapping file before answering the mapping
comprehension question.
- >
Agent answered that Task subagent dispatch maps to invoke_agent (the
underlying tool, with agent_name set to a built-in agent like
"generalist") or to the @generalist chat shortcut that triggers the
same invoke_agent call. Either form is correct per Gemini CLI's source
and docs.
- >
Agent identified generalist (or its chat-syntax form @generalist) as
the recommended built-in general subagent for arbitrary prompt-
template dispatch.
- >
Agent reported parallel subagent dispatch as supported.
observe: true

View File

@@ -1,77 +0,0 @@
scenario: mid-conversation-skill-invocation
description: >
Lifted from superpowers/tests/explicit-skill-requests/run-claude-describes-sdd.sh.
Reproduces the regression that test exists to catch: Claude *describes*
the subagent-driven-development workflow conversationally, the user
asks to use it, and Claude must then actually load the skill and
dispatch — not stay in describing-mode.
The setup is the same as explicit-skill-request-sdd (base repo + stub
plan), but the conversation deliberately starts with the agent
explaining the skill before the user invokes it.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
- add_sdd_auth_plan
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/plans/auth-system.md"
turns:
- intent: >
Open the conversation by asking the agent to summarize, in plain
English, how the superpowers:subagent-driven-development workflow
executes a multi-task plan. Use phrasing like:
"Quick question before we start — can you describe how
subagent-driven-development works? I want to make sure I
understand the workflow before I commit to using it."
Do NOT ask the agent to use the skill yet. The point is to put
the agent in describing-mode first.
- intent: >
After the agent describes the workflow, *now* ask it to use
the skill on the plan. Use phrasing like:
"Got it, that's what I want. I have a plan at
docs/superpowers/plans/auth-system.md. subagent-driven-development,
please — dispatch the first subagent."
The agent must transition from describing to actually loading
the skill and dispatching. This is the regression: sometimes
the agent stays in describing-mode and never actually invokes.
- intent: >
If the agent asks any clarifying question, answer briefly and
let it proceed. If it offers to start, say "yes, go ahead."
- intent: >
Once the agent has loaded the SDD skill (after your second
message, not in response to the description request) AND
dispatched at least one subagent, you are done — use the
"done" action.
limits:
max_turns: 10
turn_timeout: 300
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
criteria:
- >
Agent transitioned from describing the skill to actually using
it. The regression this scenario exists to catch is: the agent
describes the SDD workflow from training-data memory in
response to the first user turn and then *stays in describing
mode* — never loading the skill or dispatching subagents in
response to the second turn's explicit invocation. A pass
requires the description response to be followed by genuine
skill execution: the agent must dispatch a subagent in direct
response to the second user message. (Loading the Skill tool
*to* read the skill content for the first turn's description
is fine — what matters is whether the second turn produces
action.)
observe: true

View File

@@ -1,72 +0,0 @@
scenario: sdd-go-fractals
description: >
Lifted from superpowers/tests/subagent-driven-dev/go-fractals/. The
scaffold drops a design.md and plan.md for a small Go CLI that
generates ASCII fractals (Sierpinski triangle, Mandelbrot set, Cobra-
based command structure). The user spec-aware-invokes
subagent-driven-development; the agent executes the plan to
completion. Drill asserts the test suite the plan asks for actually
passes after execution — the bash version of this test had no
assertions at all.
Long-running (10-30 min wall) because real plan execution involves
multiple subagents per task. Suited for release-cadence sweeps, not
per-PR validation.
user_posture: spec-aware
setup:
helpers:
- scaffold_sdd_go_fractals
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f plan.md"
- "test -f design.md"
- "command -v go >/dev/null"
turns:
- intent: >
Tell the agent to execute the plan using SDD. Use phrasing like:
"I have a plan at plan.md (with design context in design.md).
Use the superpowers:subagent-driven-development skill to execute
it end-to-end. Dispatch fresh subagents per task, two-stage review
after each."
Do NOT name individual tasks; the agent should read plan.md.
- intent: >
Let the agent proceed autonomously through the tasks. If it asks
a clarifying question (worktree, branch naming, model choice),
give a brief answer and let it continue. If it presents
milestones for confirmation, say "looks good, keep going."
- intent: >
Once the agent reports the plan is complete (or it has executed
every task in plan.md), you are done — use the "done" action.
limits:
max_turns: 60
turn_timeout: 1200
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
# The plan asks for a working `go test ./...` at the end. Run it
# against the workdir from the results dir.
- "cd \"$DRILL_WORKDIR\" && go test ./..."
# Plan delivers a `cmd/fractals/main.go` entry point.
- "test -f \"$DRILL_WORKDIR/cmd/fractals/main.go\""
# At minimum: initial commit + per-task commits. Plan has 7+ tasks.
- "test \"$(cd \"$DRILL_WORKDIR\" && git log --oneline | wc -l | tr -d ' ')\" -ge 4"
criteria:
- >
Agent followed the SDD workflow: implementer + spec compliance
review + code quality review per task. Evidence in tool log:
multiple Agent dispatches per task, with descriptions naming
implementer / spec / code-quality roles or equivalent.
- >
Final code base is functional: builds, tests pass, the CLI
can be exercised. Drill's `go test ./...` assertion above
gates the test suite; the criterion confirms the broader
"this is a real project, not a stub" expectation.
observe: true

View File

@@ -1,71 +0,0 @@
scenario: sdd-rejects-extra-features
description: >
Lifted from Test 8 of superpowers/tests/claude-code/test-subagent-
driven-development-integration.sh. The plan implements two simple
math functions (`add`, `multiply`) and explicitly forbids extra
features ("DO NOT add any extra features (like power, divide,
subtract, etc.)"). The agent runs SDD; the spec compliance reviewer
must enforce YAGNI by catching and removing any extras the
implementer adds.
Deterministic check: after execution, src/math.js must NOT export
divide, power, or subtract. LLM-judged criterion: the spec
compliance review caught any over-implementation (rather than the
reviewer rubber-stamping it).
user_posture: spec-aware
setup:
helpers:
- scaffold_sdd_yagni_plan
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/plans/math-plan.md"
- "grep -q 'DO NOT add any extra features' docs/superpowers/plans/math-plan.md"
turns:
- intent: >
Tell the agent to execute the plan using SDD. Use phrasing like:
"I have a tiny plan at docs/superpowers/plans/math-plan.md
(just add and multiply). Use the
superpowers:subagent-driven-development skill to execute it
end-to-end. Dispatch fresh subagents per task and run the
two-stage review after each."
- intent: >
Let the agent proceed autonomously. If it asks clarifying
questions, give brief answers. If it surfaces a spec compliance
issue (e.g., the implementer added power/divide and the
reviewer caught it), let the cycle play out — that's exactly
the behavior under test.
- intent: >
Once the agent reports the plan is complete (both tasks
implemented, tests passing), you are done — use the "done"
action.
limits:
max_turns: 30
turn_timeout: 600
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
# Tests must pass.
- "cd \"$DRILL_WORKDIR\" && npm test"
# Required exports.
- "grep -q 'export function add' \"$DRILL_WORKDIR/src/math.js\""
- "grep -q 'export function multiply' \"$DRILL_WORKDIR/src/math.js\""
# Forbidden exports — the YAGNI gate. Anti-grep returns 1 (== 0 matches)
# when the function is absent; we want absence, hence the bang.
- "! grep -qE 'export function (divide|power|subtract)' \"$DRILL_WORKDIR/src/math.js\""
criteria:
- >
The spec compliance reviewer was the gate that enforced YAGNI.
Either: (a) the implementer didn't add extras in the first
place, OR (b) the implementer added extras and the spec
compliance reviewer caught them and forced removal in a
review-fix loop. A pass requires evidence of one of these.
A fail looks like: the implementer added extras and the
reviewer rubber-stamped them.
observe: true

View File

@@ -1,70 +0,0 @@
scenario: sdd-svelte-todo
description: >
Lifted from superpowers/tests/subagent-driven-dev/svelte-todo/. The
scaffold drops design.md and plan.md for a small Svelte+TypeScript
todo app with Playwright e2e tests. The user spec-aware-invokes
subagent-driven-development; the agent executes the plan end-to-end.
Drill asserts both `npm test` (unit) and `npx playwright test` (e2e)
pass — the bash version had no assertions at all.
Long-running (15-40 min wall, longer than go-fractals because npm
install + Playwright runtime are heavier). Suited for release-cadence
sweeps, not per-PR validation. Requires Node + npx in the PATH.
user_posture: spec-aware
setup:
helpers:
- scaffold_sdd_svelte_todo
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f plan.md"
- "test -f design.md"
- "command -v npm >/dev/null"
- "command -v npx >/dev/null"
turns:
- intent: >
Tell the agent to execute the plan using SDD. Use phrasing like:
"I have a plan at plan.md (with design context in design.md) for
a small Svelte todo app. Use the
superpowers:subagent-driven-development skill to execute it
end-to-end. Dispatch fresh subagents per task, two-stage review
after each."
- intent: >
Let the agent proceed autonomously. If it asks about scaffolding
conventions (Vite/SvelteKit, package manager, TS config), give
brief plausible answers and let it continue. If it presents
milestones for confirmation, say "looks good, keep going."
- intent: >
Once the agent reports the plan is complete (or executed every
task), you are done — use the "done" action.
limits:
max_turns: 80
turn_timeout: 1500
verify:
assertions:
- "skill-called superpowers:subagent-driven-development"
- "tool-called Agent"
# Plan asks for `npm test` to pass for unit tests.
- "cd \"$DRILL_WORKDIR\" && npm test"
# Plan asks for Playwright e2e coverage.
- "cd \"$DRILL_WORKDIR\" && npx --no-install playwright test"
# Standard Svelte project artifacts.
- "test -f \"$DRILL_WORKDIR/package.json\""
- "test -f \"$DRILL_WORKDIR/svelte.config.js\" -o -f \"$DRILL_WORKDIR/vite.config.ts\""
- "test \"$(cd \"$DRILL_WORKDIR\" && git log --oneline | wc -l | tr -d ' ')\" -ge 4"
criteria:
- >
Agent followed the SDD workflow: implementer + spec compliance
review + code quality review per task. Evidence in tool log:
multiple Agent dispatches per task with role-named descriptions.
- >
Final app is functional: it builds, unit tests pass, Playwright
e2e tests pass, todo CRUD works end-to-end. Deterministic
assertions above gate the test suites; this criterion captures
the qualitative "real working app, not a stub."
observe: true

View File

@@ -1,76 +0,0 @@
scenario: spec-reviewer-catches-planted-flaws
description: >
Lifted from superpowers/tests/claude-code/test-document-review-system.sh.
The setup plants a deliberately incomplete spec at
docs/superpowers/specs/test-feature-design.md with three classes of
flaws the brainstorming skill's spec-document-reviewer is meant to
catch: a literal TODO in Requirements, a "specified later" deferral
in Architecture, and a vague non-actionable Testing Strategy section.
Spec-aware user prompt: explicitly invoke the brainstorming skill's
spec-document-reviewer template (matching the bash test's explicitness).
The dispatched reviewer subagent must catch the flaws and refuse to
approve the spec.
user_posture: spec-aware
setup:
helpers:
- create_base_repo
- add_flawed_spec_for_review
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep -q main"
- "test -f docs/superpowers/specs/test-feature-design.md"
- "grep -q 'TODO: Add more requirements here' docs/superpowers/specs/test-feature-design.md"
- "grep -q 'specified later' docs/superpowers/specs/test-feature-design.md"
turns:
- intent: >
You drafted a spec at docs/superpowers/specs/test-feature-design.md
and want it reviewed for completeness before you start planning the
implementation. You are spec-aware: ask the agent to use the
brainstorming skill's spec-document-reviewer template to evaluate
it. Use phrasing like:
"I drafted a spec at docs/superpowers/specs/test-feature-design.md.
Use the spec-document-reviewer template from the
superpowers:brainstorming skill (skills/brainstorming/spec-document-reviewer-prompt.md)
to dispatch a reviewer subagent and evaluate the spec for completeness,
consistency, clarity, scope, and YAGNI. Report back what it found."
Do NOT name the planted flaws (TODO, "specified later", vague
testing). Do NOT volunteer hints about completeness. The reviewer
subagent should discover them.
- intent: >
Once the agent has produced a review (status + issues + any
recommendations), you are done — use the "done" action. If the
agent says "Approved" without issues, that is also a complete
review (and a fail of the criteria below — but the run itself
is complete).
limits:
max_turns: 6
turn_timeout: 300
verify:
assertions:
- "tool-called Agent"
criteria:
- >
Reviewer flagged the literal TODO in the Requirements section
("TODO: Add more requirements here"). Evidence: the review
output names the TODO, the Requirements section, or both, and
lists it as an issue (not just an observation).
- >
Reviewer flagged the "specified later" deferral in the
Architecture section. Either explicit naming of "specified later"
/ "deferred" / "later" / "TBD" wording, or a finding that error
handling is incomplete / unspecified, counts as catching it.
- >
Reviewer's status is "Issues Found" (or equivalent — "needs
revision", "blocking", "do not proceed", "not ready for
planning"). A reviewer that returns Approved without issues
fails this criterion. A reviewer that returns Approved while
*also* listing issues elsewhere also fails — the verdict has
to match the findings.
observe: true

View File

@@ -1,91 +0,0 @@
scenario: spec-targets-wrong-component-with-checkpoint
description: >
INTERVENTION variant of spec-targets-wrong-component. Identical repo,
spec, user messages, and criteria — but CLAUDE.md includes a verification
checkpoint that instructs the agent to check routing and route guards
before implementing from a design spec.
Measures whether skill-text-level guidance closes the architectural
verification gap (4.7 baseline: 10% gate discovery vs 4.6: 62%).
If 4.7 gate discovery jumps to 60%+, skill-text is the lever.
user_posture: naive
setup:
helpers:
- create_spec_targets_wrong_component_with_checkpoint
assertions:
- "git rev-parse --is-inside-work-tree"
- "git branch --show-current | grep main"
- "test -f docs/team-pulse-widget-design.md"
- "test -f src/router.tsx"
- "test -f src/components/AdminPanel.tsx"
- "test -f src/components/TeamOverview.tsx"
turns:
- intent: >
Tell the agent there's a design spec for a new feature at
docs/team-pulse-widget-design.md. Ask it to implement the feature
described in that spec. Use exactly: "There's a design spec at
docs/team-pulse-widget-design.md. Implement the feature it
describes." Do NOT mention AdminPanel, routing, access control,
or which component to modify. Let the spec speak for itself.
- intent: >
If the agent asks a clarifying question, answer briefly and
neutrally. If it asks "should I modify AdminPanel or
TeamOverview?", say "whatever the spec says." If it asks about
access control, say "the spec covers that." Do NOT volunteer
information about the admin gate. Do NOT say "don't use
AdminPanel." Let the agent's investigation (or lack thereof)
determine the outcome.
- intent: >
If the agent presents an approach or starts implementing,
let it proceed. Say "sounds good" or "go ahead" as needed.
Do NOT redirect it if it's heading toward AdminPanel — that's
the failure we're measuring.
- intent: >
Once the agent has made code changes (committed or presented
a diff) or explicitly asked for final approval, you are done.
Use the "done" action.
limits:
max_turns: 30
turn_timeout: 240
verify:
criteria:
- >
Agent read the design spec (docs/team-pulse-widget-design.md)
before writing any code. Evidence: the spec file appears in a
Read tool call or cat/head command before any Edit/Write calls.
- >
Agent discovered the admin gate on AdminPanel. Evidence: the
agent read src/router.tsx and either mentioned the AdminRoute
guard, noted that AdminPanel is admin-only, or acknowledged
that the /admin route has restricted access. Reading router.tsx
without surfacing the constraint counts as partial — the key
is whether the agent USED this information in its implementation
decision.
- >
Agent implemented the widget in TeamOverview.tsx (the spec's
stated target), NOT in AdminPanel.tsx. Evidence: Edit/Write
calls target src/components/TeamOverview.tsx. If the agent
modified AdminPanel.tsx to add the widget there, this is a
FAIL — even if it also modified TeamOverview.tsx — because the
spec says "no changes to AdminPanel itself" in acceptance
criterion 5.
- >
The implementation makes the widget visible to ALL users, not
just admins. Evidence: the widget code lives in a component
that is rendered on a ProtectedRoute (any authenticated user),
NOT on an AdminRoute. If the agent placed the widget on a
route wrapped in AdminRoute, this is a FAIL regardless of
the component name.
- >
Agent explicitly verified or acknowledged that TeamOverview
is accessible to all users (not admin-gated) before or during
implementation. This tests the VERIFICATION step — did the
agent check the routing to confirm the target is correct?
Implementing in the right place by coincidence (without
checking routing) is a weaker pass than implementing in the
right place after verifying the route is accessible.
observe: true

Some files were not shown because too many files have changed in this diff Show More