mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
5b3f0375a2
## Why Buzz currently appends its managed prompt to Goose's native prompt, so managed agents receive both instruction sets instead of the intended Buzz-only system prompt. ## What - Send Goose's custom session system-prompt request with `mode: "set"` - Lock the replacement contract in the ACP request test ## Risk Assessment Low — the change is limited to Goose session setup; adapters that do not implement Goose's custom method keep the existing method-not-found fallback behavior. ## References Goose v1.46.0 routes `set` to `override_system_prompt`, and its prompt builder selects that override instead of rendering the native `system.md`: [ACP handler](https://github.com/aaif-goose/goose/blob/98c11ce2ee7b9b302978aa64b1eab7d0895607c7/crates/goose/src/acp/server/manage_sessions.rs#L57-L93), [prompt builder](https://github.com/aaif-goose/goose/blob/98c11ce2ee7b9b302978aa64b1eab7d0895607c7/crates/goose/src/agents/prompt_manager.rs#L153-L191). Validated end to end against the official Goose v1.46.0 binary with a local OpenAI-compatible capture server: the provider request contained the exact Buzz replacement prompt and did not contain Goose's native base-prompt marker. --- **Update Aug 15, 13:17 CDT:** Added the [Terra-high prompt-ablation comparison](https://github.com/squareup/buzz-benchmarks/blob/4492f76349ccb638219f7d070735a4d2b679bc26/data/prompt-ablation/20260815-terra-high/comparison.md). The Goose conditions used GPT 5.6 Terra at high effort on the same 11 Terminal-Bench 2.1 tasks, with two attempts per task and concurrency four. The matched `append-full` and `set-full` runs used the same persona and included the same Buzz platform prompt; Active-h is the primary measure because it excludes Buzz lifecycle overhead. | Goose condition | Pass | Active-h | Median active | Agent-h | Wall-h | Tool calls | |---|---:|---:|---:|---:|---:|---:| | Native prompt + Buzz prompt (`append-full`) | 21/22 | 0.3042 | 0.85 min | 0.3974 | 0.1496 | 234 | | Native prompt + persona only (`append-persona-only`) | 22/22 | 0.3050 | 0.75 min | 0.3990 | 0.1498 | 204 | | Buzz prompt replaces native prompt (`set-full`) | 22/22 | 0.3340 | 0.87 min | 0.4296 | 0.1551 | 275 | Replacing instead of appending produced one additional passing attempt, but it was not an efficiency improvement in this small sample: versus `append-full`, `set-full` increased Active-h by 9.8%, median active by 2.0%, Agent-h by 8.1%, Wall-h by 3.7%, and tool calls by 17.5%. It was faster on only two of eleven per-task active-time medians (`distribution-search` and `prove-plus-comm`). With two attempts per task, these are directional results rather than confidence intervals; they support this change as an instruction-isolation/correctness fix, not a performance optimization, and argue against Goose's appended native prompt being the main source of active-time cost. Generated with Codex Signed-off-by: Atish Patel <atishpatel2012@gmail.com> Co-authored-by: Codex <noreply@openai.com>