mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
fix(acp): replace Goose native system prompt (#5964)
## Why Buzz currently appends its managed prompt to Goose's native prompt, so managed agents receive both instruction sets instead of the intended Buzz-only system prompt. ## What - Send Goose's custom session system-prompt request with `mode: "set"` - Lock the replacement contract in the ACP request test ## Risk Assessment Low — the change is limited to Goose session setup; adapters that do not implement Goose's custom method keep the existing method-not-found fallback behavior. ## References Goose v1.46.0 routes `set` to `override_system_prompt`, and its prompt builder selects that override instead of rendering the native `system.md`: [ACP handler](https://github.com/aaif-goose/goose/blob/98c11ce2ee7b9b302978aa64b1eab7d0895607c7/crates/goose/src/acp/server/manage_sessions.rs#L57-L93), [prompt builder](https://github.com/aaif-goose/goose/blob/98c11ce2ee7b9b302978aa64b1eab7d0895607c7/crates/goose/src/agents/prompt_manager.rs#L153-L191). Validated end to end against the official Goose v1.46.0 binary with a local OpenAI-compatible capture server: the provider request contained the exact Buzz replacement prompt and did not contain Goose's native base-prompt marker. --- **Update Aug 15, 13:17 CDT:** Added the [Terra-high prompt-ablation comparison](https://github.com/squareup/buzz-benchmarks/blob/4492f76349ccb638219f7d070735a4d2b679bc26/data/prompt-ablation/20260815-terra-high/comparison.md). The Goose conditions used GPT 5.6 Terra at high effort on the same 11 Terminal-Bench 2.1 tasks, with two attempts per task and concurrency four. The matched `append-full` and `set-full` runs used the same persona and included the same Buzz platform prompt; Active-h is the primary measure because it excludes Buzz lifecycle overhead. | Goose condition | Pass | Active-h | Median active | Agent-h | Wall-h | Tool calls | |---|---:|---:|---:|---:|---:|---:| | Native prompt + Buzz prompt (`append-full`) | 21/22 | 0.3042 | 0.85 min | 0.3974 | 0.1496 | 234 | | Native prompt + persona only (`append-persona-only`) | 22/22 | 0.3050 | 0.75 min | 0.3990 | 0.1498 | 204 | | Buzz prompt replaces native prompt (`set-full`) | 22/22 | 0.3340 | 0.87 min | 0.4296 | 0.1551 | 275 | Replacing instead of appending produced one additional passing attempt, but it was not an efficiency improvement in this small sample: versus `append-full`, `set-full` increased Active-h by 9.8%, median active by 2.0%, Agent-h by 8.1%, Wall-h by 3.7%, and tool calls by 17.5%. It was faster on only two of eleven per-task active-time medians (`distribution-search` and `prove-plus-comm`). With two attempts per task, these are directional results rather than confidence intervals; they support this change as an instruction-isolation/correctness fix, not a performance optimization, and argue against Goose's appended native prompt being the main source of active-time cost. Generated with Codex Signed-off-by: Atish Patel <atishpatel2012@gmail.com> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
@@ -702,7 +702,7 @@ impl AcpClient {
|
||||
.session_id)
|
||||
}
|
||||
|
||||
/// Send Goose's custom system-prompt request after `session/new`.
|
||||
/// Replace Goose's native system prompt after `session/new`.
|
||||
pub async fn session_set_goose_system_prompt(
|
||||
&mut self,
|
||||
session_id: &str,
|
||||
@@ -712,7 +712,7 @@ impl AcpClient {
|
||||
"_goose/unstable/session/system-prompt/set",
|
||||
serde_json::json!({
|
||||
"sessionId": session_id,
|
||||
"mode": "append",
|
||||
"mode": "set",
|
||||
"key": "buzz",
|
||||
"text": text,
|
||||
}),
|
||||
@@ -3421,7 +3421,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn goose_system_prompt_request_uses_append_contract() {
|
||||
async fn goose_system_prompt_request_uses_set_contract() {
|
||||
let script = r#"
|
||||
read -t 2 REQ
|
||||
echo '{"jsonrpc":"2.0","id":0,"result":{"_receivedRequest":'"$REQ"'}}'
|
||||
@@ -3438,7 +3438,7 @@ mod tests {
|
||||
"_goose/unstable/session/system-prompt/set"
|
||||
);
|
||||
assert_eq!(received["params"]["sessionId"], "ses_goose");
|
||||
assert_eq!(received["params"]["mode"], "append");
|
||||
assert_eq!(received["params"]["mode"], "set");
|
||||
assert_eq!(received["params"]["key"], "buzz");
|
||||
assert_eq!(received["params"]["text"], "Be terse");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user