feat(agent): optional reply guard reminds a silent turn to publish (#3763)

## Why

A Buzz agent's assistant text and reasoning are never shown to anyone —
only what it posts through the CLI. A turn that runs fifteen tool calls
and never publishes is a silent failure: the requester waits on a result
that was produced and thrown away.

This adds an optional reminder at the end-of-turn gate, off by default.

Tyler asked for it in buzz-mesh; plan iterated to **9.5/10 with @Wren**
(Minimalness 9.7, Elegance 9.5, Correctness 9.3).

## What

`BUZZ_AGENT_REQUIRE_REPLY=1` (default off, per-agent opt-in). A turn
about to end with no recognized attempt to post gets a reminder and is
rerolled. **At most two, then the turn ends regardless** — the guard
catches accidental omission, it does not compel speech. The reminder
text explicitly licenses silence so it cannot fight the base prompt's
"silence is usually correct."

**This is not a new MCP hook.** `RunCtx::run` *is* the turn, so the two
per-turn locals need no plumbing, and every tool call already passes
through it with arguments visible. The objection is appended at the
existing `_Stop` gate and rides `push_hook_outputs_as_tool_results`, so
the model receives it as a lower-trust tool result with `{hook, server,
text}` attribution. No new trust path, no new lifecycle event, no
dev-mcp or CLI protocol change.

Earlier revisions of this plan needed four crates (a `_UserPromptSubmit`
hook, a marker file, a `buzz-cli` change, dev-mcp state). Tyler pointed
out the agent already knows both facts; that deleted all of it. Net
runtime change is ~35 lines in `agent.rs` + ~4 in `config.rs`.

### Recognition contract

A registered non-hook tool whose qualified name ends in `__shell`, whose
`command` argument contains `messages send` or `reactions add`.

- **The `__` separator is exact, not approximate.** Given `has()` +
`!is_hook()`, `ends_with("__shell")` is *provably equivalent* to a bare
name of `shell`: registration forbids `__` in server and bare names
(`mcp.rs:227,268`) and qnames are `{server}__{bare}`, so a trailing
`__shell` could only straddle the separator if the bare name began with
`_` — which `is_hook` excludes. Without the separator, `powershell` and
`noshell` would match.
- **Reads the structured `command` field**, not serialized arguments, so
a `description` that quotes a send cannot disarm the guard, and a
non-string `command` is rejected rather than coerced.
- **Detects an attempt, not a successful publish.** A failed send
already returns non-zero exit and error JSON — louder than this
reminder. The variable is named `buzz_reply_call_seen` so the code can't
pretend otherwise.
- **Checked after the per-turn tool-call cap**, since a discarded call
never ran.
- `messages send` also covers `messages send-diff`. Reactions count
because the base prompt directs agents to react rather than post a bare
acknowledgement.

**Known limits, both deliberate and documented:** a command assembled at
runtime (`$CMD`) or hidden in a wrapper script is missed; text that
merely quotes a send (`echo "buzz messages send"`) matches. Missing a
real post is the expensive direction and substring matching is the
forgiving one there. Neither edge is pinned by a test, so the matcher
stays free to improve.

### Budget

Reminders share `BUZZ_AGENT_STOP_MAX_REJECTIONS`, the existing outer cap
on every end-turn objection. Default 3 fits both; at 1 only one fits; at
0 the guard is off with the hooks. A round carrying both a hook
objection and a reminder costs one rejection and delivers both texts. An
independent budget would either violate that bound or need a second
arbitration rule.

## Prior art

- **#3467** (closed) built the same detector one layer up in `buzz-acp`
for a different remedy. None of its symbols are on main — this borrows
its permission to be coarse, but reads structured data that ACP didn't
have.
- **#3648** (open) detects turns with *no output at all*; a turn with
fifteen tool calls and no post counts as output there, so it does not
cover this case.
- **#3741** (merged) is mesh-only.

## Testing

**14 new tests.** 4 unit tests on the matcher; 10 integration tests
through the ACP wire harness: off by default, `=0` still off, opted-in
silent → exactly 2 reminders then `end_turn`, registered `fake__shell`
send → 0 reminders, hallucinated `fake__shell` → still reminded, publish
call truncated past the 64-call cap → still reminded, budget 1 → 1
reminder, budget 0 → off, combined `_Stop` hook objection + reminder →
one round both texts and after 2 reminders the hook objection continues
alone, unparseable `=true` → startup error naming the key.

**10 mutation checks, each breaking a specific named test** — neutralize
the nag cap, stop sharing the budget, neutralize `buzz_reply_call_seen`,
drop `has`/`is_hook`, ignore the flag, drop the `__`, drop `reactions
add`, read serialized args, move detection before truncation.

`tests/bin/fake_mcp.rs` gains `FAKE_MCP_SHELL_TOOL=1`: it previously
exposed no tool with a bare name of `shell`, so the satisfied-guard path
was untestable.

Full `cargo test -p buzz-agent` green at 9e0ae1f04; clippy `-D warnings`
and `cargo fmt --check` clean.

**Unrelated flake found:**
`cancelled_turn_with_usage_emits_notification_before_response`
(`tests/fake_llm.rs`) is timing-sensitive. Under 10 loaded cores it
fails **2/20 on this branch and 1/20 at unmodified
`origin/main@02be413b8`** — pre-existing, not caused by this change
(which is inert without the env var). Flagging so it isn't misattributed
to the next PR that's open when CI hits it.

## Docs

`crates/buzz-agent/README.md` is the primary home (env var, recognition
contract, limits, budget interaction). `docs/MCP_DRIVEN_HOOKS.md` gets a
short cross-reference explaining this is *not* a hook — otherwise
readers hunt for a `_ReplyGuard` tool that doesn't exist.

---------

Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Co-authored-by: Dawn (sprout agent) <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
This commit is contained in:
Tyler
2026-07-31 07:02:48 -04:00
committed by GitHub
co-authored by Dawn npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta
parent d48b0e0eec
commit 081f805d5e
8 changed files with 847 additions and 3 deletions
+61
View File
@@ -163,6 +163,67 @@ Everything is environment variables. No flags, no config files. (We are a subpro
| `BUZZ_AGENT_MAX_LINE_BYTES` | `4194304` | 4 MiB. Hard cap on inbound JSON-RPC frames. |
| `BUZZ_AGENT_MAX_HISTORY_BYTES` | `1048576` | 1 MiB. Old turns are evicted past this. |
| `BUZZ_AGENT_MAX_TOOL_RESULT_TEXT_BYTES` | `51200` | 50 KiB. Per-result cap on tool-output text; oversize is middle-elided (head + tail kept) with an inline marker. Images are exempt. |
| `BUZZ_AGENT_REQUIRE_REPLY` | `0` (`1` on mesh) | `1` enables the [reply guard](#reply-guard) — remind the model to publish when a turn is about to end with nothing posted to Buzz. Desktop defaults it to `1` for Buzz shared-compute agents. |
## Reply Guard
Off by default, except on Buzz shared-compute (mesh) agents, where Buzz Desktop
sets `BUZZ_AGENT_REQUIRE_REPLY=1` automatically. With it enabled, a turn that is
about to end without any recognized attempt to post to Buzz gets a reminder that
its assistant text is invisible to humans, and is rerolled.
This exists because a Buzz agent's reasoning and tool output are not shown to
anyone. A turn that does real work and never posts is a silent failure — the
requester waits on a result that was produced and thrown away.
Mesh agents get it by default because they run on small local models, which are
the ones most likely to do the work and then end the turn without publishing it.
Setting `BUZZ_AGENT_REQUIRE_REPLY=0` on the agent, persona, or global env opts a
mesh agent back out; the default never overrides an explicit value.
**Advisory, never a trap.** At most two reminders, then the turn ends whether or
not anything was published. The guard catches accidental omission; it does not
compel speech. The reminder text explicitly licenses silence, because the
built-in system prompt says publishing is optional and silence is often the
correct outcome.
**Recognition contract.** A turn counts as having replied when it issues a call
that:
- resolves to a registered, non-hook tool (a hallucinated tool name is rejected
at preflight and never runs, so it must not disarm the guard),
- whose qualified name ends in `__shell` — i.e. the bare tool name is exactly
`shell`, which is `buzz-dev-mcp`'s shell tool and any other server's, and
- whose `command` argument contains `messages send` or `reactions add`.
`messages send` also covers `messages send-diff`. Reactions count because the
built-in prompt directs agents to react rather than post a bare
acknowledgement, so nagging an agent that reacted would punish documented
behavior.
Detection is checked **after** the per-turn tool-call cap
(`MAX_TOOL_CALLS_PER_TURN`) is applied: a publish-shaped call that was discarded
never ran.
**It recognizes an attempt, not a successful publish.** Only the command text is
inspected, never the exit status. A send that fails still satisfies the guard —
which is fine, since a failed send already returns a non-zero exit and error
JSON to the model, louder feedback than a reminder.
**Known limits**, both deliberate. A command assembled at runtime (`$CMD`) or
buried in a wrapper script is missed, so that turn is reminded despite having
posted. Text that merely quotes a send (`echo "buzz messages send"`) matches, so
that turn is not reminded. Missing a real post is the expensive direction, and
substring matching is the forgiving one there. Neither edge is pinned by a test;
the matcher is free to improve.
**Budget.** Reminders ride the existing `_Stop` gate and share
`BUZZ_AGENT_STOP_MAX_REJECTIONS` — the outer cap on every end-turn objection.
At the default 3 both reminders fit; at 1 only one does; at 0 the guard is off
along with the hooks. A round carrying both a `_Stop` hook objection and a
reminder costs one rejection and delivers both texts. This is not a new
lifecycle hook — see [MCP_DRIVEN_HOOKS.md](../../docs/MCP_DRIVEN_HOOKS.md).
## Providers
+181 -1
View File
@@ -21,6 +21,80 @@ use crate::wire::{self, WireSender};
const ERROR_REFLECTION_SUFFIX: &str =
"\n\n[Reflect] Before retrying, identify the cause and change your approach.";
/// Maximum reply reminders emitted per prompt when `require_reply` is on.
///
/// After this many, the turn is allowed to end whether or not anything was
/// published: the guard exists to catch accidental omission, not to compel
/// speech. The shared `stop_max_rejections` budget can cut this lower — see
/// [`Config::require_reply`](crate::config::Config::require_reply).
const MAX_REPLY_NAGS: u32 = 2;
/// Server label on the synthetic reply-guard objection.
///
/// Not a real MCP server. It rides the same tool-result path as `_Stop` hook
/// output, so the model sees `{hook, server, text}` attribution naming the
/// in-process guard rather than an MCP server that could be impersonated.
const REPLY_GUARD_SERVER: &str = "buzz-agent";
/// Reminder text emitted when a turn is about to end with nothing published.
///
/// Explicitly licenses silence. The base prompt tells agents that publishing is
/// optional and "silence is usually correct"; a reminder that argued otherwise
/// would fight that instruction and make agents chattier.
const REPLY_GUARD_NAG: &str = "You are about to end this turn without calling `buzz messages send`. \
Your assistant text and reasoning are never shown to anyone — if you did work, found an answer, \
or hit a blocker that someone is waiting on, it exists only if you publish it. \
If you already posted, or if silence is genuinely correct for this turn, ignore this and end your turn.";
/// Whether `call` is a recognized attempt to publish a reply to Buzz.
///
/// Recognizes an *attempt*, not a successful publish: the command text is
/// inspected, never the exit status. That is deliberate — a send that fails
/// already returns a non-zero exit and error JSON to the model, which is louder
/// feedback than the reminder this gates.
///
/// `has` + `!is_hook` are the same checks the dispatcher uses to accept a call
/// (see `execute_calls`), so a hallucinated `fake__shell` — rejected at preflight
/// and never executed — cannot disarm the guard. They must stay *before*
/// [`is_reply_shaped`]: together with them, and only with them, the `__shell`
/// suffix is exactly equivalent to "the bare tool name is `shell`".
fn is_buzz_reply_call(call: &ToolCall, mcp: &McpRegistry) -> bool {
mcp.has(&call.name) && !mcp.is_hook(&call.name) && is_reply_shaped(&call.name, &call.arguments)
}
/// Whether a tool name and arguments have the shape of a Buzz publish command.
///
/// Split from [`is_buzz_reply_call`] only so the matcher is testable without a
/// live [`McpRegistry`]; callers must apply the registry checks first.
///
/// On the name: `ends_with("__shell")` is exact rather than approximate *given*
/// those checks. Registration rejects `__` in both server names and bare tool
/// names, and qualified names are `{server}__{bare}`, so a trailing `__shell` can
/// only straddle the separator if the bare name starts with `_` — which `is_hook`
/// already excludes. Dropping the separator would not be exact: `powershell` and
/// `noshell` both end in `shell`.
///
/// On the command: a deliberately coarse substring test, scoped to the structured
/// `command` field so unrelated metadata — a `description` that quotes a send —
/// cannot suppress the guard, and a non-string `command` is rejected rather than
/// coerced. Known limits, both accepted: a command assembled at runtime (`$CMD`)
/// or hidden in a wrapper script is missed, and text that merely quotes a send
/// (`echo "buzz messages send"`) matches. Missing a real post is the expensive
/// direction, and substring matching is the more forgiving one there.
fn is_reply_shaped(name: &str, arguments: &serde_json::Value) -> bool {
name.ends_with("__shell")
&& arguments
.get("command")
.and_then(|v| v.as_str())
.is_some_and(|cmd| {
// `messages send` also covers `messages send-diff`. `reactions
// add` counts because the base prompt directs agents to react
// rather than post a bare acknowledgement, so nagging an agent
// that reacted would punish documented-correct behavior.
cmd.contains("messages send") || cmd.contains("reactions add")
})
}
pub struct RunCtx<'a> {
pub cfg: &'a Config,
/// Effective model for this session. Usually equals `cfg.model`; overridden
@@ -102,6 +176,14 @@ impl RunCtx<'_> {
// session) so a stubborn exchange can't permanently disable the stop
// guard for a long-lived session; `max_rounds` still caps the loop.
let mut stop_rejections = 0u32;
// Reply-guard state for this prompt. `prompt()` *is* the turn, so
// locals here are per-turn by construction — same shape as
// `stop_rejections` above.
//
// Named for what it proves: a *recognized attempt* to publish, not a
// successful publish. See `is_buzz_reply_call`.
let mut buzz_reply_call_seen = false;
let mut reply_nags = 0u32;
loop {
if self.cfg.max_rounds > 0 && round >= self.cfg.max_rounds {
return Ok(StopReason::MaxTurnRequests);
@@ -264,7 +346,7 @@ impl RunCtx<'_> {
if stop_rejections >= self.cfg.stop_max_rejections {
return Ok(stop);
}
let objections = self
let mut objections = self
.mcp
.call_hooks(
"_Stop",
@@ -273,6 +355,17 @@ impl RunCtx<'_> {
&self.cfg.hook_servers,
)
.await;
// Reply guard shares this gate and this budget, so a round
// carrying both a hook objection and a reply reminder costs
// one rejection and delivers both texts.
if self.cfg.require_reply
&& !buzz_reply_call_seen
&& reply_nags < MAX_REPLY_NAGS
{
reply_nags += 1;
objections
.push((REPLY_GUARD_SERVER.to_string(), REPLY_GUARD_NAG.to_string()));
}
if !objections.is_empty() {
stop_rejections = stop_rejections.saturating_add(1);
push_hook_outputs_as_tool_results(self.history, "_Stop", &objections);
@@ -290,6 +383,11 @@ impl RunCtx<'_> {
);
calls.truncate(MAX_TOOL_CALLS_PER_TURN);
}
// Deliberately after truncation: a publish-shaped call that was
// discarded never runs, so it must not suppress the reminder.
if self.cfg.require_reply && !buzz_reply_call_seen {
buzz_reply_call_seen = calls.iter().any(|c| is_buzz_reply_call(c, self.mcp));
}
self.history.push(HistoryItem::Assistant {
text: response.text,
tool_calls: calls.clone(),
@@ -799,6 +897,88 @@ mod tests {
use super::*;
use serde_json::json;
/// The shapes the guard must recognize as a publish attempt. Callers apply
/// the registry checks first; these cover the name suffix and command text.
#[test]
fn reply_shape_matches_documented_send_forms() {
for cmd in [
"buzz messages send --channel X --content Y",
"buzz --relay wss://r messages send --channel X --content Y",
"/abs/path/buzz messages send",
"printf 'hi' | buzz messages send --content -",
"buzz messages send-diff --diff -",
"buzz reactions add --event E --emoji +",
// Assembled through another shell: rev 3's tokenizer missed this.
r#"sh -c "buzz messages send --channel X""#,
] {
assert!(
is_reply_shaped("dev__shell", &json!({ "command": cmd })),
"expected {cmd:?} to count as a publish attempt"
);
}
}
/// Commands that do real work but do not reply in the originating
/// conversation must still be nagged.
#[test]
fn reply_shape_rejects_non_reply_commands() {
for cmd in [
"buzz messages get --channel X",
"buzz channels list",
"buzz reactions remove --event E",
"buzz pr open --title T",
"buzz social publish --content hi",
"buzz notes set --name n",
"cargo test -p buzz-agent",
] {
assert!(
!is_reply_shaped("dev__shell", &json!({ "command": cmd })),
"expected {cmd:?} not to count as a publish attempt"
);
}
}
/// The `__` separator is load-bearing: `ends_with("shell")` alone would
/// accept any registered tool whose name merely ends in those letters, and
/// `has()` proves registration, not the bare name.
#[test]
fn reply_shape_requires_the_qname_separator() {
let args = json!({ "command": "buzz messages send --channel X" });
for name in [
"dev__powershell",
"dev__noshell",
"shell",
"dev__send_message",
] {
assert!(
!is_reply_shaped(name, &args),
"{name} must not satisfy the shell-tool check"
);
}
assert!(is_reply_shaped("dev__shell", &args));
assert!(is_reply_shaped("buzz-dev-mcp__shell", &args));
}
/// Only the field that carries the executable command counts. Searching
/// serialized arguments instead would let arbitrary metadata disarm the
/// guard, turning a description into an attempted send.
#[test]
fn reply_shape_reads_only_the_command_field() {
assert!(!is_reply_shaped(
"dev__shell",
&json!({ "description": "buzz messages send --channel X" })
));
assert!(!is_reply_shaped(
"dev__shell",
&json!({ "workdir": "buzz messages send" })
));
// Malformed `command` is rejected, not coerced — and must not panic.
assert!(!is_reply_shaped("dev__shell", &json!({ "command": 42 })));
assert!(!is_reply_shaped("dev__shell", &json!({ "command": null })));
assert!(!is_reply_shaped("dev__shell", &json!({})));
assert!(!is_reply_shaped("dev__shell", &json!("not an object")));
}
/// A9 regression: `reasoning_details` contributes real bytes to
/// `estimated_bytes` (see `types.rs::HistoryItem::size_with`), so a
/// history item carrying a large opaque reasoning array must actually
+12
View File
@@ -720,6 +720,16 @@ pub struct Config {
/// Maximum `_Stop` rejections per prompt. Default 3. Set to 0 to
/// disable `_Stop` hooks entirely (agent always honors end_turn).
pub stop_max_rejections: u32,
/// Remind the model to publish when a turn is about to end without any
/// recognized attempt to post to Buzz. Default off; opt in per agent with
/// `BUZZ_AGENT_REQUIRE_REPLY=1`.
///
/// Advisory only: at most `MAX_REPLY_NAGS` reminders (see `agent.rs`),
/// then the turn ends regardless. Bounded by the same
/// `stop_max_rejections` budget as `_Stop` hooks, which is the outer cap on
/// all end-turn objections — at the default 3 both reminders fit; at 1 only
/// one does; at 0 the guard is off with the hooks.
pub require_reply: bool,
/// Hook server allowlist. See [`HookServers`] for variant semantics.
/// Default (env unset/empty) is `None` — hooks are off unless the
/// operator explicitly opts in.
@@ -851,6 +861,7 @@ impl Config {
max_parallel_tools: parse_env("BUZZ_AGENT_MAX_PARALLEL_TOOLS", 8usize)?,
hook_timeout: Duration::from_millis(parse_env("BUZZ_AGENT_HOOK_TIMEOUT_MS", 2500u64)?),
stop_max_rejections: parse_env("BUZZ_AGENT_STOP_MAX_REJECTIONS", 3u32)?,
require_reply: parse_env("BUZZ_AGENT_REQUIRE_REPLY", 0u8)? != 0,
hook_servers: parse_hook_servers_env("MCP_HOOK_SERVERS"),
hints_enabled: parse_env("BUZZ_AGENT_NO_HINTS", 0u8)? == 0,
thinking_effort: parse_thinking_effort(env("BUZZ_AGENT_THINKING_EFFORT").as_deref())?,
@@ -893,6 +904,7 @@ impl Config {
max_parallel_tools: 1,
hook_timeout: Duration::from_secs(1),
stop_max_rejections: 0,
require_reply: false,
hook_servers: HookServers::None,
hints_enabled: false,
thinking_effort: None,
+1
View File
@@ -2355,6 +2355,7 @@ mod tests {
max_parallel_tools: 1,
hook_timeout: Duration::from_secs(1),
stop_max_rejections: 0,
require_reply: false,
hook_servers: HookServers::None,
api_key: "key".into(),
model: "model".into(),
+25 -1
View File
@@ -33,6 +33,11 @@
//! — expose a `_PostCompact` hook tool
//! FAKE_MCP_POSTCOMPACT_TEXT=text
//! — `_PostCompact` returns this (default: "")
//! FAKE_MCP_SHELL_TOOL=1 — expose a tool whose bare name is `shell`
//! (registered as `<server>__shell`), taking a
//! `command` string. Lets a test drive the
//! reply guard's recognition of a real,
//! registered shell tool.
use std::io::{BufRead, Write};
@@ -76,6 +81,7 @@ fn make_tools(
desc: &str,
include_stop_hook: bool,
include_post_compact_hook: bool,
include_shell_tool: bool,
) -> Vec<Value> {
let mut tools: Vec<Value> = (0..count)
.map(|i| {
@@ -100,6 +106,17 @@ fn make_tools(
"inputSchema": { "type": "object", "properties": {} },
}));
}
if include_shell_tool {
tools.push(json!({
"name": "shell",
"description": "run a shell command",
"inputSchema": {
"type": "object",
"properties": { "command": { "type": "string" } },
"required": ["command"],
},
}));
}
tools
}
@@ -136,6 +153,7 @@ fn main() {
let stop_count_limit: usize = env_usize("FAKE_MCP_STOP_COUNT", usize::MAX);
let mut stop_calls_seen: usize = 0;
let post_compact_hook = env_flag("FAKE_MCP_POSTCOMPACT_HOOK");
let shell_tool = env_flag("FAKE_MCP_SHELL_TOOL");
let post_compact_text = std::env::var("FAKE_MCP_POSTCOMPACT_TEXT").unwrap_or_default();
// Use a channel-based stdin reader so notifications (which carry no id)
@@ -206,7 +224,13 @@ fn main() {
write_response(
id,
json!({
"tools": make_tools(tool_count, &desc, stop_hook, post_compact_hook)
"tools": make_tools(
tool_count,
&desc,
stop_hook,
post_compact_hook,
shell_tool,
)
}),
);
}
+462
View File
@@ -1819,3 +1819,465 @@ async fn cancel_sends_notifications_cancelled_to_any_mcp_server() {
let _ = std::fs::remove_file(&call_received_marker);
h.shutdown().await;
}
// ---------------------------------------------------------------------------
// Reply guard (`BUZZ_AGENT_REQUIRE_REPLY`)
//
// The guard reminds the model to publish when a turn is about to end without
// any recognized attempt to post to Buzz. It rides the existing `_Stop` gate
// and shares its rejection budget, so most of these tests count LLM calls:
// each reminder costs exactly one extra round.
// ---------------------------------------------------------------------------
/// Number of reply-guard reminders present in one captured LLM request.
///
/// A reminder is a tool-role message whose JSON body is attributed to the
/// in-process guard (`server: "buzz-agent"`) at the `_Stop` hook point — the
/// same lower-trust shape as real hook output.
fn reply_nag_count(request: &Value) -> usize {
request["messages"]
.as_array()
.map(|msgs| {
msgs.iter()
.filter(|m| {
m["role"] == "tool"
&& serde_json::from_str::<Value>(m["content"].as_str().unwrap_or(""))
.map(|p| p["hook"] == "_Stop" && p["server"] == "buzz-agent")
.unwrap_or(false)
})
.count()
})
.unwrap_or(0)
}
/// A publish-shaped call to a real registered shell tool.
fn openai_shell_send(id: &str) -> Value {
openai_tool_call(
id,
"fake__shell",
json!({ "command": "buzz messages send --channel c --content hi" }),
)
}
/// Run one prompt to completion, answering any permission requests, and
/// return the final response.
async fn prompt_to_completion(h: &mut Harness, sid: &str) -> Value {
let p = h
.send(
"session/prompt",
json!({"sessionId": sid, "prompt": [{"type":"text","text":"go"}]}),
)
.await;
loop {
let v = h.recv().await;
if v.get("method") == Some(&json!("session/request_permission")) {
let id = v["id"].clone();
h.write(json!({
"jsonrpc": "2.0",
"id": id,
"result": { "outcome": { "outcome": "selected", "optionId": "allow" } },
}))
.await;
continue;
}
if v["id"] == json!(p) {
return v;
}
}
}
/// Default off: a silent turn ends on the first end_turn with no extra round.
/// This is the invariant that keeps the feature free for everyone who hasn't
/// opted in.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_off_by_default() {
let llm = spawn_capturing_llm(vec![openai_text("done"), openai_text("unexpected")]).await;
let mut h = Harness::spawn(&llm.url).await;
let sid = init_session(&mut h, json!([])).await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
1,
"guard must be inert when unset, got {} LLM calls",
captured.len()
);
h.shutdown().await;
}
/// `BUZZ_AGENT_REQUIRE_REPLY=0` is off too — the toggle is numeric, so a
/// literal `0` must not read as "set, therefore on".
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_explicit_zero_is_off() {
let llm = spawn_capturing_llm(vec![openai_text("done"), openai_text("unexpected")]).await;
let mut h = Harness::spawn_with_env(&llm.url, &[("BUZZ_AGENT_REQUIRE_REPLY", "0")]).await;
let sid = init_session(&mut h, json!([])).await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
1,
"REQUIRE_REPLY=0 must behave as off, got {} LLM calls",
captured.len()
);
h.shutdown().await;
}
/// Opted in and silent: exactly two reminders, then the turn is allowed to
/// end. The guard is advisory — it must never trap a turn.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_nags_twice_then_lets_the_turn_end() {
// Budget defaults to 3, so the cap that stops the loop here is
// MAX_REPLY_NAGS = 2, not the rejection budget.
let llm = spawn_capturing_llm(vec![
openai_text("silent-1"),
openai_text("silent-2"),
openai_text("silent-3"),
openai_text("must-not-be-requested"),
])
.await;
let mut h = Harness::spawn_with_env(&llm.url, &[("BUZZ_AGENT_REQUIRE_REPLY", "1")]).await;
let sid = init_session(&mut h, json!([])).await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
3,
"expected 2 reminders then end_turn (3 LLM calls), got {}",
captured.len()
);
assert_eq!(
reply_nag_count(&captured[0]),
0,
"reminder before any end_turn"
);
assert_eq!(reply_nag_count(&captured[1]), 1);
assert_eq!(reply_nag_count(&captured[2]), 2);
// The reminder must name the command it wants and license silence, so it
// cannot fight the base prompt's "silence is usually correct".
let msgs = captured[2]["messages"].as_array().unwrap();
let nag = msgs
.iter()
.filter_map(|m| serde_json::from_str::<Value>(m["content"].as_str().unwrap_or("")).ok())
.find(|p| p["server"] == "buzz-agent")
.expect("reminder body");
let text = nag["text"].as_str().unwrap_or("");
assert!(
text.contains("buzz messages send"),
"reminder should name the command: {text}"
);
assert!(
text.contains("silence is genuinely correct"),
"reminder must license silence: {text}"
);
h.shutdown().await;
}
/// A real publish attempt through a registered shell tool satisfies the guard:
/// no reminder, no extra round.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_satisfied_by_registered_shell_send() {
let llm = spawn_capturing_llm(vec![
openai_shell_send("tc1"),
openai_text("posted"),
openai_text("must-not-be-requested"),
])
.await;
let mut h = Harness::spawn_with_env(&llm.url, &[("BUZZ_AGENT_REQUIRE_REPLY", "1")]).await;
let sid = init_session_with_fake_mcp(
&mut h,
&[("FAKE_MCP_TOOL_COUNT", "1"), ("FAKE_MCP_SHELL_TOOL", "1")],
)
.await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
2,
"a recognized send must not be nagged, got {} LLM calls",
captured.len()
);
assert_eq!(reply_nag_count(&captured[1]), 0);
h.shutdown().await;
}
/// A publish-shaped call to a shell tool that is *not registered* never runs —
/// preflight rejects it — so it must not disarm the guard. This is what the
/// `has`/`is_hook` checks in the predicate buy.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_ignores_unregistered_shell_tool() {
// FAKE_MCP_SHELL_TOOL is absent, so `fake__shell` is a hallucination.
let llm = spawn_capturing_llm(vec![
openai_shell_send("tc1"),
openai_text("silent-1"),
openai_text("silent-2"),
])
.await;
let mut h = Harness::spawn_with_env(
&llm.url,
&[
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "1"),
],
)
.await;
let sid = init_session_with_fake_mcp(&mut h, &[("FAKE_MCP_TOOL_COUNT", "1")]).await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
3,
"expected the hallucinated call to still be nagged, got {} LLM calls",
captured.len()
);
let msgs = captured[1]["messages"].as_array().unwrap();
assert!(
msgs.iter()
.any(|m| m["role"] == "tool"
&& m["content"].as_str().unwrap_or("").contains("unknown tool")),
"expected preflight to reject the call: {msgs:?}"
);
assert_eq!(reply_nag_count(&captured[2]), 1);
h.shutdown().await;
}
/// A publish-shaped call discarded by the per-turn tool-call cap never runs,
/// so it must not suppress the reminder either. Pins the check's placement
/// after `calls.truncate(MAX_TOOL_CALLS_PER_TURN)`.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_ignores_calls_lost_to_the_turn_cap() {
// 64 filler calls (the cap) followed by the publish attempt, which is
// therefore truncated away. The shell tool *is* registered here, so only
// the placement — not tool identity — can explain the reminder.
let mut calls: Vec<Value> = (0..64)
.map(|i| {
json!({
"id": format!("c{i}"),
"type": "function",
"function": { "name": "fake__tool_0", "arguments": "{}" },
})
})
.collect();
calls.push(json!({
"id": "c-send",
"type": "function",
"function": {
"name": "fake__shell",
"arguments": json!({ "command": "buzz messages send --channel c --content hi" })
.to_string(),
},
}));
let truncated_send = json!({
"id": "cc-trunc", "object": "chat.completion", "model": "fake-model",
"choices": [{
"index": 0,
"message": { "role": "assistant", "content": null, "tool_calls": calls },
"finish_reason": "tool_calls",
}],
});
let llm = spawn_capturing_llm(vec![
truncated_send,
openai_text("silent-1"),
openai_text("silent-2"),
])
.await;
let mut h = Harness::spawn_with_env(
&llm.url,
&[
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "1"),
],
)
.await;
let sid = init_session_with_fake_mcp(
&mut h,
&[("FAKE_MCP_TOOL_COUNT", "1"), ("FAKE_MCP_SHELL_TOOL", "1")],
)
.await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
3,
"a truncated send must still be nagged, got {} LLM calls",
captured.len()
);
assert_eq!(reply_nag_count(&captured[2]), 1);
h.shutdown().await;
}
/// The shared `_Stop` rejection budget is the outer cap: at 1 the guard gets
/// one reminder instead of two. Documented degradation, not a bug.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_bounded_by_stop_rejection_budget() {
let llm = spawn_capturing_llm(vec![
openai_text("silent-1"),
openai_text("silent-2"),
openai_text("must-not-be-requested"),
])
.await;
let mut h = Harness::spawn_with_env(
&llm.url,
&[
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "1"),
],
)
.await;
let sid = init_session(&mut h, json!([])).await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
2,
"budget 1 must allow exactly one reminder, got {} LLM calls",
captured.len()
);
assert_eq!(reply_nag_count(&captured[1]), 1);
h.shutdown().await;
}
/// Budget 0 disables every objection at the gate, including this one.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_off_when_stop_budget_is_zero() {
let llm = spawn_capturing_llm(vec![openai_text("done"), openai_text("unexpected")]).await;
let mut h = Harness::spawn_with_env(
&llm.url,
&[
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "0"),
],
)
.await;
let sid = init_session(&mut h, json!([])).await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
1,
"budget 0 must disable the guard, got {} LLM calls",
captured.len()
);
h.shutdown().await;
}
/// The two axes are independent inside one shared budget: a round carrying
/// both a `_Stop` hook objection and a reminder costs one rejection and
/// delivers both texts, and once the reminders are spent the hook objection
/// continues alone.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn reply_guard_combines_with_stop_hook_objection() {
// The hook objects on its first 3 calls, then clears. Reminders stop
// after 2, so round 3 must carry the hook text and no new reminder.
let llm = spawn_capturing_llm(vec![
openai_text("silent-1"),
openai_text("silent-2"),
openai_text("silent-3"),
openai_text("silent-4"),
])
.await;
let mut h = Harness::spawn_with_env(
&llm.url,
&[
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
("MCP_HOOK_SERVERS", "fake"),
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "10"),
],
)
.await;
let sid = init_session_with_fake_mcp(
&mut h,
&[
("FAKE_MCP_TOOL_COUNT", "1"),
("FAKE_MCP_STOP_HOOK", "1"),
("FAKE_MCP_STOP_TEXT", "you have open todos"),
("FAKE_MCP_STOP_COUNT", "3"),
],
)
.await;
let r = prompt_to_completion(&mut h, &sid).await;
assert_eq!(r["result"]["stopReason"], "end_turn");
let captured = llm.captured.lock().await;
assert_eq!(
captured.len(),
4,
"expected 3 objecting rounds then a clear end, got {}",
captured.len()
);
let hook_objections = |req: &Value| -> usize {
req["messages"]
.as_array()
.map(|msgs| {
msgs.iter()
.filter(|m| {
m["content"]
.as_str()
.unwrap_or("")
.contains("you have open todos")
})
.count()
})
.unwrap_or(0)
};
// Round 2 carries one of each — a single rejection bought both texts.
assert_eq!(reply_nag_count(&captured[1]), 1);
assert_eq!(hook_objections(&captured[1]), 1);
// Round 4: the hook objected three times, the guard only twice.
assert_eq!(reply_nag_count(&captured[3]), 2);
assert_eq!(hook_objections(&captured[3]), 3);
h.shutdown().await;
}
/// An unparseable toggle is a startup error, not a silent default. `parse_env`
/// is generic over `FromStr`, so this also pins the numeric type: a `bool`
/// field would have rejected the documented `1`.
#[test]
fn reply_guard_rejects_unparseable_toggle() {
let out = std::process::Command::new(env!("CARGO_BIN_EXE_buzz-agent"))
.env("BUZZ_AGENT_PROVIDER", "openai")
.env("OPENAI_COMPAT_API_KEY", "test")
.env("OPENAI_COMPAT_MODEL", "fake-model")
.env("BUZZ_AGENT_REQUIRE_REPLY", "true")
.stdin(Stdio::null())
.output()
.expect("run buzz-agent");
assert!(
!out.status.success(),
"expected a config error exit, got {:?}",
out.status
);
let stderr = String::from_utf8_lossy(&out.stderr);
assert!(
stderr.contains("BUZZ_AGENT_REQUIRE_REPLY"),
"expected the offending key in the error, got: {stderr}"
);
}
@@ -47,6 +47,13 @@ pub fn apply_relay_mesh_env(
// may deliberately choose a smaller cap or a different effort. This function
// runs after those layers during readiness, so never clobber their values.
insert_default_if_unset(env, "BUZZ_AGENT_MAX_OUTPUT_TOKENS", "4096");
// Mesh agents run on small local models, which are the ones most likely to
// do the work and then end the turn without publishing it — the failure the
// reply guard exists to catch. Everywhere else it stays opt-in and unset.
// A default, not policy: an explicit `0` from the agent/persona/global env
// survives (see `insert_default_if_unset`, and the copy-forward list in
// `relay_mesh_process_env` that preserves it through the spawn path).
insert_default_if_unset(env, "BUZZ_AGENT_REQUIRE_REPLY", "1");
// Deliberately no BUZZ_AGENT_THINKING_EFFORT default: mesh translates
// `reasoning_effort` into the chat template's `enable_thinking` flag, so any
// value we pick overrides each model's own template default — and the right
@@ -80,7 +87,15 @@ pub fn relay_mesh_process_env(
model: &str,
) -> std::collections::BTreeMap<String, String> {
let mut env = std::collections::BTreeMap::new();
for key in ["BUZZ_AGENT_MAX_OUTPUT_TOKENS", "BUZZ_AGENT_THINKING_EFFORT"] {
for key in [
"BUZZ_AGENT_MAX_OUTPUT_TOKENS",
"BUZZ_AGENT_THINKING_EFFORT",
// Must be copied forward for the user's value to survive: this map is
// written onto the command *after* the layered user env, so a key absent
// here is re-defaulted by `apply_relay_mesh_env` below and an explicit
// `BUZZ_AGENT_REQUIRE_REPLY=0` would be silently overridden back to `1`.
"BUZZ_AGENT_REQUIRE_REPLY",
] {
if let Some(value) = effective_env.get(key) {
env.insert(key.to_string(), value.clone());
}
@@ -145,6 +160,78 @@ mod tests {
);
}
#[test]
fn native_provider_enables_reply_guard_by_default() {
let mut env = BTreeMap::new();
apply_relay_mesh_env(
&mut env,
Some(RELAY_MESH_PROVIDER_ID),
Some(RELAY_MESH_AUTO_MODEL_ID),
);
assert_eq!(
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
Some("1"),
"mesh agents opt into the reply guard automatically"
);
}
#[test]
fn native_provider_preserves_explicit_reply_guard_opt_out() {
let mut env = BTreeMap::from([("BUZZ_AGENT_REQUIRE_REPLY".to_string(), "0".to_string())]);
apply_relay_mesh_env(
&mut env,
Some(RELAY_MESH_PROVIDER_ID),
Some(RELAY_MESH_AUTO_MODEL_ID),
);
assert_eq!(
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
Some("0"),
"an explicit opt-out is a user decision, not a value to re-default"
);
}
#[test]
fn non_mesh_provider_leaves_reply_guard_unset() {
let mut env = BTreeMap::new();
apply_relay_mesh_env(&mut env, Some("anthropic"), Some("claude-haiku-4.5"));
assert_eq!(
env.get("BUZZ_AGENT_REQUIRE_REPLY"),
None,
"the guard stays opt-in everywhere except mesh"
);
assert!(env.is_empty(), "non-mesh providers get no mesh env at all");
}
/// The spawn path writes this map onto the command *after* the layered user
/// env, so an explicit opt-out only survives if it is copied forward. Without
/// the copy-forward, `apply_relay_mesh_env` re-defaults it to `1` here and
/// silently overrides the user at spawn while readiness still shows `0`.
#[test]
fn process_env_preserves_explicit_reply_guard_opt_out() {
let effective_env =
BTreeMap::from([("BUZZ_AGENT_REQUIRE_REPLY".to_string(), "0".to_string())]);
let env = relay_mesh_process_env(&effective_env, "Gemma-4");
assert_eq!(
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
Some("0")
);
}
#[test]
fn process_env_enables_reply_guard_when_user_is_silent() {
let env = relay_mesh_process_env(&BTreeMap::new(), "Gemma-4");
assert_eq!(
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
Some("1")
);
}
#[test]
fn process_env_seeds_controls_without_restoring_unrelated_credentials() {
let effective_env = BTreeMap::from([
+17
View File
@@ -65,6 +65,23 @@ These constraints ensure a buggy or malicious hook cannot trap the agent.
Hooks are **off by default**. The operator must explicitly opt in via
`MCP_HOOK_SERVERS`.
### Not a hook: the reply guard
`buzz-agent` has one in-process objection at the `_Stop` gate that is **not** an
MCP hook and exposes no hook tool: the reply guard
(`BUZZ_AGENT_REQUIRE_REPLY=1`), which reminds the model to publish when a turn is
about to end with nothing posted to Buzz. There is no `_ReplyGuard` tool to
implement and no server to allowlist — the env var and the recognition contract
are documented in
[crates/buzz-agent/README.md](../crates/buzz-agent/README.md#reply-guard).
It is mentioned here only because it shares this lifecycle point and this
budget: its reminders count against `BUZZ_AGENT_STOP_MAX_REJECTIONS` like any
hook objection, and a round carrying both a hook objection and a reminder costs
one rejection and delivers both texts. Setting the budget to 0 disables both.
That the gate can carry in-process objections alongside hook output is
deliberate; hooks see no difference.
## Implementing a Hook
Any MCP server can expose hooks. Example: a test-runner server that blocks