mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
feat(agent): optional reply guard reminds a silent turn to publish (#3763)
## Why
A Buzz agent's assistant text and reasoning are never shown to anyone —
only what it posts through the CLI. A turn that runs fifteen tool calls
and never publishes is a silent failure: the requester waits on a result
that was produced and thrown away.
This adds an optional reminder at the end-of-turn gate, off by default.
Tyler asked for it in buzz-mesh; plan iterated to **9.5/10 with @Wren**
(Minimalness 9.7, Elegance 9.5, Correctness 9.3).
## What
`BUZZ_AGENT_REQUIRE_REPLY=1` (default off, per-agent opt-in). A turn
about to end with no recognized attempt to post gets a reminder and is
rerolled. **At most two, then the turn ends regardless** — the guard
catches accidental omission, it does not compel speech. The reminder
text explicitly licenses silence so it cannot fight the base prompt's
"silence is usually correct."
**This is not a new MCP hook.** `RunCtx::run` *is* the turn, so the two
per-turn locals need no plumbing, and every tool call already passes
through it with arguments visible. The objection is appended at the
existing `_Stop` gate and rides `push_hook_outputs_as_tool_results`, so
the model receives it as a lower-trust tool result with `{hook, server,
text}` attribution. No new trust path, no new lifecycle event, no
dev-mcp or CLI protocol change.
Earlier revisions of this plan needed four crates (a `_UserPromptSubmit`
hook, a marker file, a `buzz-cli` change, dev-mcp state). Tyler pointed
out the agent already knows both facts; that deleted all of it. Net
runtime change is ~35 lines in `agent.rs` + ~4 in `config.rs`.
### Recognition contract
A registered non-hook tool whose qualified name ends in `__shell`, whose
`command` argument contains `messages send` or `reactions add`.
- **The `__` separator is exact, not approximate.** Given `has()` +
`!is_hook()`, `ends_with("__shell")` is *provably equivalent* to a bare
name of `shell`: registration forbids `__` in server and bare names
(`mcp.rs:227,268`) and qnames are `{server}__{bare}`, so a trailing
`__shell` could only straddle the separator if the bare name began with
`_` — which `is_hook` excludes. Without the separator, `powershell` and
`noshell` would match.
- **Reads the structured `command` field**, not serialized arguments, so
a `description` that quotes a send cannot disarm the guard, and a
non-string `command` is rejected rather than coerced.
- **Detects an attempt, not a successful publish.** A failed send
already returns non-zero exit and error JSON — louder than this
reminder. The variable is named `buzz_reply_call_seen` so the code can't
pretend otherwise.
- **Checked after the per-turn tool-call cap**, since a discarded call
never ran.
- `messages send` also covers `messages send-diff`. Reactions count
because the base prompt directs agents to react rather than post a bare
acknowledgement.
**Known limits, both deliberate and documented:** a command assembled at
runtime (`$CMD`) or hidden in a wrapper script is missed; text that
merely quotes a send (`echo "buzz messages send"`) matches. Missing a
real post is the expensive direction and substring matching is the
forgiving one there. Neither edge is pinned by a test, so the matcher
stays free to improve.
### Budget
Reminders share `BUZZ_AGENT_STOP_MAX_REJECTIONS`, the existing outer cap
on every end-turn objection. Default 3 fits both; at 1 only one fits; at
0 the guard is off with the hooks. A round carrying both a hook
objection and a reminder costs one rejection and delivers both texts. An
independent budget would either violate that bound or need a second
arbitration rule.
## Prior art
- **#3467** (closed) built the same detector one layer up in `buzz-acp`
for a different remedy. None of its symbols are on main — this borrows
its permission to be coarse, but reads structured data that ACP didn't
have.
- **#3648** (open) detects turns with *no output at all*; a turn with
fifteen tool calls and no post counts as output there, so it does not
cover this case.
- **#3741** (merged) is mesh-only.
## Testing
**14 new tests.** 4 unit tests on the matcher; 10 integration tests
through the ACP wire harness: off by default, `=0` still off, opted-in
silent → exactly 2 reminders then `end_turn`, registered `fake__shell`
send → 0 reminders, hallucinated `fake__shell` → still reminded, publish
call truncated past the 64-call cap → still reminded, budget 1 → 1
reminder, budget 0 → off, combined `_Stop` hook objection + reminder →
one round both texts and after 2 reminders the hook objection continues
alone, unparseable `=true` → startup error naming the key.
**10 mutation checks, each breaking a specific named test** — neutralize
the nag cap, stop sharing the budget, neutralize `buzz_reply_call_seen`,
drop `has`/`is_hook`, ignore the flag, drop the `__`, drop `reactions
add`, read serialized args, move detection before truncation.
`tests/bin/fake_mcp.rs` gains `FAKE_MCP_SHELL_TOOL=1`: it previously
exposed no tool with a bare name of `shell`, so the satisfied-guard path
was untestable.
Full `cargo test -p buzz-agent` green at 9e0ae1f04; clippy `-D warnings`
and `cargo fmt --check` clean.
**Unrelated flake found:**
`cancelled_turn_with_usage_emits_notification_before_response`
(`tests/fake_llm.rs`) is timing-sensitive. Under 10 loaded cores it
fails **2/20 on this branch and 1/20 at unmodified
`origin/main@02be413b8`** — pre-existing, not caused by this change
(which is inert without the env var). Flagging so it isn't misattributed
to the next PR that's open when CI hits it.
## Docs
`crates/buzz-agent/README.md` is the primary home (env var, recognition
contract, limits, budget interaction). `docs/MCP_DRIVEN_HOOKS.md` gets a
short cross-reference explaining this is *not* a hook — otherwise
readers hunt for a `_ReplyGuard` tool that doesn't exist.
---------
Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
Co-authored-by: Dawn (sprout agent) <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
This commit is contained in:
co-authored by
Dawn
npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta
parent
d48b0e0eec
commit
081f805d5e
@@ -163,6 +163,67 @@ Everything is environment variables. No flags, no config files. (We are a subpro
|
||||
| `BUZZ_AGENT_MAX_LINE_BYTES` | `4194304` | 4 MiB. Hard cap on inbound JSON-RPC frames. |
|
||||
| `BUZZ_AGENT_MAX_HISTORY_BYTES` | `1048576` | 1 MiB. Old turns are evicted past this. |
|
||||
| `BUZZ_AGENT_MAX_TOOL_RESULT_TEXT_BYTES` | `51200` | 50 KiB. Per-result cap on tool-output text; oversize is middle-elided (head + tail kept) with an inline marker. Images are exempt. |
|
||||
| `BUZZ_AGENT_REQUIRE_REPLY` | `0` (`1` on mesh) | `1` enables the [reply guard](#reply-guard) — remind the model to publish when a turn is about to end with nothing posted to Buzz. Desktop defaults it to `1` for Buzz shared-compute agents. |
|
||||
|
||||
|
||||
## Reply Guard
|
||||
|
||||
Off by default, except on Buzz shared-compute (mesh) agents, where Buzz Desktop
|
||||
sets `BUZZ_AGENT_REQUIRE_REPLY=1` automatically. With it enabled, a turn that is
|
||||
about to end without any recognized attempt to post to Buzz gets a reminder that
|
||||
its assistant text is invisible to humans, and is rerolled.
|
||||
|
||||
This exists because a Buzz agent's reasoning and tool output are not shown to
|
||||
anyone. A turn that does real work and never posts is a silent failure — the
|
||||
requester waits on a result that was produced and thrown away.
|
||||
|
||||
Mesh agents get it by default because they run on small local models, which are
|
||||
the ones most likely to do the work and then end the turn without publishing it.
|
||||
Setting `BUZZ_AGENT_REQUIRE_REPLY=0` on the agent, persona, or global env opts a
|
||||
mesh agent back out; the default never overrides an explicit value.
|
||||
|
||||
**Advisory, never a trap.** At most two reminders, then the turn ends whether or
|
||||
not anything was published. The guard catches accidental omission; it does not
|
||||
compel speech. The reminder text explicitly licenses silence, because the
|
||||
built-in system prompt says publishing is optional and silence is often the
|
||||
correct outcome.
|
||||
|
||||
**Recognition contract.** A turn counts as having replied when it issues a call
|
||||
that:
|
||||
|
||||
- resolves to a registered, non-hook tool (a hallucinated tool name is rejected
|
||||
at preflight and never runs, so it must not disarm the guard),
|
||||
- whose qualified name ends in `__shell` — i.e. the bare tool name is exactly
|
||||
`shell`, which is `buzz-dev-mcp`'s shell tool and any other server's, and
|
||||
- whose `command` argument contains `messages send` or `reactions add`.
|
||||
|
||||
`messages send` also covers `messages send-diff`. Reactions count because the
|
||||
built-in prompt directs agents to react rather than post a bare
|
||||
acknowledgement, so nagging an agent that reacted would punish documented
|
||||
behavior.
|
||||
|
||||
Detection is checked **after** the per-turn tool-call cap
|
||||
(`MAX_TOOL_CALLS_PER_TURN`) is applied: a publish-shaped call that was discarded
|
||||
never ran.
|
||||
|
||||
**It recognizes an attempt, not a successful publish.** Only the command text is
|
||||
inspected, never the exit status. A send that fails still satisfies the guard —
|
||||
which is fine, since a failed send already returns a non-zero exit and error
|
||||
JSON to the model, louder feedback than a reminder.
|
||||
|
||||
**Known limits**, both deliberate. A command assembled at runtime (`$CMD`) or
|
||||
buried in a wrapper script is missed, so that turn is reminded despite having
|
||||
posted. Text that merely quotes a send (`echo "buzz messages send"`) matches, so
|
||||
that turn is not reminded. Missing a real post is the expensive direction, and
|
||||
substring matching is the forgiving one there. Neither edge is pinned by a test;
|
||||
the matcher is free to improve.
|
||||
|
||||
**Budget.** Reminders ride the existing `_Stop` gate and share
|
||||
`BUZZ_AGENT_STOP_MAX_REJECTIONS` — the outer cap on every end-turn objection.
|
||||
At the default 3 both reminders fit; at 1 only one does; at 0 the guard is off
|
||||
along with the hooks. A round carrying both a `_Stop` hook objection and a
|
||||
reminder costs one rejection and delivers both texts. This is not a new
|
||||
lifecycle hook — see [MCP_DRIVEN_HOOKS.md](../../docs/MCP_DRIVEN_HOOKS.md).
|
||||
|
||||
|
||||
## Providers
|
||||
|
||||
@@ -21,6 +21,80 @@ use crate::wire::{self, WireSender};
|
||||
const ERROR_REFLECTION_SUFFIX: &str =
|
||||
"\n\n[Reflect] Before retrying, identify the cause and change your approach.";
|
||||
|
||||
/// Maximum reply reminders emitted per prompt when `require_reply` is on.
|
||||
///
|
||||
/// After this many, the turn is allowed to end whether or not anything was
|
||||
/// published: the guard exists to catch accidental omission, not to compel
|
||||
/// speech. The shared `stop_max_rejections` budget can cut this lower — see
|
||||
/// [`Config::require_reply`](crate::config::Config::require_reply).
|
||||
const MAX_REPLY_NAGS: u32 = 2;
|
||||
|
||||
/// Server label on the synthetic reply-guard objection.
|
||||
///
|
||||
/// Not a real MCP server. It rides the same tool-result path as `_Stop` hook
|
||||
/// output, so the model sees `{hook, server, text}` attribution naming the
|
||||
/// in-process guard rather than an MCP server that could be impersonated.
|
||||
const REPLY_GUARD_SERVER: &str = "buzz-agent";
|
||||
|
||||
/// Reminder text emitted when a turn is about to end with nothing published.
|
||||
///
|
||||
/// Explicitly licenses silence. The base prompt tells agents that publishing is
|
||||
/// optional and "silence is usually correct"; a reminder that argued otherwise
|
||||
/// would fight that instruction and make agents chattier.
|
||||
const REPLY_GUARD_NAG: &str = "You are about to end this turn without calling `buzz messages send`. \
|
||||
Your assistant text and reasoning are never shown to anyone — if you did work, found an answer, \
|
||||
or hit a blocker that someone is waiting on, it exists only if you publish it. \
|
||||
If you already posted, or if silence is genuinely correct for this turn, ignore this and end your turn.";
|
||||
|
||||
/// Whether `call` is a recognized attempt to publish a reply to Buzz.
|
||||
///
|
||||
/// Recognizes an *attempt*, not a successful publish: the command text is
|
||||
/// inspected, never the exit status. That is deliberate — a send that fails
|
||||
/// already returns a non-zero exit and error JSON to the model, which is louder
|
||||
/// feedback than the reminder this gates.
|
||||
///
|
||||
/// `has` + `!is_hook` are the same checks the dispatcher uses to accept a call
|
||||
/// (see `execute_calls`), so a hallucinated `fake__shell` — rejected at preflight
|
||||
/// and never executed — cannot disarm the guard. They must stay *before*
|
||||
/// [`is_reply_shaped`]: together with them, and only with them, the `__shell`
|
||||
/// suffix is exactly equivalent to "the bare tool name is `shell`".
|
||||
fn is_buzz_reply_call(call: &ToolCall, mcp: &McpRegistry) -> bool {
|
||||
mcp.has(&call.name) && !mcp.is_hook(&call.name) && is_reply_shaped(&call.name, &call.arguments)
|
||||
}
|
||||
|
||||
/// Whether a tool name and arguments have the shape of a Buzz publish command.
|
||||
///
|
||||
/// Split from [`is_buzz_reply_call`] only so the matcher is testable without a
|
||||
/// live [`McpRegistry`]; callers must apply the registry checks first.
|
||||
///
|
||||
/// On the name: `ends_with("__shell")` is exact rather than approximate *given*
|
||||
/// those checks. Registration rejects `__` in both server names and bare tool
|
||||
/// names, and qualified names are `{server}__{bare}`, so a trailing `__shell` can
|
||||
/// only straddle the separator if the bare name starts with `_` — which `is_hook`
|
||||
/// already excludes. Dropping the separator would not be exact: `powershell` and
|
||||
/// `noshell` both end in `shell`.
|
||||
///
|
||||
/// On the command: a deliberately coarse substring test, scoped to the structured
|
||||
/// `command` field so unrelated metadata — a `description` that quotes a send —
|
||||
/// cannot suppress the guard, and a non-string `command` is rejected rather than
|
||||
/// coerced. Known limits, both accepted: a command assembled at runtime (`$CMD`)
|
||||
/// or hidden in a wrapper script is missed, and text that merely quotes a send
|
||||
/// (`echo "buzz messages send"`) matches. Missing a real post is the expensive
|
||||
/// direction, and substring matching is the more forgiving one there.
|
||||
fn is_reply_shaped(name: &str, arguments: &serde_json::Value) -> bool {
|
||||
name.ends_with("__shell")
|
||||
&& arguments
|
||||
.get("command")
|
||||
.and_then(|v| v.as_str())
|
||||
.is_some_and(|cmd| {
|
||||
// `messages send` also covers `messages send-diff`. `reactions
|
||||
// add` counts because the base prompt directs agents to react
|
||||
// rather than post a bare acknowledgement, so nagging an agent
|
||||
// that reacted would punish documented-correct behavior.
|
||||
cmd.contains("messages send") || cmd.contains("reactions add")
|
||||
})
|
||||
}
|
||||
|
||||
pub struct RunCtx<'a> {
|
||||
pub cfg: &'a Config,
|
||||
/// Effective model for this session. Usually equals `cfg.model`; overridden
|
||||
@@ -102,6 +176,14 @@ impl RunCtx<'_> {
|
||||
// session) so a stubborn exchange can't permanently disable the stop
|
||||
// guard for a long-lived session; `max_rounds` still caps the loop.
|
||||
let mut stop_rejections = 0u32;
|
||||
// Reply-guard state for this prompt. `prompt()` *is* the turn, so
|
||||
// locals here are per-turn by construction — same shape as
|
||||
// `stop_rejections` above.
|
||||
//
|
||||
// Named for what it proves: a *recognized attempt* to publish, not a
|
||||
// successful publish. See `is_buzz_reply_call`.
|
||||
let mut buzz_reply_call_seen = false;
|
||||
let mut reply_nags = 0u32;
|
||||
loop {
|
||||
if self.cfg.max_rounds > 0 && round >= self.cfg.max_rounds {
|
||||
return Ok(StopReason::MaxTurnRequests);
|
||||
@@ -264,7 +346,7 @@ impl RunCtx<'_> {
|
||||
if stop_rejections >= self.cfg.stop_max_rejections {
|
||||
return Ok(stop);
|
||||
}
|
||||
let objections = self
|
||||
let mut objections = self
|
||||
.mcp
|
||||
.call_hooks(
|
||||
"_Stop",
|
||||
@@ -273,6 +355,17 @@ impl RunCtx<'_> {
|
||||
&self.cfg.hook_servers,
|
||||
)
|
||||
.await;
|
||||
// Reply guard shares this gate and this budget, so a round
|
||||
// carrying both a hook objection and a reply reminder costs
|
||||
// one rejection and delivers both texts.
|
||||
if self.cfg.require_reply
|
||||
&& !buzz_reply_call_seen
|
||||
&& reply_nags < MAX_REPLY_NAGS
|
||||
{
|
||||
reply_nags += 1;
|
||||
objections
|
||||
.push((REPLY_GUARD_SERVER.to_string(), REPLY_GUARD_NAG.to_string()));
|
||||
}
|
||||
if !objections.is_empty() {
|
||||
stop_rejections = stop_rejections.saturating_add(1);
|
||||
push_hook_outputs_as_tool_results(self.history, "_Stop", &objections);
|
||||
@@ -290,6 +383,11 @@ impl RunCtx<'_> {
|
||||
);
|
||||
calls.truncate(MAX_TOOL_CALLS_PER_TURN);
|
||||
}
|
||||
// Deliberately after truncation: a publish-shaped call that was
|
||||
// discarded never runs, so it must not suppress the reminder.
|
||||
if self.cfg.require_reply && !buzz_reply_call_seen {
|
||||
buzz_reply_call_seen = calls.iter().any(|c| is_buzz_reply_call(c, self.mcp));
|
||||
}
|
||||
self.history.push(HistoryItem::Assistant {
|
||||
text: response.text,
|
||||
tool_calls: calls.clone(),
|
||||
@@ -799,6 +897,88 @@ mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
/// The shapes the guard must recognize as a publish attempt. Callers apply
|
||||
/// the registry checks first; these cover the name suffix and command text.
|
||||
#[test]
|
||||
fn reply_shape_matches_documented_send_forms() {
|
||||
for cmd in [
|
||||
"buzz messages send --channel X --content Y",
|
||||
"buzz --relay wss://r messages send --channel X --content Y",
|
||||
"/abs/path/buzz messages send",
|
||||
"printf 'hi' | buzz messages send --content -",
|
||||
"buzz messages send-diff --diff -",
|
||||
"buzz reactions add --event E --emoji +",
|
||||
// Assembled through another shell: rev 3's tokenizer missed this.
|
||||
r#"sh -c "buzz messages send --channel X""#,
|
||||
] {
|
||||
assert!(
|
||||
is_reply_shaped("dev__shell", &json!({ "command": cmd })),
|
||||
"expected {cmd:?} to count as a publish attempt"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Commands that do real work but do not reply in the originating
|
||||
/// conversation must still be nagged.
|
||||
#[test]
|
||||
fn reply_shape_rejects_non_reply_commands() {
|
||||
for cmd in [
|
||||
"buzz messages get --channel X",
|
||||
"buzz channels list",
|
||||
"buzz reactions remove --event E",
|
||||
"buzz pr open --title T",
|
||||
"buzz social publish --content hi",
|
||||
"buzz notes set --name n",
|
||||
"cargo test -p buzz-agent",
|
||||
] {
|
||||
assert!(
|
||||
!is_reply_shaped("dev__shell", &json!({ "command": cmd })),
|
||||
"expected {cmd:?} not to count as a publish attempt"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The `__` separator is load-bearing: `ends_with("shell")` alone would
|
||||
/// accept any registered tool whose name merely ends in those letters, and
|
||||
/// `has()` proves registration, not the bare name.
|
||||
#[test]
|
||||
fn reply_shape_requires_the_qname_separator() {
|
||||
let args = json!({ "command": "buzz messages send --channel X" });
|
||||
for name in [
|
||||
"dev__powershell",
|
||||
"dev__noshell",
|
||||
"shell",
|
||||
"dev__send_message",
|
||||
] {
|
||||
assert!(
|
||||
!is_reply_shaped(name, &args),
|
||||
"{name} must not satisfy the shell-tool check"
|
||||
);
|
||||
}
|
||||
assert!(is_reply_shaped("dev__shell", &args));
|
||||
assert!(is_reply_shaped("buzz-dev-mcp__shell", &args));
|
||||
}
|
||||
|
||||
/// Only the field that carries the executable command counts. Searching
|
||||
/// serialized arguments instead would let arbitrary metadata disarm the
|
||||
/// guard, turning a description into an attempted send.
|
||||
#[test]
|
||||
fn reply_shape_reads_only_the_command_field() {
|
||||
assert!(!is_reply_shaped(
|
||||
"dev__shell",
|
||||
&json!({ "description": "buzz messages send --channel X" })
|
||||
));
|
||||
assert!(!is_reply_shaped(
|
||||
"dev__shell",
|
||||
&json!({ "workdir": "buzz messages send" })
|
||||
));
|
||||
// Malformed `command` is rejected, not coerced — and must not panic.
|
||||
assert!(!is_reply_shaped("dev__shell", &json!({ "command": 42 })));
|
||||
assert!(!is_reply_shaped("dev__shell", &json!({ "command": null })));
|
||||
assert!(!is_reply_shaped("dev__shell", &json!({})));
|
||||
assert!(!is_reply_shaped("dev__shell", &json!("not an object")));
|
||||
}
|
||||
|
||||
/// A9 regression: `reasoning_details` contributes real bytes to
|
||||
/// `estimated_bytes` (see `types.rs::HistoryItem::size_with`), so a
|
||||
/// history item carrying a large opaque reasoning array must actually
|
||||
|
||||
@@ -720,6 +720,16 @@ pub struct Config {
|
||||
/// Maximum `_Stop` rejections per prompt. Default 3. Set to 0 to
|
||||
/// disable `_Stop` hooks entirely (agent always honors end_turn).
|
||||
pub stop_max_rejections: u32,
|
||||
/// Remind the model to publish when a turn is about to end without any
|
||||
/// recognized attempt to post to Buzz. Default off; opt in per agent with
|
||||
/// `BUZZ_AGENT_REQUIRE_REPLY=1`.
|
||||
///
|
||||
/// Advisory only: at most `MAX_REPLY_NAGS` reminders (see `agent.rs`),
|
||||
/// then the turn ends regardless. Bounded by the same
|
||||
/// `stop_max_rejections` budget as `_Stop` hooks, which is the outer cap on
|
||||
/// all end-turn objections — at the default 3 both reminders fit; at 1 only
|
||||
/// one does; at 0 the guard is off with the hooks.
|
||||
pub require_reply: bool,
|
||||
/// Hook server allowlist. See [`HookServers`] for variant semantics.
|
||||
/// Default (env unset/empty) is `None` — hooks are off unless the
|
||||
/// operator explicitly opts in.
|
||||
@@ -851,6 +861,7 @@ impl Config {
|
||||
max_parallel_tools: parse_env("BUZZ_AGENT_MAX_PARALLEL_TOOLS", 8usize)?,
|
||||
hook_timeout: Duration::from_millis(parse_env("BUZZ_AGENT_HOOK_TIMEOUT_MS", 2500u64)?),
|
||||
stop_max_rejections: parse_env("BUZZ_AGENT_STOP_MAX_REJECTIONS", 3u32)?,
|
||||
require_reply: parse_env("BUZZ_AGENT_REQUIRE_REPLY", 0u8)? != 0,
|
||||
hook_servers: parse_hook_servers_env("MCP_HOOK_SERVERS"),
|
||||
hints_enabled: parse_env("BUZZ_AGENT_NO_HINTS", 0u8)? == 0,
|
||||
thinking_effort: parse_thinking_effort(env("BUZZ_AGENT_THINKING_EFFORT").as_deref())?,
|
||||
@@ -893,6 +904,7 @@ impl Config {
|
||||
max_parallel_tools: 1,
|
||||
hook_timeout: Duration::from_secs(1),
|
||||
stop_max_rejections: 0,
|
||||
require_reply: false,
|
||||
hook_servers: HookServers::None,
|
||||
hints_enabled: false,
|
||||
thinking_effort: None,
|
||||
|
||||
@@ -2355,6 +2355,7 @@ mod tests {
|
||||
max_parallel_tools: 1,
|
||||
hook_timeout: Duration::from_secs(1),
|
||||
stop_max_rejections: 0,
|
||||
require_reply: false,
|
||||
hook_servers: HookServers::None,
|
||||
api_key: "key".into(),
|
||||
model: "model".into(),
|
||||
|
||||
@@ -33,6 +33,11 @@
|
||||
//! — expose a `_PostCompact` hook tool
|
||||
//! FAKE_MCP_POSTCOMPACT_TEXT=text
|
||||
//! — `_PostCompact` returns this (default: "")
|
||||
//! FAKE_MCP_SHELL_TOOL=1 — expose a tool whose bare name is `shell`
|
||||
//! (registered as `<server>__shell`), taking a
|
||||
//! `command` string. Lets a test drive the
|
||||
//! reply guard's recognition of a real,
|
||||
//! registered shell tool.
|
||||
|
||||
use std::io::{BufRead, Write};
|
||||
|
||||
@@ -76,6 +81,7 @@ fn make_tools(
|
||||
desc: &str,
|
||||
include_stop_hook: bool,
|
||||
include_post_compact_hook: bool,
|
||||
include_shell_tool: bool,
|
||||
) -> Vec<Value> {
|
||||
let mut tools: Vec<Value> = (0..count)
|
||||
.map(|i| {
|
||||
@@ -100,6 +106,17 @@ fn make_tools(
|
||||
"inputSchema": { "type": "object", "properties": {} },
|
||||
}));
|
||||
}
|
||||
if include_shell_tool {
|
||||
tools.push(json!({
|
||||
"name": "shell",
|
||||
"description": "run a shell command",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": { "command": { "type": "string" } },
|
||||
"required": ["command"],
|
||||
},
|
||||
}));
|
||||
}
|
||||
tools
|
||||
}
|
||||
|
||||
@@ -136,6 +153,7 @@ fn main() {
|
||||
let stop_count_limit: usize = env_usize("FAKE_MCP_STOP_COUNT", usize::MAX);
|
||||
let mut stop_calls_seen: usize = 0;
|
||||
let post_compact_hook = env_flag("FAKE_MCP_POSTCOMPACT_HOOK");
|
||||
let shell_tool = env_flag("FAKE_MCP_SHELL_TOOL");
|
||||
let post_compact_text = std::env::var("FAKE_MCP_POSTCOMPACT_TEXT").unwrap_or_default();
|
||||
|
||||
// Use a channel-based stdin reader so notifications (which carry no id)
|
||||
@@ -206,7 +224,13 @@ fn main() {
|
||||
write_response(
|
||||
id,
|
||||
json!({
|
||||
"tools": make_tools(tool_count, &desc, stop_hook, post_compact_hook)
|
||||
"tools": make_tools(
|
||||
tool_count,
|
||||
&desc,
|
||||
stop_hook,
|
||||
post_compact_hook,
|
||||
shell_tool,
|
||||
)
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1819,3 +1819,465 @@ async fn cancel_sends_notifications_cancelled_to_any_mcp_server() {
|
||||
let _ = std::fs::remove_file(&call_received_marker);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Reply guard (`BUZZ_AGENT_REQUIRE_REPLY`)
|
||||
//
|
||||
// The guard reminds the model to publish when a turn is about to end without
|
||||
// any recognized attempt to post to Buzz. It rides the existing `_Stop` gate
|
||||
// and shares its rejection budget, so most of these tests count LLM calls:
|
||||
// each reminder costs exactly one extra round.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Number of reply-guard reminders present in one captured LLM request.
|
||||
///
|
||||
/// A reminder is a tool-role message whose JSON body is attributed to the
|
||||
/// in-process guard (`server: "buzz-agent"`) at the `_Stop` hook point — the
|
||||
/// same lower-trust shape as real hook output.
|
||||
fn reply_nag_count(request: &Value) -> usize {
|
||||
request["messages"]
|
||||
.as_array()
|
||||
.map(|msgs| {
|
||||
msgs.iter()
|
||||
.filter(|m| {
|
||||
m["role"] == "tool"
|
||||
&& serde_json::from_str::<Value>(m["content"].as_str().unwrap_or(""))
|
||||
.map(|p| p["hook"] == "_Stop" && p["server"] == "buzz-agent")
|
||||
.unwrap_or(false)
|
||||
})
|
||||
.count()
|
||||
})
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// A publish-shaped call to a real registered shell tool.
|
||||
fn openai_shell_send(id: &str) -> Value {
|
||||
openai_tool_call(
|
||||
id,
|
||||
"fake__shell",
|
||||
json!({ "command": "buzz messages send --channel c --content hi" }),
|
||||
)
|
||||
}
|
||||
|
||||
/// Run one prompt to completion, answering any permission requests, and
|
||||
/// return the final response.
|
||||
async fn prompt_to_completion(h: &mut Harness, sid: &str) -> Value {
|
||||
let p = h
|
||||
.send(
|
||||
"session/prompt",
|
||||
json!({"sessionId": sid, "prompt": [{"type":"text","text":"go"}]}),
|
||||
)
|
||||
.await;
|
||||
loop {
|
||||
let v = h.recv().await;
|
||||
if v.get("method") == Some(&json!("session/request_permission")) {
|
||||
let id = v["id"].clone();
|
||||
h.write(json!({
|
||||
"jsonrpc": "2.0",
|
||||
"id": id,
|
||||
"result": { "outcome": { "outcome": "selected", "optionId": "allow" } },
|
||||
}))
|
||||
.await;
|
||||
continue;
|
||||
}
|
||||
if v["id"] == json!(p) {
|
||||
return v;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Default off: a silent turn ends on the first end_turn with no extra round.
|
||||
/// This is the invariant that keeps the feature free for everyone who hasn't
|
||||
/// opted in.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_off_by_default() {
|
||||
let llm = spawn_capturing_llm(vec![openai_text("done"), openai_text("unexpected")]).await;
|
||||
let mut h = Harness::spawn(&llm.url).await;
|
||||
let sid = init_session(&mut h, json!([])).await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
1,
|
||||
"guard must be inert when unset, got {} LLM calls",
|
||||
captured.len()
|
||||
);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// `BUZZ_AGENT_REQUIRE_REPLY=0` is off too — the toggle is numeric, so a
|
||||
/// literal `0` must not read as "set, therefore on".
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_explicit_zero_is_off() {
|
||||
let llm = spawn_capturing_llm(vec![openai_text("done"), openai_text("unexpected")]).await;
|
||||
let mut h = Harness::spawn_with_env(&llm.url, &[("BUZZ_AGENT_REQUIRE_REPLY", "0")]).await;
|
||||
let sid = init_session(&mut h, json!([])).await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
1,
|
||||
"REQUIRE_REPLY=0 must behave as off, got {} LLM calls",
|
||||
captured.len()
|
||||
);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// Opted in and silent: exactly two reminders, then the turn is allowed to
|
||||
/// end. The guard is advisory — it must never trap a turn.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_nags_twice_then_lets_the_turn_end() {
|
||||
// Budget defaults to 3, so the cap that stops the loop here is
|
||||
// MAX_REPLY_NAGS = 2, not the rejection budget.
|
||||
let llm = spawn_capturing_llm(vec![
|
||||
openai_text("silent-1"),
|
||||
openai_text("silent-2"),
|
||||
openai_text("silent-3"),
|
||||
openai_text("must-not-be-requested"),
|
||||
])
|
||||
.await;
|
||||
let mut h = Harness::spawn_with_env(&llm.url, &[("BUZZ_AGENT_REQUIRE_REPLY", "1")]).await;
|
||||
let sid = init_session(&mut h, json!([])).await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
3,
|
||||
"expected 2 reminders then end_turn (3 LLM calls), got {}",
|
||||
captured.len()
|
||||
);
|
||||
assert_eq!(
|
||||
reply_nag_count(&captured[0]),
|
||||
0,
|
||||
"reminder before any end_turn"
|
||||
);
|
||||
assert_eq!(reply_nag_count(&captured[1]), 1);
|
||||
assert_eq!(reply_nag_count(&captured[2]), 2);
|
||||
|
||||
// The reminder must name the command it wants and license silence, so it
|
||||
// cannot fight the base prompt's "silence is usually correct".
|
||||
let msgs = captured[2]["messages"].as_array().unwrap();
|
||||
let nag = msgs
|
||||
.iter()
|
||||
.filter_map(|m| serde_json::from_str::<Value>(m["content"].as_str().unwrap_or("")).ok())
|
||||
.find(|p| p["server"] == "buzz-agent")
|
||||
.expect("reminder body");
|
||||
let text = nag["text"].as_str().unwrap_or("");
|
||||
assert!(
|
||||
text.contains("buzz messages send"),
|
||||
"reminder should name the command: {text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("silence is genuinely correct"),
|
||||
"reminder must license silence: {text}"
|
||||
);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// A real publish attempt through a registered shell tool satisfies the guard:
|
||||
/// no reminder, no extra round.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_satisfied_by_registered_shell_send() {
|
||||
let llm = spawn_capturing_llm(vec![
|
||||
openai_shell_send("tc1"),
|
||||
openai_text("posted"),
|
||||
openai_text("must-not-be-requested"),
|
||||
])
|
||||
.await;
|
||||
let mut h = Harness::spawn_with_env(&llm.url, &[("BUZZ_AGENT_REQUIRE_REPLY", "1")]).await;
|
||||
let sid = init_session_with_fake_mcp(
|
||||
&mut h,
|
||||
&[("FAKE_MCP_TOOL_COUNT", "1"), ("FAKE_MCP_SHELL_TOOL", "1")],
|
||||
)
|
||||
.await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
2,
|
||||
"a recognized send must not be nagged, got {} LLM calls",
|
||||
captured.len()
|
||||
);
|
||||
assert_eq!(reply_nag_count(&captured[1]), 0);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// A publish-shaped call to a shell tool that is *not registered* never runs —
|
||||
/// preflight rejects it — so it must not disarm the guard. This is what the
|
||||
/// `has`/`is_hook` checks in the predicate buy.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_ignores_unregistered_shell_tool() {
|
||||
// FAKE_MCP_SHELL_TOOL is absent, so `fake__shell` is a hallucination.
|
||||
let llm = spawn_capturing_llm(vec![
|
||||
openai_shell_send("tc1"),
|
||||
openai_text("silent-1"),
|
||||
openai_text("silent-2"),
|
||||
])
|
||||
.await;
|
||||
let mut h = Harness::spawn_with_env(
|
||||
&llm.url,
|
||||
&[
|
||||
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
|
||||
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "1"),
|
||||
],
|
||||
)
|
||||
.await;
|
||||
let sid = init_session_with_fake_mcp(&mut h, &[("FAKE_MCP_TOOL_COUNT", "1")]).await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
3,
|
||||
"expected the hallucinated call to still be nagged, got {} LLM calls",
|
||||
captured.len()
|
||||
);
|
||||
let msgs = captured[1]["messages"].as_array().unwrap();
|
||||
assert!(
|
||||
msgs.iter()
|
||||
.any(|m| m["role"] == "tool"
|
||||
&& m["content"].as_str().unwrap_or("").contains("unknown tool")),
|
||||
"expected preflight to reject the call: {msgs:?}"
|
||||
);
|
||||
assert_eq!(reply_nag_count(&captured[2]), 1);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// A publish-shaped call discarded by the per-turn tool-call cap never runs,
|
||||
/// so it must not suppress the reminder either. Pins the check's placement
|
||||
/// after `calls.truncate(MAX_TOOL_CALLS_PER_TURN)`.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_ignores_calls_lost_to_the_turn_cap() {
|
||||
// 64 filler calls (the cap) followed by the publish attempt, which is
|
||||
// therefore truncated away. The shell tool *is* registered here, so only
|
||||
// the placement — not tool identity — can explain the reminder.
|
||||
let mut calls: Vec<Value> = (0..64)
|
||||
.map(|i| {
|
||||
json!({
|
||||
"id": format!("c{i}"),
|
||||
"type": "function",
|
||||
"function": { "name": "fake__tool_0", "arguments": "{}" },
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
calls.push(json!({
|
||||
"id": "c-send",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "fake__shell",
|
||||
"arguments": json!({ "command": "buzz messages send --channel c --content hi" })
|
||||
.to_string(),
|
||||
},
|
||||
}));
|
||||
let truncated_send = json!({
|
||||
"id": "cc-trunc", "object": "chat.completion", "model": "fake-model",
|
||||
"choices": [{
|
||||
"index": 0,
|
||||
"message": { "role": "assistant", "content": null, "tool_calls": calls },
|
||||
"finish_reason": "tool_calls",
|
||||
}],
|
||||
});
|
||||
let llm = spawn_capturing_llm(vec![
|
||||
truncated_send,
|
||||
openai_text("silent-1"),
|
||||
openai_text("silent-2"),
|
||||
])
|
||||
.await;
|
||||
let mut h = Harness::spawn_with_env(
|
||||
&llm.url,
|
||||
&[
|
||||
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
|
||||
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "1"),
|
||||
],
|
||||
)
|
||||
.await;
|
||||
let sid = init_session_with_fake_mcp(
|
||||
&mut h,
|
||||
&[("FAKE_MCP_TOOL_COUNT", "1"), ("FAKE_MCP_SHELL_TOOL", "1")],
|
||||
)
|
||||
.await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
3,
|
||||
"a truncated send must still be nagged, got {} LLM calls",
|
||||
captured.len()
|
||||
);
|
||||
assert_eq!(reply_nag_count(&captured[2]), 1);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// The shared `_Stop` rejection budget is the outer cap: at 1 the guard gets
|
||||
/// one reminder instead of two. Documented degradation, not a bug.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_bounded_by_stop_rejection_budget() {
|
||||
let llm = spawn_capturing_llm(vec![
|
||||
openai_text("silent-1"),
|
||||
openai_text("silent-2"),
|
||||
openai_text("must-not-be-requested"),
|
||||
])
|
||||
.await;
|
||||
let mut h = Harness::spawn_with_env(
|
||||
&llm.url,
|
||||
&[
|
||||
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
|
||||
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "1"),
|
||||
],
|
||||
)
|
||||
.await;
|
||||
let sid = init_session(&mut h, json!([])).await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
2,
|
||||
"budget 1 must allow exactly one reminder, got {} LLM calls",
|
||||
captured.len()
|
||||
);
|
||||
assert_eq!(reply_nag_count(&captured[1]), 1);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// Budget 0 disables every objection at the gate, including this one.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_off_when_stop_budget_is_zero() {
|
||||
let llm = spawn_capturing_llm(vec![openai_text("done"), openai_text("unexpected")]).await;
|
||||
let mut h = Harness::spawn_with_env(
|
||||
&llm.url,
|
||||
&[
|
||||
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
|
||||
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "0"),
|
||||
],
|
||||
)
|
||||
.await;
|
||||
let sid = init_session(&mut h, json!([])).await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
1,
|
||||
"budget 0 must disable the guard, got {} LLM calls",
|
||||
captured.len()
|
||||
);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// The two axes are independent inside one shared budget: a round carrying
|
||||
/// both a `_Stop` hook objection and a reminder costs one rejection and
|
||||
/// delivers both texts, and once the reminders are spent the hook objection
|
||||
/// continues alone.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn reply_guard_combines_with_stop_hook_objection() {
|
||||
// The hook objects on its first 3 calls, then clears. Reminders stop
|
||||
// after 2, so round 3 must carry the hook text and no new reminder.
|
||||
let llm = spawn_capturing_llm(vec![
|
||||
openai_text("silent-1"),
|
||||
openai_text("silent-2"),
|
||||
openai_text("silent-3"),
|
||||
openai_text("silent-4"),
|
||||
])
|
||||
.await;
|
||||
let mut h = Harness::spawn_with_env(
|
||||
&llm.url,
|
||||
&[
|
||||
("BUZZ_AGENT_REQUIRE_REPLY", "1"),
|
||||
("MCP_HOOK_SERVERS", "fake"),
|
||||
("BUZZ_AGENT_STOP_MAX_REJECTIONS", "10"),
|
||||
],
|
||||
)
|
||||
.await;
|
||||
let sid = init_session_with_fake_mcp(
|
||||
&mut h,
|
||||
&[
|
||||
("FAKE_MCP_TOOL_COUNT", "1"),
|
||||
("FAKE_MCP_STOP_HOOK", "1"),
|
||||
("FAKE_MCP_STOP_TEXT", "you have open todos"),
|
||||
("FAKE_MCP_STOP_COUNT", "3"),
|
||||
],
|
||||
)
|
||||
.await;
|
||||
|
||||
let r = prompt_to_completion(&mut h, &sid).await;
|
||||
assert_eq!(r["result"]["stopReason"], "end_turn");
|
||||
|
||||
let captured = llm.captured.lock().await;
|
||||
assert_eq!(
|
||||
captured.len(),
|
||||
4,
|
||||
"expected 3 objecting rounds then a clear end, got {}",
|
||||
captured.len()
|
||||
);
|
||||
|
||||
let hook_objections = |req: &Value| -> usize {
|
||||
req["messages"]
|
||||
.as_array()
|
||||
.map(|msgs| {
|
||||
msgs.iter()
|
||||
.filter(|m| {
|
||||
m["content"]
|
||||
.as_str()
|
||||
.unwrap_or("")
|
||||
.contains("you have open todos")
|
||||
})
|
||||
.count()
|
||||
})
|
||||
.unwrap_or(0)
|
||||
};
|
||||
|
||||
// Round 2 carries one of each — a single rejection bought both texts.
|
||||
assert_eq!(reply_nag_count(&captured[1]), 1);
|
||||
assert_eq!(hook_objections(&captured[1]), 1);
|
||||
// Round 4: the hook objected three times, the guard only twice.
|
||||
assert_eq!(reply_nag_count(&captured[3]), 2);
|
||||
assert_eq!(hook_objections(&captured[3]), 3);
|
||||
h.shutdown().await;
|
||||
}
|
||||
|
||||
/// An unparseable toggle is a startup error, not a silent default. `parse_env`
|
||||
/// is generic over `FromStr`, so this also pins the numeric type: a `bool`
|
||||
/// field would have rejected the documented `1`.
|
||||
#[test]
|
||||
fn reply_guard_rejects_unparseable_toggle() {
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_buzz-agent"))
|
||||
.env("BUZZ_AGENT_PROVIDER", "openai")
|
||||
.env("OPENAI_COMPAT_API_KEY", "test")
|
||||
.env("OPENAI_COMPAT_MODEL", "fake-model")
|
||||
.env("BUZZ_AGENT_REQUIRE_REPLY", "true")
|
||||
.stdin(Stdio::null())
|
||||
.output()
|
||||
.expect("run buzz-agent");
|
||||
assert!(
|
||||
!out.status.success(),
|
||||
"expected a config error exit, got {:?}",
|
||||
out.status
|
||||
);
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(
|
||||
stderr.contains("BUZZ_AGENT_REQUIRE_REPLY"),
|
||||
"expected the offending key in the error, got: {stderr}"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -47,6 +47,13 @@ pub fn apply_relay_mesh_env(
|
||||
// may deliberately choose a smaller cap or a different effort. This function
|
||||
// runs after those layers during readiness, so never clobber their values.
|
||||
insert_default_if_unset(env, "BUZZ_AGENT_MAX_OUTPUT_TOKENS", "4096");
|
||||
// Mesh agents run on small local models, which are the ones most likely to
|
||||
// do the work and then end the turn without publishing it — the failure the
|
||||
// reply guard exists to catch. Everywhere else it stays opt-in and unset.
|
||||
// A default, not policy: an explicit `0` from the agent/persona/global env
|
||||
// survives (see `insert_default_if_unset`, and the copy-forward list in
|
||||
// `relay_mesh_process_env` that preserves it through the spawn path).
|
||||
insert_default_if_unset(env, "BUZZ_AGENT_REQUIRE_REPLY", "1");
|
||||
// Deliberately no BUZZ_AGENT_THINKING_EFFORT default: mesh translates
|
||||
// `reasoning_effort` into the chat template's `enable_thinking` flag, so any
|
||||
// value we pick overrides each model's own template default — and the right
|
||||
@@ -80,7 +87,15 @@ pub fn relay_mesh_process_env(
|
||||
model: &str,
|
||||
) -> std::collections::BTreeMap<String, String> {
|
||||
let mut env = std::collections::BTreeMap::new();
|
||||
for key in ["BUZZ_AGENT_MAX_OUTPUT_TOKENS", "BUZZ_AGENT_THINKING_EFFORT"] {
|
||||
for key in [
|
||||
"BUZZ_AGENT_MAX_OUTPUT_TOKENS",
|
||||
"BUZZ_AGENT_THINKING_EFFORT",
|
||||
// Must be copied forward for the user's value to survive: this map is
|
||||
// written onto the command *after* the layered user env, so a key absent
|
||||
// here is re-defaulted by `apply_relay_mesh_env` below and an explicit
|
||||
// `BUZZ_AGENT_REQUIRE_REPLY=0` would be silently overridden back to `1`.
|
||||
"BUZZ_AGENT_REQUIRE_REPLY",
|
||||
] {
|
||||
if let Some(value) = effective_env.get(key) {
|
||||
env.insert(key.to_string(), value.clone());
|
||||
}
|
||||
@@ -145,6 +160,78 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_provider_enables_reply_guard_by_default() {
|
||||
let mut env = BTreeMap::new();
|
||||
apply_relay_mesh_env(
|
||||
&mut env,
|
||||
Some(RELAY_MESH_PROVIDER_ID),
|
||||
Some(RELAY_MESH_AUTO_MODEL_ID),
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
|
||||
Some("1"),
|
||||
"mesh agents opt into the reply guard automatically"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_provider_preserves_explicit_reply_guard_opt_out() {
|
||||
let mut env = BTreeMap::from([("BUZZ_AGENT_REQUIRE_REPLY".to_string(), "0".to_string())]);
|
||||
apply_relay_mesh_env(
|
||||
&mut env,
|
||||
Some(RELAY_MESH_PROVIDER_ID),
|
||||
Some(RELAY_MESH_AUTO_MODEL_ID),
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
|
||||
Some("0"),
|
||||
"an explicit opt-out is a user decision, not a value to re-default"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn non_mesh_provider_leaves_reply_guard_unset() {
|
||||
let mut env = BTreeMap::new();
|
||||
apply_relay_mesh_env(&mut env, Some("anthropic"), Some("claude-haiku-4.5"));
|
||||
|
||||
assert_eq!(
|
||||
env.get("BUZZ_AGENT_REQUIRE_REPLY"),
|
||||
None,
|
||||
"the guard stays opt-in everywhere except mesh"
|
||||
);
|
||||
assert!(env.is_empty(), "non-mesh providers get no mesh env at all");
|
||||
}
|
||||
|
||||
/// The spawn path writes this map onto the command *after* the layered user
|
||||
/// env, so an explicit opt-out only survives if it is copied forward. Without
|
||||
/// the copy-forward, `apply_relay_mesh_env` re-defaults it to `1` here and
|
||||
/// silently overrides the user at spawn while readiness still shows `0`.
|
||||
#[test]
|
||||
fn process_env_preserves_explicit_reply_guard_opt_out() {
|
||||
let effective_env =
|
||||
BTreeMap::from([("BUZZ_AGENT_REQUIRE_REPLY".to_string(), "0".to_string())]);
|
||||
|
||||
let env = relay_mesh_process_env(&effective_env, "Gemma-4");
|
||||
|
||||
assert_eq!(
|
||||
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
|
||||
Some("0")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn process_env_enables_reply_guard_when_user_is_silent() {
|
||||
let env = relay_mesh_process_env(&BTreeMap::new(), "Gemma-4");
|
||||
|
||||
assert_eq!(
|
||||
env.get("BUZZ_AGENT_REQUIRE_REPLY").map(String::as_str),
|
||||
Some("1")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn process_env_seeds_controls_without_restoring_unrelated_credentials() {
|
||||
let effective_env = BTreeMap::from([
|
||||
|
||||
@@ -65,6 +65,23 @@ These constraints ensure a buggy or malicious hook cannot trap the agent.
|
||||
Hooks are **off by default**. The operator must explicitly opt in via
|
||||
`MCP_HOOK_SERVERS`.
|
||||
|
||||
### Not a hook: the reply guard
|
||||
|
||||
`buzz-agent` has one in-process objection at the `_Stop` gate that is **not** an
|
||||
MCP hook and exposes no hook tool: the reply guard
|
||||
(`BUZZ_AGENT_REQUIRE_REPLY=1`), which reminds the model to publish when a turn is
|
||||
about to end with nothing posted to Buzz. There is no `_ReplyGuard` tool to
|
||||
implement and no server to allowlist — the env var and the recognition contract
|
||||
are documented in
|
||||
[crates/buzz-agent/README.md](../crates/buzz-agent/README.md#reply-guard).
|
||||
|
||||
It is mentioned here only because it shares this lifecycle point and this
|
||||
budget: its reminders count against `BUZZ_AGENT_STOP_MAX_REJECTIONS` like any
|
||||
hook objection, and a round carrying both a hook objection and a reminder costs
|
||||
one rejection and delivers both texts. Setting the budget to 0 disables both.
|
||||
That the gate can carry in-process objections alongside hook output is
|
||||
deliberate; hooks see no difference.
|
||||
|
||||
## Implementing a Hook
|
||||
|
||||
Any MCP server can expose hooks. Example: a test-runner server that blocks
|
||||
|
||||
Reference in New Issue
Block a user