mirror of
https://github.com/block/buzz.git
synced 2026-08-18 06:50:31 +02:00
Sami
8cbe20dd65
The HTTP bridge's retry ladder treated a 429 as a generic transient
failure. It retried on its own rungs — 500ms, 1s, 2s — and never read
the `retry in {N}s` hint out of the response body, even though the relay
puts one there on every quota rejection and the client already has a
parser for it, used on the WS path.
Every rung is shorter than any window the relay would name. A 429 with a
51s hint was retried three times inside that window, and all three were
refused: the limiter's INCR runs on denied checks too, so each one made
the caller's own situation slightly worse while it waited. The ladder
exhausted in about 3.5s and the call failed anyway.
request_with_retry now parses the hint from a 429 body and passes it to
a new pure rest_retry_delay(rung, hint_secs, jitter_nanos), which takes
whichever of the rung and the hint is longer. The hint is extended with
the one-sided jitter added in the earlier commit, so jitter can never
pull the wake-up back inside the window the relay named; an absent or
sub-rung hint keeps the ladder rung and its symmetric jitter, which is
correct for a self-chosen delay. A hint is capped at REST_RETRY_HINT_MAX
(90s) so a pathological value cannot park a caller for hours, with the
same `const` assertion the CLI cap carries: the property pinned is that
the cap outlasts the relay's 60s window, not the number.
Only a 429 is read for a hint. 502/503/504 stay on the ladder, and a
body that cannot be read or parsed leaves the rung in charge, so the
worst case is the behaviour that exists today.
Reachability is partial and the docstring says so. Most REST callers
wrap these requests in their own 500ms-5s tokio timeout, which cancels
a multi-second hint sleep rather than sleeping it out. That is the right
direction — those callers were going to fail regardless, and this way
they stop spending the relay's counter on attempts that cannot succeed —
but it does mean the full benefit lands on the unbudgeted callers, which
now include the setup nudge.
Mutation results. Ignoring the hint, swapping the one-sided extension
for a symmetric one, and dropping the cap are each killed by the pure
tests (3, 3, and 1 failures respectively). Reverting the *wiring* — so
the hint is never parsed from the body — initially survived all 686
tests, because pure tests of the delay function structurally cannot see
whether the call site supplies a hint. That is the same blind spot the
transport tests closed for the nudge. The added paused-time test drives
a real 429-then-200 exchange and asserts the retry slept 51s rather than
the 560ms rung; it kills that mutant with the measured sleep in the
failure message.
Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz>
Signed-off-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz>