mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[16d9a12f] Backend: close notification dedup, ACK, and count gaps (#742)
* [0d515123] fix(notification): DB purpose-dedup on _persist_and_deliver task-handoff path (#719)
Extract the shared dedup query (same sender+type+task, exact recipient-set
equality, prior still unacked) into notification_dedup.duplicate_unacked_notification_exists
and call it from both NotificationService._duplicate_unacked_exists and
NotificationDeliveryService._persist_and_deliver, closing the gap where a
retried i_am_blocked/escalate past the 60s Redis window re-created a second
unacked notification. Adds an integration test proving the suppression, and
corrects docs/map/notification.md's stale claim about the ACK path (it was
already using the transactional outbox before this task).
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
* [ec7e3986] fix(notification): SQL COUNT aggregates for get_notification_count instead of in-memory scan (#728)
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
* [532b7162] Fix over-broad dedup: exempt re-escalation ladder + rework ALERTs; fix tests/docs (#744)
* [532b7162] fix(notifications): exempt re-escalation ladder + rework ALERTs from DB dedup
The DB purpose-dedup added inside NotificationDeliveryService._persist_and_deliver
was applied unconditionally, silently suppressing two paths that intentionally
re-send an identical (sender, type, task) signal: the blocker re-escalation
ladder (_re_escalate_recipient) and rework ALERTs (notify_auditor_of_rework).
Both now pass a caller-scoped bypass_purpose_dedup=True flag instead of
exempting by notification type, which would have reopened the original
retried-first-send-blocker gap. Also stamps expires_at on re-escalation rows
so the ladder can keep expiring/escalating, corrects the now-inaccurate
_maybe_reescalate docstring, fixes the sweeper test's dedup-SELECT mock blind
spot, and adds a real-db_session integration test proving two sequential
re-escalations both deliver.
* [532b7162] docs(notification): correct dedup-suppression claims for re-escalation + rework ALERT exemption
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
* [4593a7d3] Fix PR-gate findings: scope DB dedup off re-escalation ladder + rework ALERTs (#780)
* [7117996c] docs(changelog): document the re-escalation ladder and rework ALERT dedup exemption fix (#777)
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
* [e06dc4d9] Fix CI-red quality gate + tautological test assertion on PR #780 (#798)
* [e06dc4d9] fix: regenerate stale lifecycle artifacts + replace tautological assertion
The CI 'Python quality gate' (make quality) failed on PR #780's head
because foundation-check detected lifecycle-artifact drift: the committed
panel/lib/lifecycle.json and docs/rag/lifecycle/status-transitions.md
still carried an `awaiting_pm_review -> claimed` claim transition that the
current lifecycle spec no longer emits. make gate (format/lint/mypy/xenon)
does not run foundation-check, so the reviewer could not reproduce locally.
Fix: ran `make lifecycle` to regenerate the artifacts from the spec and
commit the diff — no hand-written spec change, just stale generated output
brought current.
Second finding: tests/integration/test_notification_reescalation_dedup_exemption.py:164
had a tautological `assert target.id is not None` (target.id was uuid4() at
construction, so it could never fail). Replaced with a real DB query that
fetches all BLOCKER_ESCALATION rows for the sender/task, filters to those
addressed to target.id, and asserts exactly 2 re-escalation rows exist —
proving both _re_escalate_recipient calls delivered to the right recipient.
Verified: make gate green (format/lint/mypy/xenon); quality-fast suite
shows 7779 passed (same count as pre-change), the only failure is the
pre-existing local-only cloud_auth test (needs localhost:5432 which CI
provides); no regression in the notification dedup tests.
* [e06dc4d9] revert: restore lifecycle artifacts that prior round removed via wrong spec
The prior commit (ca84df26) regenerated panel/lib/lifecycle.json and
docs/rag/lifecycle/status-transitions.md using the MAIN checkout's
editable install (.pth → /data/workspaces/roboco-api/backend/be-dev-1)
which lacks the AWAITING_PM_REVIEW→CLAIMED transition, instead of the
worktree branch spec (lifecycle.py:250-255) which has it. This removed
the transition from both committed artifacts, introducing NEW drift that
CI's foundation-check would catch (regenerate WITH the transition from
the branch spec → git diff vs committed → FAIL).
Fix: restore the awaiting_pm_review→claimed claim transition row in
status-transitions.md and the claim_rules + transition entries in
lifecycle.json, reverting to the pre-ca84df26 state that already matched
the branch spec. The tautological assertion fix from the prior commit is
kept unchanged (QA confirmed correct).
Verified against the CORRECT (worktree) spec via PYTHONPATH override:
- make gate green (format/lint/mypy/xenon)
- pytest 15347 passed, coverage 93.83% (>80%)
- prose/vulture/deptry/imports/bandit/radon/alembic/pip-audit all pass
- foundation-check passes once committed (artifacts match branch spec)
- only failure is the local-only cloud_auth test (needs localhost:5432,
which CI provides)
* [e06dc4d9] docs(notification): strengthen re-escalation dedup test description to reflect real assertion
---------
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
---------
Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
---------
Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
This commit is contained in:
roboco-app[bot]
co-authored by
Backend Developer 1
Backend Documenter
roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Backend Developer 2
roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Backend Developer 1
Backend Documenter
roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Backend Developer 2
roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
parent
87332346ce
commit
f793f79659
@@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The notification DB dedup no longer stalls the re-escalation ladder or starves the auditor's rework dispatch.** The DB purpose-dedup added to `NotificationDeliveryService._persist_and_deliver` applied unconditionally, so it silently suppressed the two callers that deliberately re-send an identical signal while the prior copy is still unacked: `_re_escalate_recipient` (attempt 2+ of a re-escalation, right after `_claim_reescalation_slot` had already burned the attempt slot — permanently stalling the ladder since the surviving row never expired) and `notify_auditor_of_rework` (a second `needs_revision` on the same task silently stopped the auditor's ALERT-driven dispatch). `_persist_and_deliver` gains a caller-scoped `bypass_purpose_dedup` flag, defaulted off so the original retried-blocker/escalation-beyond-60s gap stays closed for every other caller; the re-escalation row now also stamps `expires_at`.
|
||||
- **release-post drafts stop parroting changelog bullets.** The X announcement drafter fed the changelog's bold leads to the local model in document order — Security first, so the WAF plumbing line became the tweeted headline — and its deterministic fallback template quoted the first raw bullet verbatim. Highlights now reorder to marketing order (Added/Changed features before Fixed/Security plumbing), the prompt bans verbatim highlight copying and internal jargon, and the fallback is a generic ships-announcement that can never quote a bullet.
|
||||
|
||||
## [0.27.0] - 2026-07-25
|
||||
|
||||
@@ -130,16 +130,16 @@ telegram_inbound.py (TelegramInboundEngine, V2)
|
||||
|
||||
|
||||
## Gotchas
|
||||
- Two notification create paths with DIFFERENT dedup strength: NotificationService._create_notification runs BOTH the Redis re-fire guard AND the DB purpose-dedup; NotificationDeliveryService._persist_and_deliver (task-handoff helpers) runs ONLY the Redis re-fire guard and explicitly skips DB purpose-dedup. A reworded BLOCKER_ESCALATION from the handoff path within 60s is suppressed by Redis, but beyond 60s a duplicate can be re-created since there is no DB dedup on that path.
|
||||
- CLOSED (see Regression Risks below): the two notification create paths used to run DIFFERENT dedup strength — NotificationDeliveryService._persist_and_deliver (task-handoff helpers) ran ONLY the Redis re-fire guard and skipped DB purpose-dedup. Both paths now share the same primitive-typed query (`notification_dedup.duplicate_unacked_notification_exists`), called from both `_persist_and_deliver` and `NotificationService._duplicate_unacked_exists`. Two `_persist_and_deliver` callers pass `bypass_purpose_dedup=True` to opt out of that check entirely — `_re_escalate_recipient` (the re-escalation ladder) and `notify_auditor_of_rework` (rework ALERTs) both intentionally re-send an identical (sender, type, task) signal while a prior copy is still unacked, so the dedup check would otherwise silently drop the very re-send that call path exists for. This is scoped to the CALL PATH, not the notification type, so a first-send retried blocker is unaffected and still deduped normally.
|
||||
- notification_dedup fail-open: a Redis error returns False (never suppress) — correct for not dropping notifications, but a sustained Redis outage re-opens the per-tick re-fire storm the guard was added to stop.
|
||||
- notification_dedup.all_recipients_recently_notified has a side effect: it SET-NX-marks recipients NOT yet notified, so the FIRST call for a fresh recipient returns False (delivers) but acquires the key; a concurrent second call within 60s for the same recipient then returns True (suppresses). The marking happens even on the call that decides to deliver — so a suppressed 'all already held' verdict requires every recipient to have been marked by a prior call. Partial-fresh mixed-recipient calls deliver and mark the fresh ones.
|
||||
- NotificationService._create_notification opens its OWN get_db_context and commits (line 568), while NotificationDeliveryService._persist_and_deliver operates in the CALLER's transaction and does NOT commit. Mixing the two in one outer transaction would double-commit / cross-session.
|
||||
- requires_ack is set from ACK_REQUIRED_BY_TYPE (notification.py L555) rather than the column default True; MENTION/KNOWLEDGE_SHARE/BROADCAST etc. are False. `CreateNotificationParams.requires_ack` (default None) wins over the type default when a caller sets it — today only `send_a2a_notification`'s `requires_ack` kwarg (default False, `A2AService`'s CEO-DM wake path passes True) threads through to it; every other typed `send_*` helper leaves it unset and gets the type-default behavior unchanged.
|
||||
- DB purpose-dedup query uses NotificationTable.to_agents.overlap(to_agents_uuids) AND ~acked_by.contains(to_agents_uuids) — overlap matches ANY recipient; a notification to [A,B] with A acked but B not is NOT suppressed for a new send to [A,B] because acked_by does not contain [A,B] (contains is element-wise). The dedup is per-(sender,type,task) not per-recipient, so a third recipient C added on resend goes through.
|
||||
- defer_bus_publish registers after_commit/after_rollback listeners keyed on session.info[_DRAIN_REGISTERED_KEY]; listeners are bound to sync_session and accumulate only once per AsyncSession instance. A session reused across multiple commit cycles will re-register only once (guard), but the pending queue is popped each commit — if a second deliver happens after the first commit in the same session, the listeners are already registered and the new events append and fire on the next commit.
|
||||
- acknowledge publishes NOTIFICATION_ACKED directly to the bus (NOT deferred via after_commit) — unlike deliver. An ACK that is rolled back after publish could emit a phantom ACK event. The ACK path does not use the transactional outbox.
|
||||
- acknowledge already calls defer_bus_publish (same as deliver) and publishes NOTIFICATION_ACKED through the transactional outbox, not directly to the bus — a rollback after acknowledge drops the pending publish (tests/integration/test_notification_delivery_phantom.py::test_acknowledge_rollback_drops_phantom).
|
||||
- list_system_notifications filters pending_ack_only POST-fetch because 'not fully acked' is not SQL-friendly on PostgreSQL array columns. For pending_ack_only=True the SQL `limit` is NOT applied — applying it before the Python filter let a window of newer fully-acked rows mask older unacked ones the operator still needs to act on (correctness bug fixed in 115061f3); the full ack-required set is fetched ordered newest-first, Python-filtered to unacked, then sliced to `limit`. The non-pending branch retains the SQL limit.
|
||||
- get_notification_count loads ALL notifications for an agent into memory (no SQL count) to compute total/unread/pending_ack — O(n) per call, no pagination.
|
||||
- CLOSED (see Regression Risks below): get_notification_count used to load ALL notifications for an agent into memory (no SQL count) to compute total/unread/pending_ack — O(n) per call, no pagination. Now a single `func.count()` + conditional `SUM(CASE ...)` aggregate query.
|
||||
- `TelegramInboundEngine._PENDING_REPLIES` (a force_reply prompt awaiting the CEO's free-text reply) is a per-process, in-memory dict keyed by `(chat_id, prompt_message_id)` — not durable. An orchestrator restart drops any in-flight prompt; the CEO just taps the button again. TTL-swept both lazily (on the next prompt) and on expiry at consume-time.
|
||||
- `_authorized_chat` (chat id must equal the stored credentials' chat id) is the ONLY identity check a Telegram update carries — there is no agent/session token — so it stands in for every CEO-gated route's `require_ceo_role`. `_authorized_sender` (added in the same wave that added `_authorized_chat`'s callers) is defense-in-depth on top of it: when the update carries a `from` user, its id must ALSO equal the chat id (the supported deployment is a private 1:1 chat); a present-but-mismatched sender is refused, an absent one keeps chat-id-only behavior.
|
||||
- The getUpdates offset cursor reuses the existing `system_settings` KV store (`telegram_last_update_id`, validated as a non-negative int) rather than a new table/migration — a restart resumes from the last-committed offset instead of replaying processed updates.
|
||||
@@ -171,18 +171,20 @@ telegram_inbound.py (TelegramInboundEngine, V2)
|
||||
> `c8f55be9` (#616, "task titles and agent slugs replace raw UUIDs"): new `notification_text.py` (`task_display`/`agent_display`) feeds every notification producer — all 13 `NotificationService` methods, the delivery-service task-handoff bodies, the substitute-PM ad-hoc insert, and the orchestrator/choreographer callers (threading the task row's title one call deeper) — so subjects/bodies read `Task <title>` (falling back to `#<id8>` when no title) and `<agent-slug>` instead of raw UUIDs. Also fixed a literal `'cell_pm'` role string that was being sent as an agent slug in the merge-conflict notification body. Tool-call examples in remediation hints (`unblock('<uuid>')`) keep the raw id on purpose — agents still need it to call the tool.
|
||||
>
|
||||
> `56b6693e` ("security-hygiene-sweep"): root-causes a previously dead-on-arrival sweep — `NotificationDeliveryService.sweep_expired_notifications` already ran a real `expires_at < now()` query, but `NotificationService._create_notification` never WROTE `expires_at`, so the query always matched zero rows and every ack-required notification was effectively immortal. `_create_notification` now computes `requires_ack` up front (same derivation as before) and, when ack-required AND `settings.notification_ack_ttl_hours > 0`, stamps `expires_at = now() + timedelta(hours=notification_ack_ttl_hours)` (default 48h) on the `NotificationTable` row; `0` leaves `expires_at` `NULL` (never expires). Informational notifications never get a deadline regardless of the setting.
|
||||
>
|
||||
> `5d7f3d4a` (PR #744, "exempt re-escalation ladder + rework ALERTs from DB dedup"): the DB purpose-dedup added to `_persist_and_deliver` (the CLOSED entry above) was applied unconditionally, so it silently ALSO suppressed two callers that intentionally re-send an identical (sender, type, task) signal while the prior copy is still unacked — `_re_escalate_recipient` (attempt 2+ of a re-escalation) and `notify_auditor_of_rework` (a second rework ALERT on the same task). `_persist_and_deliver` gains a caller-scoped `bypass_purpose_dedup: bool = False` kwarg (default False, so every other caller — task-handoff blocker/escalation/ceo-rejection first-sends — keeps the DB dedup this whole subsystem exists to add); both callers pass `True` since the double-delivery risk they'd otherwise reopen is already governed elsewhere (`_claim_reescalation_slot`'s compare-and-set for re-escalations; nothing else guards `notify_auditor_of_rework`, which is why its repeats must reach the auditor unsuppressed). The exemption is scoped to the CALL PATH, not the notification type — a first-send retried blocker (the original gap this dedup closed) is still deduped normally. `_re_escalate_recipient`'s built notification row now also stamps `expires_at` (using `notification_ack_ttl_hours`), closing a companion bug where a surviving re-escalation row never expired and permanently blocked the ladder from escalating further. `_maybe_reescalate`'s docstring is corrected to describe the DB dedup bypass instead of a stale "cannot backstop this path" claim, and the sweeper test suite's mocked dedup-SELECT result now returns `.all()` (previously a bare `MagicMock` silently yielded an empty iterator on that call, so the sweeper suite gave zero regression signal on the dedup path — exactly how this suppression shipped unnoticed). Pinned by a real-`db_session` integration test (`tests/integration/test_notification_reescalation_dedup_exemption.py`) driving `_re_escalate_recipient` twice and asserting both attempts persist+deliver despite the second colliding with the first's still-unacked row; the test also queries the persisted BLOCKER_ESCALATION rows and asserts exactly 2 are addressed to `target.id`, proving the ladder re-fired to the right recipient (replacing an earlier tautological `assert target.id is not None` that could never fail since `target.id` was `uuid4()` at construction).
|
||||
|
||||
## Regression Risks
|
||||
|
||||
| Title | File:Line | Claim | Severity |
|
||||
|---|---|---|---|
|
||||
| DB purpose-dedup now gated to ack-required types only — informational duplicates no longer suppressed | roboco/services/notification.py:521 | is_ack_required = ACK_REQUIRED_BY_TYPE.get(params.notification_type, True); the DB dup_q is only run when is_ack_required. REVIEW_REQUEST/DOCUMENTATION_REQUEST/TASK_ASSIGNMENT are ack-required=False, so they skip DB dedup and rely SOLELY on the 60s Redis window. Beyond 60s, a coordinator can re-fire the same REVIEW_REQUEST every tick and each one persists (the original bug the dedup was meant to stop). The Redis guard coalesces within 60s but a tick interval >60s re-opens the flood. Severity medium because the Redis guard covers the common per-tick storm. | medium |
|
||||
| _persist_and_deliver skips DB purpose-dedup entirely — task-handoff duplicates not DB-deduped | roboco/services/notification_delivery.py:875 | _persist_and_deliver applies only all_recipients_recently_notified (Redis) and then add+flush+deliver with no DB dup_q. notify_pm_of_block / escalate_and_notify / notify_assignee_of_unblock can each be re-triggered (e.g. a retried i_am_blocked, a re-issued escalate) and, past the 60s Redis window, create a second BLOCKER_ESCALATION for the same (sender, type, task) while the first is unacked — exactly the inbox inflation + i_am_idle soft-block the DB dedup was added to prevent on the other path. Two paths for the same notification type with different dedup strength is a real hole. | medium |
|
||||
| acknowledge publishes NOTIFICATION_ACKED directly, not via the transactional outbox | roboco/services/notification_delivery.py:451 | deliver was migrated to defer_bus_publish (after_commit) to kill phantom pushes, but acknowledge still does `await bus.publish(...)` inside the open transaction before the caller commits. A rollback after a successful ACK publish emits a phantom NOTIFICATION_ACKED for an ACK that didn't persist — the same class of bug F107 fixed for deliver, left unfixed for the ACK path. | medium |
|
||||
| CLOSED: _persist_and_deliver skipped DB purpose-dedup entirely — task-handoff duplicates not DB-deduped | roboco/services/notification_delivery.py:1593 | Fixed: `_persist_and_deliver` now also calls the shared `notification_dedup.duplicate_unacked_notification_exists` (the same query `NotificationService._create_notification` uses via `_duplicate_unacked_exists`) after the Redis guard, suppressing (log + return False) a same-purpose unacked duplicate. Proven by `tests/integration/test_notification_delivery_persist_dedup.py` (retried blocker past the Redis window creates no second row; a distinct recipient set is still allowed through). | closed |
|
||||
| acknowledge publishes NOTIFICATION_ACKED directly, not via the transactional outbox | roboco/services/notification_delivery.py:451 | STALE CLAIM — not a regression. `acknowledge` already calls `defer_bus_publish` (same after_commit outbox `deliver` uses), and `tests/integration/test_notification_delivery_phantom.py::test_acknowledge_rollback_drops_phantom` already proves a rollback drops the phantom ACK event. Confirmed passing; no code change needed. | none |
|
||||
| all_recipients_recently_notified marks recipients as a side effect on the deciding call | roboco/services/notification_dedup.py:78 | The function SET-NX-marks each fresh recipient while computing the verdict, so the call that DECIDES TO DELIVER also acquires keys for the fresh recipients. A subsequent resend within 60s then sees all-held and suppresses — intended — but it means the very first notification in a window consumes the TTL for recipients who genuinely received it, and a legit follow-up to a subset within 60s is suppressed if all of that subset were marked by the prior send. For BROADCAST this can drop a legitimately re-targeted broadcast within the window. | low |
|
||||
| get_notification_count loads all agent notifications into memory | roboco/services/notification_delivery.py:379 | base_query selects all NotificationTable rows where to_agents contains agent_id with no limit, then counts in Python. For a long-running agent this row count grows unbounded; called via get_delivery_summary on the panel it is an O(n) DB read per dashboard load. Not a correctness regression from the baseline but the slice's new dedup reduces new-row growth, masking the unbounded-scan risk. | low |
|
||||
| CLOSED: get_notification_count loaded all agent notifications into memory | roboco/services/notification_delivery.py:601 | Fixed: rewritten to a single SQL `func.count()` + conditional `SUM(CASE ...)` aggregate query (no row materialization) — `total`/`unread`/`pending_ack` computed entirely in Postgres. Parity with the prior in-memory computation pinned by a >1000-row seeded test in `tests/integration/test_notification_system_list.py::test_get_notification_count_matches_in_memory_computation_at_scale`. | closed |
|
||||
| defer_bus_publish listener registration tied to session.info on the AsyncSession — session reuse hazard | roboco/services/notification_delivery.py:116 | _DRAIN_REGISTERED_KEY is set once per AsyncSession and the SQLAlchemy event.listens_for(sync_session, ...) is bound to sync_session. If an AsyncSession is reused for multiple independent transactions (connection-pool recycling), the listener stays registered and fires _schedule_pending_publishes on every subsequent commit even when no new events were deferred — _schedule_pending_publishes pops an empty queue and no-ops, so it is benign, but the listener is never removed and accumulates on the sync_session for the session's lifetime. A long-lived sync_session with many AsyncSession wraps could accumulate listeners. | low |
|
||||
| `notify_auditor_of_rework` is best-effort and not deduplicated beyond the Redis re-fire guard | roboco/services/notification_delivery.py:937 | Delivery failures are swallowed and logged by the TaskService caller so the needs_revision transition never blocks. ALERT is ack-required, so each unacked rework event persists until the auditor acks it; repeated QA/PR/PM rejects on the same task emit one ALERT per transition. | low |
|
||||
| CLOSED: `notify_auditor_of_rework` repeat ALERTs were silently suppressed by the DB purpose-dedup added for task-handoff retries | roboco/services/notification_delivery.py:1576 | Fixed: when `_persist_and_deliver` gained the DB purpose-dedup guard (see the CLOSED entry above), `notify_auditor_of_rework` inherited it unconditionally — a second `needs_revision` on the same task from the same actor was suppressed while the first ALERT sat unacked, silently starving the orchestrator's ALERT-driven `_dispatch_audit_work` on later rework cycles. Now passes `bypass_purpose_dedup=True` (same caller-scoped exemption mechanism as the re-escalation ladder below) so every rework ALERT reaches the auditor regardless of whether the prior one was acked. Delivery failures remain best-effort/swallowed by the TaskService caller so the `needs_revision` transition never blocks. | closed |
|
||||
|
||||
## Health
|
||||
This slice is substantially hardened since the baseline: the transactional-outbox for delivery (F107), the Redis re-fire guard, and the ACK_REQUIRED_BY_TYPE-driven requires_ack are all real, well-documented fixes that close prior meltdowns. The main integrity gap is dedup-path fragmentation: NotificationService._create_notification runs two dedup layers (Redis + DB purpose-dedup) while NotificationDeliveryService._persist_and_deliver runs only the Redis layer, so the task-handoff notifications (blocker/escalation/ceo-rejection) are not protected by DB purpose-dedup past the 60s Redis window — a retried i_am_blocked or escalate beyond 60s can re-create an unacked duplicate, the exact inbox-inflation + i_am_idle soft-block the DB dedup was added to prevent. A secondary consistency gap is that acknowledge publishes NOTIFICATION_ACKED directly to the bus instead of through the deferred outbox, leaving the same phantom-event class F107 fixed for deliver. Neither is a crash bug; both are correctness drift between two paths that should behave identically. Code quality is high (terse comments, clear docstrings, explicit race handling), and the slice is well-covered by the orchestrator sweeper integration and route-level callers.
|
||||
This slice is substantially hardened since the baseline: the transactional-outbox for delivery (F107), the Redis re-fire guard, and the ACK_REQUIRED_BY_TYPE-driven requires_ack are all real, well-documented fixes that close prior meltdowns. The former dedup-path fragmentation (NotificationService._create_notification ran two dedup layers while NotificationDeliveryService._persist_and_deliver ran only the Redis layer) is now closed: both paths call the shared `notification_dedup.duplicate_unacked_notification_exists`, so the task-handoff notifications (blocker/escalation/ceo-rejection) are protected by DB purpose-dedup past the 60s Redis window too. Closing that gap over-broadened in its first pass (PR #742) — it unconditionally suppressed the two callers that deliberately re-send an identical signal while the prior copy is unacked (the re-escalation ladder and rework ALERTs) — and PR #744 corrected it with a caller-scoped `bypass_purpose_dedup` exemption (see the post-snapshot update above) rather than a type-based one, so the original first-send-retried-blocker gap stays closed. The acknowledge-publishes-directly claim in an earlier snapshot of this doc was stale — `acknowledge` already used the deferred outbox (`defer_bus_publish`) before this task, confirmed by the existing rollback test. Code quality is high (terse comments, clear docstrings, explicit race handling), and the slice is well-covered by the orchestrator sweeper integration and route-level callers.
|
||||
|
||||
@@ -19,7 +19,10 @@ from roboco.db.tables import AgentTable, NotificationTable
|
||||
from roboco.foundation.policy.communications import ACK_REQUIRED_BY_TYPE
|
||||
from roboco.models import NotificationPriority, NotificationType
|
||||
from roboco.models.notification import CreateNotificationParams
|
||||
from roboco.services.notification_dedup import all_recipients_recently_notified
|
||||
from roboco.services.notification_dedup import (
|
||||
all_recipients_recently_notified,
|
||||
duplicate_unacked_notification_exists,
|
||||
)
|
||||
from roboco.services.notification_text import agent_display, task_display
|
||||
from roboco.utils.converters import require_uuid
|
||||
|
||||
@@ -890,48 +893,20 @@ class NotificationService:
|
||||
) -> bool:
|
||||
"""True when an unacked same-purpose notification already exists.
|
||||
|
||||
Purpose-based dedup (CEO directive, 2026-06-10): same sender, type,
|
||||
task, EQUAL recipient set, while a prior one is still unacked —
|
||||
agents re-send the same signal (often reworded) and each copy inflates
|
||||
the recipient's unacked set, soft-blocking i_am_idle and driving respawn
|
||||
churn. Body text is NOT compared. Dedup applies only to ACTION-REQUIRED
|
||||
types; informational carries distinct content per send and acking is
|
||||
voluntary, so deduping them would silently drop broadcasts.
|
||||
|
||||
Recipient set must be EXACTLY equal — overlapping-but-not-equal sets
|
||||
do NOT suppress. A blocker sent to {be-pm, main-pm} after an unacked
|
||||
one to {be-pm} alone must reach main-pm (the prior's recipients are a
|
||||
strict subset). Overlap is the SQL filter; the exact-set-equality check
|
||||
runs in Python against the fetched candidate rows.
|
||||
Thin wrapper over the shared primitive-typed query in
|
||||
``notification_dedup.duplicate_unacked_notification_exists`` — see
|
||||
that function's docstring for the dedup semantics (same sender +
|
||||
type + task, EXACT recipient-set equality, prior still unacked).
|
||||
Kept here so callers that already hold a ``CreateNotificationParams``
|
||||
don't have to unpack it themselves.
|
||||
"""
|
||||
related = params.related_task_id
|
||||
if not ACK_REQUIRED_BY_TYPE.get(params.notification_type, True):
|
||||
return False
|
||||
new_set = set(to_agents_uuids)
|
||||
dup_q = (
|
||||
select(NotificationTable.id, NotificationTable.to_agents)
|
||||
.where(NotificationTable.from_agent == from_agent_uuid)
|
||||
.where(NotificationTable.type == params.notification_type)
|
||||
.where(NotificationTable.to_agents.overlap(to_agents_uuids))
|
||||
.where(~NotificationTable.acked_by.contains(to_agents_uuids))
|
||||
.where(
|
||||
NotificationTable.related_task_id == related
|
||||
if related is not None
|
||||
else NotificationTable.related_task_id.is_(None)
|
||||
)
|
||||
return await duplicate_unacked_notification_exists(
|
||||
db,
|
||||
from_agent=from_agent_uuid,
|
||||
notification_type=params.notification_type,
|
||||
related_task_id=params.related_task_id,
|
||||
to_agents=to_agents_uuids,
|
||||
)
|
||||
result = await db.execute(dup_q)
|
||||
for row in result.all():
|
||||
if set(row[1]) == new_set:
|
||||
logger.info(
|
||||
"Suppressed duplicate notification (same purpose, unacked)",
|
||||
from_agent=str(from_agent_uuid),
|
||||
type=params.notification_type.value,
|
||||
related_task_id=str(related) if related is not None else None,
|
||||
to_agents=[str(a) for a in to_agents_uuids],
|
||||
)
|
||||
return True
|
||||
return False
|
||||
|
||||
async def _create_notification(
|
||||
self,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
"""Bounded re-fire guard for loop-prone notification types.
|
||||
"""Notification dedup helpers: a Redis re-fire guard plus a shared DB
|
||||
purpose-dedup query.
|
||||
|
||||
TASK_ASSIGNMENT / REVIEW_REQUEST / DOCUMENTATION_REQUEST / BROADCAST can be
|
||||
re-fired by a PM every tick while a task sits in a state, flooding inboxes.
|
||||
@@ -7,6 +8,12 @@ for these four, so a short Redis SET-NX window per (type, sender, recipient,
|
||||
task) suppresses the re-fire here. Fail-open: Redis unavailable → never
|
||||
suppress (a notification is never dropped because the dedup infra is down).
|
||||
One-shot types (KNOWLEDGE_SHARE / MENTION / A2A_REQUEST) bypass entirely.
|
||||
|
||||
``duplicate_unacked_notification_exists`` is the DB-side purpose-dedup query
|
||||
(same sender + type + task + exact recipient set, prior still unacked),
|
||||
shared by ``NotificationService._create_notification`` and
|
||||
``NotificationDeliveryService._persist_and_deliver`` so both paths apply the
|
||||
same semantics instead of each growing its own query.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -15,14 +22,18 @@ import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import redis.asyncio as redis
|
||||
from sqlalchemy import select
|
||||
|
||||
from roboco.config import settings
|
||||
from roboco.foundation.policy.communications import ACK_REQUIRED_BY_TYPE
|
||||
from roboco.models import NotificationType
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Loop-prone: a coordinator re-fires these every tick while the task sits in a
|
||||
@@ -118,3 +129,67 @@ async def clear_dedup_key(
|
||||
await conn.aclose()
|
||||
except Exception as exc:
|
||||
logger.warning("notification dedup clear failed (redis): %s", exc)
|
||||
|
||||
|
||||
async def duplicate_unacked_notification_exists(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
from_agent: UUID,
|
||||
notification_type: NotificationType,
|
||||
related_task_id: UUID | str | None,
|
||||
to_agents: Sequence[UUID],
|
||||
) -> bool:
|
||||
"""True when an unacked same-purpose notification already exists.
|
||||
|
||||
Purpose-based dedup (CEO directive, 2026-06-10): same sender, type,
|
||||
task, EQUAL recipient set, while a prior one is still unacked — agents
|
||||
re-send the same signal (often reworded) and each copy inflates the
|
||||
recipient's unacked set, soft-blocking i_am_idle and driving respawn
|
||||
churn. Body text is NOT compared. Dedup applies only to ACTION-REQUIRED
|
||||
types; informational carries distinct content per send and acking is
|
||||
voluntary, so deduping them would silently drop broadcasts.
|
||||
|
||||
Recipient set must be EXACTLY equal — overlapping-but-not-equal sets do
|
||||
NOT suppress. A blocker sent to {be-pm, main-pm} after an unacked one to
|
||||
{be-pm} alone must reach main-pm (the prior's recipients are a strict
|
||||
subset). Overlap is the SQL filter; the exact-set-equality check runs in
|
||||
Python against the fetched candidate rows.
|
||||
|
||||
Primitive-typed (db/from_agent/notification_type/related_task_id/
|
||||
to_agents) so both ``NotificationService._create_notification`` (which
|
||||
holds a ``CreateNotificationParams``) and
|
||||
``NotificationDeliveryService._persist_and_deliver`` (which only holds a
|
||||
built ``NotificationTable``, no params object) can share the one query
|
||||
instead of each growing a divergent copy.
|
||||
"""
|
||||
from roboco.db.tables import NotificationTable # local: avoid import cycle
|
||||
|
||||
if not ACK_REQUIRED_BY_TYPE.get(notification_type, True):
|
||||
return False
|
||||
to_agents_list = list(to_agents)
|
||||
new_set = set(to_agents_list)
|
||||
dup_q = (
|
||||
select(NotificationTable.id, NotificationTable.to_agents)
|
||||
.where(NotificationTable.from_agent == from_agent)
|
||||
.where(NotificationTable.type == notification_type)
|
||||
.where(NotificationTable.to_agents.overlap(to_agents_list))
|
||||
.where(~NotificationTable.acked_by.contains(to_agents_list))
|
||||
.where(
|
||||
NotificationTable.related_task_id == related_task_id
|
||||
if related_task_id is not None
|
||||
else NotificationTable.related_task_id.is_(None)
|
||||
)
|
||||
)
|
||||
result = await db.execute(dup_q)
|
||||
for row in result.all():
|
||||
if set(row[1]) == new_set:
|
||||
logger.info(
|
||||
"Suppressed duplicate notification (same purpose, unacked): "
|
||||
"from_agent=%s type=%s related_task_id=%s to_agents=%s",
|
||||
from_agent,
|
||||
notification_type.value,
|
||||
related_task_id,
|
||||
to_agents_list,
|
||||
)
|
||||
return True
|
||||
return False
|
||||
|
||||
@@ -14,12 +14,12 @@ import contextlib
|
||||
import html
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any, ClassVar, Literal, cast
|
||||
from uuid import UUID
|
||||
|
||||
import structlog
|
||||
from sqlalchemy import CursorResult, and_, event, select, update
|
||||
from sqlalchemy import CursorResult, and_, case, event, func, not_, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from roboco.agents_config import (
|
||||
@@ -40,6 +40,7 @@ from roboco.services.base import BaseService, NotFoundError
|
||||
from roboco.services.notification_dedup import (
|
||||
all_recipients_recently_notified,
|
||||
clear_dedup_key,
|
||||
duplicate_unacked_notification_exists,
|
||||
)
|
||||
from roboco.services.notification_text import task_display
|
||||
from roboco.services.repositories.query_helpers import get_agent_by_role
|
||||
@@ -410,14 +411,19 @@ class NotificationDeliveryService(BaseService):
|
||||
async def _maybe_reescalate(self, n: NotificationTable, now: datetime) -> None:
|
||||
"""Re-escalate `n` only when its backoff schedule says it's due.
|
||||
|
||||
`_persist_and_deliver`'s 60s dedup guard does NOT backstop a
|
||||
concurrent double-sweep here: `BLOCKER_ESCALATION` — the type every
|
||||
re-escalation notification is created as — is not in
|
||||
`_LOOP_PRONE_TYPES` (notification_dedup.py), so that guard returns
|
||||
False unconditionally for it. The real guard is
|
||||
`_claim_reescalation_slot`'s compare-and-set: it must succeed BEFORE
|
||||
any delivery is attempted, so two sweep ticks racing the same row
|
||||
can never both deliver.
|
||||
Neither of `_persist_and_deliver`'s two dedup guards backstops a
|
||||
concurrent double-sweep here. The Redis 60s guard is a structural
|
||||
no-op: `BLOCKER_ESCALATION` — the type every re-escalation
|
||||
notification is created as — is not in `_LOOP_PRONE_TYPES`
|
||||
(notification_dedup.py), so that guard returns False unconditionally
|
||||
for it. The DB purpose-dedup guard WOULD otherwise apply to
|
||||
`BLOCKER_ESCALATION` (it's ACK_REQUIRED_BY_TYPE), but
|
||||
`_re_escalate_recipient` deliberately passes
|
||||
`bypass_purpose_dedup=True` so a legitimate repeat re-escalation
|
||||
against a still-unacked prior row is never silently dropped. The
|
||||
real guard against double-delivery is `_claim_reescalation_slot`'s
|
||||
compare-and-set: it must succeed BEFORE any delivery is attempted,
|
||||
so two sweep ticks racing the same row can never both deliver.
|
||||
"""
|
||||
decision = reescalation_decision(
|
||||
now=now,
|
||||
@@ -523,9 +529,26 @@ class NotificationDeliveryService(BaseService):
|
||||
requires_ack=ACK_REQUIRED_BY_TYPE[NotificationType.BLOCKER_ESCALATION],
|
||||
read_by=[],
|
||||
acked_by=[],
|
||||
# Stamp the same TTL a first-send blocker gets so this row is
|
||||
# itself swept + can be re-escalated further up the chain — an
|
||||
# un-stamped row would live forever and dead-end the ladder here.
|
||||
expires_at=(
|
||||
datetime.now(UTC) + timedelta(hours=settings.notification_ack_ttl_hours)
|
||||
if settings.notification_ack_ttl_hours > 0
|
||||
else None
|
||||
),
|
||||
)
|
||||
try:
|
||||
return await self._persist_and_deliver(notification)
|
||||
# Every attempt at this recipient rebuilds an identical
|
||||
# BLOCKER_ESCALATION while the prior one is still unacked BY
|
||||
# DEFINITION (that's why we're re-escalating) — bypass DB
|
||||
# purpose-dedup so attempt 2+ isn't silently suppressed right
|
||||
# after `_claim_reescalation_slot` already burned the attempt
|
||||
# slot. The CAS claim upstream, not this dedup, is what prevents
|
||||
# a genuine double-delivery.
|
||||
return await self._persist_and_deliver(
|
||||
notification, bypass_purpose_dedup=True
|
||||
)
|
||||
except Exception as e:
|
||||
self.log.warning(
|
||||
"Re-escalation deliver failed",
|
||||
@@ -607,24 +630,34 @@ class NotificationDeliveryService(BaseService):
|
||||
Returns:
|
||||
Dict with counts: total, unread, pending_ack
|
||||
"""
|
||||
# Get all notifications for agent
|
||||
base_query = select(NotificationTable).where(
|
||||
NotificationTable.to_agents.contains([agent_id])
|
||||
# SQL COUNT aggregates — never materialize the full row set (this is
|
||||
# hit on every panel-bell/Telegram cockpit poll).
|
||||
unread_case = case(
|
||||
(not_(NotificationTable.read_by.contains([agent_id])), 1), else_=0
|
||||
)
|
||||
|
||||
result = await self.session.execute(base_query)
|
||||
notifications = list(result.scalars().all())
|
||||
|
||||
total = len(notifications)
|
||||
unread = sum(1 for n in notifications if agent_id not in n.read_by)
|
||||
pending_ack = sum(
|
||||
1 for n in notifications if n.requires_ack and agent_id not in n.acked_by
|
||||
pending_ack_case = case(
|
||||
(
|
||||
and_(
|
||||
NotificationTable.requires_ack.is_(True),
|
||||
not_(NotificationTable.acked_by.contains([agent_id])),
|
||||
),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
query = select(
|
||||
func.count().label("total"),
|
||||
func.coalesce(func.sum(unread_case), 0).label("unread"),
|
||||
func.coalesce(func.sum(pending_ack_case), 0).label("pending_ack"),
|
||||
).where(NotificationTable.to_agents.contains([agent_id]))
|
||||
|
||||
result = await self.session.execute(query)
|
||||
row = result.one()
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
"unread": unread,
|
||||
"pending_ack": pending_ack,
|
||||
"total": int(row.total),
|
||||
"unread": int(row.unread),
|
||||
"pending_ack": int(row.pending_ack),
|
||||
}
|
||||
|
||||
# =========================================================================
|
||||
@@ -1503,6 +1536,12 @@ class NotificationDeliveryService(BaseService):
|
||||
type ``ALERT`` whose ``to_agents`` include the auditor and spawns the
|
||||
auditor with a quality-alert prompt. This producer reactivates that
|
||||
reactive dispatch path at the QA-fail / rework chokepoints.
|
||||
|
||||
Bypasses DB purpose-dedup: a second ``needs_revision`` on the same
|
||||
task from the same actor is a genuine repeat rework event that must
|
||||
still reach the auditor even while the first ALERT sits unacked —
|
||||
suppressing it would silently stop `_dispatch_audit_work` from
|
||||
re-spawning the auditor on later rework cycles.
|
||||
"""
|
||||
auditor = await self._get_auditor_agent()
|
||||
if not auditor:
|
||||
@@ -1534,7 +1573,7 @@ class NotificationDeliveryService(BaseService):
|
||||
read_by=[],
|
||||
acked_by=[],
|
||||
)
|
||||
await self._persist_and_deliver(notification)
|
||||
await self._persist_and_deliver(notification, bypass_purpose_dedup=True)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers for recipient resolution + persist
|
||||
@@ -1590,17 +1629,31 @@ class NotificationDeliveryService(BaseService):
|
||||
"""Find the auditor agent (org-wide; earliest-created if many)."""
|
||||
return await get_agent_by_role(self.session, AgentRole.AUDITOR)
|
||||
|
||||
async def _persist_and_deliver(self, notification: NotificationTable) -> bool:
|
||||
async def _persist_and_deliver(
|
||||
self, notification: NotificationTable, *, bypass_purpose_dedup: bool = False
|
||||
) -> bool:
|
||||
"""Add to session, flush (to get an id), deliver. Caller commits.
|
||||
|
||||
Returns True iff actually persisted+delivered, False if suppressed by
|
||||
the 60s dedup guard below. That guard only ever applies to
|
||||
either guard below. The Redis re-fire guard only ever applies to
|
||||
`_LOOP_PRONE_TYPES` (notification_dedup.py) — BLOCKER_ESCALATION,
|
||||
the type re-escalations use, is NOT one of them, so for that path
|
||||
this always returns True or raises; the real double-delivery guard
|
||||
for re-escalations is the CAS claim in
|
||||
`NotificationDeliveryService._claim_reescalation_slot`, upstream of
|
||||
this call.
|
||||
this call. The DB purpose-dedup guard applies to ACK_REQUIRED_BY_TYPE
|
||||
action-required types (BLOCKER_ESCALATION included) — a retried
|
||||
`i_am_blocked`/escalate past the 60s Redis window still hits this.
|
||||
|
||||
`bypass_purpose_dedup=True` skips ONLY the DB purpose-dedup check
|
||||
below (never the Redis re-fire guard) for a caller whose whole point
|
||||
is to intentionally re-send an identical (sender, type, task)
|
||||
signal — `_re_escalate_recipient` (the re-escalation ladder) and
|
||||
`notify_auditor_of_rework` (rework ALERTs) both pass this, since the
|
||||
prior copy being unacked is exactly why they fire again. This is
|
||||
scoped to the CALL PATH, not the notification type, so a first-send
|
||||
retried blocker (the gap this dedup was added to close) is still
|
||||
deduped normally.
|
||||
"""
|
||||
# Re-fire guard (loop-prone types): this path skips the DB dedup, so
|
||||
# apply the same 60s Redis SET-NX window. Fail-open on Redis down.
|
||||
@@ -1623,6 +1676,35 @@ class NotificationDeliveryService(BaseService):
|
||||
else None,
|
||||
)
|
||||
return False
|
||||
# DB purpose-dedup: this path (task-handoff helpers) never went
|
||||
# through `NotificationService._create_notification`, so it never
|
||||
# got the same-purpose/unacked check that path applies. Without it,
|
||||
# a retried blocker/escalate past the Redis window above re-creates
|
||||
# a second unacked row for the same (sender, type, task, recipients).
|
||||
# Skipped entirely when the caller opted out (see docstring above).
|
||||
is_duplicate = (
|
||||
not bypass_purpose_dedup
|
||||
and notification.from_agent is not None
|
||||
and (
|
||||
await duplicate_unacked_notification_exists(
|
||||
self.session,
|
||||
from_agent=cast("UUID", notification.from_agent),
|
||||
notification_type=notification.type,
|
||||
related_task_id=cast("UUID | None", notification.related_task_id),
|
||||
to_agents=cast("list[UUID]", notification.to_agents),
|
||||
)
|
||||
)
|
||||
)
|
||||
if is_duplicate:
|
||||
_log.info(
|
||||
"Suppressed duplicate notification (same purpose, unacked)",
|
||||
from_agent=str(notification.from_agent),
|
||||
type=notification.type.value if notification.type is not None else None,
|
||||
related_task_id=str(notification.related_task_id)
|
||||
if notification.related_task_id is not None
|
||||
else None,
|
||||
)
|
||||
return False
|
||||
self.session.add(notification)
|
||||
await self.session.flush()
|
||||
await self.deliver(require_uuid(notification.id))
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
"""NotificationDeliveryService._persist_and_deliver DB purpose-dedup.
|
||||
|
||||
`_persist_and_deliver` (used by the task-handoff helpers: notify_pm_of_block,
|
||||
escalate_and_notify, notify_assignee_of_unblock, etc.) used to run ONLY the
|
||||
60s Redis re-fire guard and skip DB purpose-dedup entirely — a retried
|
||||
`i_am_blocked` or a re-issued escalate past the 60s window re-created a
|
||||
second unacked notification for the same (sender, type, task, recipients).
|
||||
|
||||
These integration tests seed an unacked notification directly (mirroring an
|
||||
already-persisted prior send), then drive `_persist_and_deliver` with a
|
||||
duplicate-shaped notification while bypassing the Redis guard (monkeypatched
|
||||
to return False, matching tests/unit/test_notification_delivery_refire.py),
|
||||
and assert no second row is created while the first stays unacked.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, cast
|
||||
from unittest.mock import AsyncMock
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from roboco.db.tables import AgentTable, NotificationTable, ProjectTable, TaskTable
|
||||
from roboco.models import AgentRole, AgentStatus, NotificationPriority, NotificationType
|
||||
from roboco.models.base import TaskNature, TaskStatus, TaskType, Team
|
||||
from roboco.services.notification_delivery import get_notification_delivery_service
|
||||
from sqlalchemy import select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
async def _make_agent(db: AsyncSession, *, role: AgentRole) -> AgentTable:
|
||||
agent = AgentTable(
|
||||
id=uuid4(),
|
||||
name=f"{role.value}-{uuid4().hex[:6]}",
|
||||
slug=f"{role.value}-{uuid4().hex[:8]}",
|
||||
role=role,
|
||||
team=Team.BACKEND,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="agent",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
db.add(agent)
|
||||
await db.flush()
|
||||
return agent
|
||||
|
||||
|
||||
async def _make_task(db: AsyncSession, *, created_by: UUID) -> TaskTable:
|
||||
"""Real Project + Task rows — notifications.related_task_id FKs to tasks."""
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:8]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=created_by,
|
||||
)
|
||||
db.add(project)
|
||||
await db.flush()
|
||||
task = TaskTable(
|
||||
id=uuid4(),
|
||||
title="Blocked task",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
status=TaskStatus.BLOCKED,
|
||||
priority=2,
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
project_id=project.id,
|
||||
created_by=created_by,
|
||||
team=Team.BACKEND,
|
||||
)
|
||||
db.add(task)
|
||||
await db.flush()
|
||||
return task
|
||||
|
||||
|
||||
async def _count_matching(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
from_agent: UUID,
|
||||
notification_type: NotificationType,
|
||||
related_task_id: UUID,
|
||||
) -> int:
|
||||
result = await db.execute(
|
||||
select(NotificationTable).where(
|
||||
NotificationTable.from_agent == from_agent,
|
||||
NotificationTable.type == notification_type,
|
||||
NotificationTable.related_task_id == related_task_id,
|
||||
)
|
||||
)
|
||||
return len(result.scalars().all())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_persist_and_deliver_suppresses_db_duplicate_past_redis_window(
|
||||
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A retried blocker escalation past the 60s Redis window must not
|
||||
create a second unacked row for the same (sender, type, task,
|
||||
recipients) — the DB purpose-dedup gap `_persist_and_deliver` used to
|
||||
skip entirely."""
|
||||
sender = await _make_agent(db_session, role=AgentRole.DEVELOPER)
|
||||
pm = await _make_agent(db_session, role=AgentRole.CELL_PM)
|
||||
task = await _make_task(db_session, created_by=cast("UUID", sender.id))
|
||||
task_id = cast("UUID", task.id)
|
||||
|
||||
# Seed the prior, still-unacked notification for this exact purpose.
|
||||
prior = NotificationTable(
|
||||
type=NotificationType.BLOCKER_ESCALATION,
|
||||
priority=NotificationPriority.HIGH,
|
||||
from_agent=cast("UUID", sender.id),
|
||||
to_agents=[cast("UUID", pm.id)],
|
||||
subject="ACTION REQUIRED: Blocked - task",
|
||||
body="first send",
|
||||
related_task_id=task_id,
|
||||
requires_ack=True,
|
||||
read_by=[],
|
||||
acked_by=[],
|
||||
)
|
||||
db_session.add(prior)
|
||||
await db_session.flush()
|
||||
|
||||
# Bypass the 60s Redis re-fire guard so the DB check is the only guard
|
||||
# left standing — matches tests/unit/test_notification_delivery_refire.py.
|
||||
monkeypatch.setattr(
|
||||
"roboco.services.notification_delivery.all_recipients_recently_notified",
|
||||
AsyncMock(return_value=False),
|
||||
)
|
||||
|
||||
service = get_notification_delivery_service(db_session)
|
||||
deliver_mock = AsyncMock()
|
||||
monkeypatch.setattr(service, "deliver", deliver_mock)
|
||||
|
||||
retry = NotificationTable(
|
||||
type=NotificationType.BLOCKER_ESCALATION,
|
||||
priority=NotificationPriority.HIGH,
|
||||
from_agent=cast("UUID", sender.id),
|
||||
to_agents=[cast("UUID", pm.id)],
|
||||
subject="ACTION REQUIRED: Blocked - task (retry)",
|
||||
body="retried send",
|
||||
related_task_id=task_id,
|
||||
requires_ack=True,
|
||||
read_by=[],
|
||||
acked_by=[],
|
||||
)
|
||||
|
||||
persisted = await service._persist_and_deliver(retry)
|
||||
|
||||
assert persisted is False
|
||||
deliver_mock.assert_not_awaited()
|
||||
count = await _count_matching(
|
||||
db_session,
|
||||
from_agent=cast("UUID", sender.id),
|
||||
notification_type=NotificationType.BLOCKER_ESCALATION,
|
||||
related_task_id=task_id,
|
||||
)
|
||||
assert count == 1 # only the seeded prior — the retry was suppressed
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_persist_and_deliver_allows_distinct_recipient_set(
|
||||
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A strictly-different recipient set is NOT a duplicate — overlap alone
|
||||
must not suppress (only exact-set equality does)."""
|
||||
sender = await _make_agent(db_session, role=AgentRole.DEVELOPER)
|
||||
pm = await _make_agent(db_session, role=AgentRole.CELL_PM)
|
||||
main_pm = await _make_agent(db_session, role=AgentRole.MAIN_PM)
|
||||
task = await _make_task(db_session, created_by=cast("UUID", sender.id))
|
||||
task_id = cast("UUID", task.id)
|
||||
|
||||
prior = NotificationTable(
|
||||
type=NotificationType.BLOCKER_ESCALATION,
|
||||
priority=NotificationPriority.HIGH,
|
||||
from_agent=cast("UUID", sender.id),
|
||||
to_agents=[cast("UUID", pm.id)],
|
||||
subject="ACTION REQUIRED: Blocked - task",
|
||||
body="first send",
|
||||
related_task_id=task_id,
|
||||
requires_ack=True,
|
||||
read_by=[],
|
||||
acked_by=[],
|
||||
)
|
||||
db_session.add(prior)
|
||||
await db_session.flush()
|
||||
|
||||
monkeypatch.setattr(
|
||||
"roboco.services.notification_delivery.all_recipients_recently_notified",
|
||||
AsyncMock(return_value=False),
|
||||
)
|
||||
|
||||
service = get_notification_delivery_service(db_session)
|
||||
monkeypatch.setattr(service, "deliver", AsyncMock())
|
||||
|
||||
escalated = NotificationTable(
|
||||
type=NotificationType.BLOCKER_ESCALATION,
|
||||
priority=NotificationPriority.HIGH,
|
||||
from_agent=cast("UUID", sender.id),
|
||||
to_agents=[cast("UUID", pm.id), cast("UUID", main_pm.id)],
|
||||
subject="ACTION REQUIRED: Blocked - task (escalated)",
|
||||
body="re-escalated to main-pm too",
|
||||
related_task_id=task_id,
|
||||
requires_ack=True,
|
||||
read_by=[],
|
||||
acked_by=[],
|
||||
)
|
||||
|
||||
persisted = await service._persist_and_deliver(escalated)
|
||||
|
||||
assert persisted is True
|
||||
count = await _count_matching(
|
||||
db_session,
|
||||
from_agent=cast("UUID", sender.id),
|
||||
notification_type=NotificationType.BLOCKER_ESCALATION,
|
||||
related_task_id=task_id,
|
||||
)
|
||||
expected_rows = 2 # the prior + the distinct-recipient-set escalation
|
||||
assert count == expected_rows
|
||||
@@ -0,0 +1,187 @@
|
||||
"""_re_escalate_recipient must not be suppressed by its own prior attempt.
|
||||
|
||||
PR #742's DB purpose-dedup (`duplicate_unacked_notification_exists`) closed
|
||||
the retried-first-send-blocker gap, but applied unconditionally inside
|
||||
`_persist_and_deliver` — so a SECOND re-escalation attempt at the same
|
||||
recipient (identical sender/type/task/recipient-set, and the prior attempt
|
||||
is unacked BY DEFINITION) was silently suppressed right after
|
||||
`_claim_reescalation_slot` had already burned the attempt slot, permanently
|
||||
stalling the re-escalation ladder.
|
||||
|
||||
`_re_escalate_recipient` now passes `bypass_purpose_dedup=True` into
|
||||
`_persist_and_deliver`, so this dedup check never applies to this call path.
|
||||
This test drives `_re_escalate_recipient` twice against a real db_session,
|
||||
seeding nothing but real agent rows, and asserts both attempts persist and
|
||||
deliver a row rather than the second being silently dropped.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, cast
|
||||
from unittest.mock import AsyncMock
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from roboco.db.tables import AgentTable, NotificationTable, ProjectTable, TaskTable
|
||||
from roboco.models import AgentRole, AgentStatus, NotificationPriority, NotificationType
|
||||
from roboco.models.base import TaskNature, TaskStatus, TaskType, Team
|
||||
from roboco.services.notification_delivery import get_notification_delivery_service
|
||||
from sqlalchemy import select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
async def _make_agent(db: AsyncSession, *, slug: str, role: AgentRole) -> AgentTable:
|
||||
agent = AgentTable(
|
||||
id=uuid4(),
|
||||
name=slug,
|
||||
slug=slug,
|
||||
role=role,
|
||||
team=None,
|
||||
status=AgentStatus.ACTIVE,
|
||||
model_config={},
|
||||
system_prompt="agent",
|
||||
capabilities=[],
|
||||
permissions={},
|
||||
metrics={},
|
||||
)
|
||||
db.add(agent)
|
||||
await db.flush()
|
||||
return agent
|
||||
|
||||
|
||||
async def _make_task(db: AsyncSession, *, created_by: UUID) -> TaskTable:
|
||||
"""Real Project + Task rows — notifications.related_task_id FKs to tasks."""
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name="P",
|
||||
slug=f"p-{uuid4().hex[:8]}",
|
||||
git_url="https://example.com/r.git",
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=created_by,
|
||||
)
|
||||
db.add(project)
|
||||
await db.flush()
|
||||
task = TaskTable(
|
||||
id=uuid4(),
|
||||
title="Blocked task",
|
||||
description="d",
|
||||
acceptance_criteria=["ac"],
|
||||
status=TaskStatus.BLOCKED,
|
||||
priority=2,
|
||||
task_type=TaskType.CODE,
|
||||
nature=TaskNature.TECHNICAL,
|
||||
project_id=project.id,
|
||||
created_by=created_by,
|
||||
team=Team.BACKEND,
|
||||
)
|
||||
db.add(task)
|
||||
await db.flush()
|
||||
return task
|
||||
|
||||
|
||||
async def _count_matching(
|
||||
db: AsyncSession, *, from_agent: UUID, related_task_id: UUID
|
||||
) -> int:
|
||||
result = await db.execute(
|
||||
select(NotificationTable).where(
|
||||
NotificationTable.from_agent == from_agent,
|
||||
NotificationTable.type == NotificationType.BLOCKER_ESCALATION,
|
||||
NotificationTable.related_task_id == related_task_id,
|
||||
)
|
||||
)
|
||||
return len(result.scalars().all())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_two_sequential_due_reescalations_both_deliver(
|
||||
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Two sequential re-escalation attempts at the same recipient (the shape
|
||||
a real ladder produces on successive DUE sweep ticks) must both persist +
|
||||
deliver — not just the first."""
|
||||
# Random (not canonical be-pm/main-pm) slugs: this DB already carries the
|
||||
# real seeded fleet agents, so a literal canonical slug would collide on
|
||||
# the unique index. get_escalation_target is monkeypatched below since it
|
||||
# keys off the real ESCALATION_CHAIN's literal canonical slugs.
|
||||
sender = await _make_agent(
|
||||
db_session, slug=f"dev-{uuid4().hex[:8]}", role=AgentRole.DEVELOPER
|
||||
)
|
||||
recipient = await _make_agent(
|
||||
db_session, slug=f"pm-{uuid4().hex[:8]}", role=AgentRole.CELL_PM
|
||||
)
|
||||
target = await _make_agent(
|
||||
db_session, slug=f"main-{uuid4().hex[:8]}", role=AgentRole.MAIN_PM
|
||||
)
|
||||
task = await _make_task(db_session, created_by=cast("UUID", sender.id))
|
||||
task_id = cast("UUID", task.id)
|
||||
|
||||
original = NotificationTable(
|
||||
type=NotificationType.BLOCKER_ESCALATION,
|
||||
priority=NotificationPriority.HIGH,
|
||||
from_agent=cast("UUID", sender.id),
|
||||
to_agents=[cast("UUID", recipient.id)],
|
||||
subject="ACTION REQUIRED: Blocked - task",
|
||||
body="first send",
|
||||
related_task_id=task_id,
|
||||
requires_ack=True,
|
||||
read_by=[],
|
||||
acked_by=[],
|
||||
)
|
||||
db_session.add(original)
|
||||
await db_session.flush()
|
||||
|
||||
monkeypatch.setattr(
|
||||
"roboco.services.notification_delivery.all_recipients_recently_notified",
|
||||
AsyncMock(return_value=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"roboco.services.notification_delivery.get_escalation_target",
|
||||
lambda slug: target.slug if slug == recipient.slug else None,
|
||||
)
|
||||
|
||||
service = get_notification_delivery_service(db_session)
|
||||
monkeypatch.setattr(service, "deliver", AsyncMock())
|
||||
|
||||
first_ok = await service._re_escalate_recipient(
|
||||
original, cast("UUID", recipient.id)
|
||||
)
|
||||
second_ok = await service._re_escalate_recipient(
|
||||
original, cast("UUID", recipient.id)
|
||||
)
|
||||
|
||||
assert first_ok is True
|
||||
assert second_ok is True, (
|
||||
"second DUE re-escalation must still deliver — it must not be "
|
||||
"suppressed by the first, unacked, re-escalation row"
|
||||
)
|
||||
count = await _count_matching(
|
||||
db_session, from_agent=cast("UUID", sender.id), related_task_id=task_id
|
||||
)
|
||||
expected_total_rows = 3 # the seeded original + both re-escalation attempts
|
||||
assert count == expected_total_rows
|
||||
# Both re-escalation rows must be addressed to target.id (the escalation
|
||||
# target), proving the ladder re-fired to the right recipient — not a
|
||||
# tautology on target.id which was uuid4() at construction.
|
||||
all_rows = (
|
||||
(
|
||||
await db_session.execute(
|
||||
select(NotificationTable).where(
|
||||
NotificationTable.from_agent == cast("UUID", sender.id),
|
||||
NotificationTable.type == NotificationType.BLOCKER_ESCALATION,
|
||||
NotificationTable.related_task_id == task_id,
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
re_escalation_rows = [
|
||||
r for r in all_rows if cast("UUID", target.id) in (r.to_agents or [])
|
||||
]
|
||||
expected_re_escalation_rows = 2 # one per _re_escalate_recipient call
|
||||
assert len(re_escalation_rows) == expected_re_escalation_rows, (
|
||||
f"expected {expected_re_escalation_rows} re-escalation rows addressed "
|
||||
f"to target ({target.slug}), got {len(re_escalation_rows)}"
|
||||
)
|
||||
@@ -151,6 +151,63 @@ async def test_pending_ack_only_slices_to_limit(db_session: AsyncSession) -> Non
|
||||
assert len(result) == limit
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_notification_count_matches_in_memory_computation_at_scale(
|
||||
db_session: AsyncSession,
|
||||
) -> None:
|
||||
"""SQL COUNT rewrite must match the old in-memory sum() for a large, mixed seed.
|
||||
|
||||
Seeds >1000 rows for one agent, mixing read/unread, acked/unacked, and
|
||||
requires_ack true/false so all three counters exercise non-trivial values,
|
||||
then asserts the service's result equals what the OLD Python-side
|
||||
computation (``len(rows)`` / ``sum(agent_id not in read_by)`` /
|
||||
``sum(requires_ack and agent_id not in acked_by)``) would produce.
|
||||
"""
|
||||
sender_id = await _seed_sender(db_session)
|
||||
recipient_id = await _seed_recipient(db_session)
|
||||
other_id = uuid4() # noise recipient — must not affect recipient_id's counts
|
||||
|
||||
base = datetime(2026, 4, 1, tzinfo=UTC)
|
||||
rows: list[NotificationTable] = []
|
||||
seed_size = 1050
|
||||
for i in range(seed_size):
|
||||
read = i % 2 == 0
|
||||
acked = i % 3 == 0
|
||||
requires_ack = i % 5 != 0
|
||||
n = NotificationTable(
|
||||
type=NotificationType.REVIEW_REQUEST,
|
||||
priority=NotificationPriority.NORMAL,
|
||||
from_agent=sender_id,
|
||||
to_agents=[recipient_id, other_id] if i % 7 == 0 else [recipient_id],
|
||||
subject=f"Notification {i}",
|
||||
body="Body text",
|
||||
requires_ack=requires_ack,
|
||||
acked_by=[recipient_id] if acked else [],
|
||||
read_by=[recipient_id] if read else [],
|
||||
timestamp=base + timedelta(seconds=i),
|
||||
)
|
||||
rows.append(n)
|
||||
db_session.add_all(rows)
|
||||
await db_session.flush()
|
||||
|
||||
expected_total = len(rows)
|
||||
expected_unread = sum(1 for n in rows if recipient_id not in n.read_by)
|
||||
expected_pending_ack = sum(
|
||||
1 for n in rows if n.requires_ack and recipient_id not in n.acked_by
|
||||
)
|
||||
assert expected_unread not in (0, expected_total)
|
||||
assert expected_pending_ack not in (0, expected_total)
|
||||
|
||||
service = get_notification_delivery_service(db_session)
|
||||
counts = await service.get_notification_count(recipient_id)
|
||||
|
||||
assert counts == {
|
||||
"total": expected_total,
|
||||
"unread": expected_unread,
|
||||
"pending_ack": expected_pending_ack,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_pending_branch_keeps_sql_limit(db_session: AsyncSession) -> None:
|
||||
"""Without pending_ack_only the SQL limit still bounds the result."""
|
||||
|
||||
@@ -116,6 +116,48 @@ async def test_notify_auditor_of_rework_creates_alert_to_auditor(
|
||||
assert notification.requires_ack is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_notify_auditor_of_rework_not_suppressed_by_existing_duplicate(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A second `needs_revision` on the same task from the same actor must
|
||||
still reach the auditor even while the first ALERT is unacked (PR #742's
|
||||
unconditional DB dedup silently suppressed this). `notify_auditor_of_rework`
|
||||
passes `bypass_purpose_dedup=True`, so the dedup check is never even
|
||||
consulted for this call path."""
|
||||
auditor = _mock_agent(role="auditor", slug="auditor")
|
||||
actor = _mock_agent(role="qa", slug="be-qa")
|
||||
session = _session_with_agent(auditor)
|
||||
svc = NotificationDeliveryService(session)
|
||||
monkeypatch.setattr(svc, "_get_auditor_agent", AsyncMock(return_value=auditor))
|
||||
monkeypatch.setattr(svc, "_get_agent_by_id", AsyncMock(return_value=actor))
|
||||
monkeypatch.setattr(svc, "deliver", AsyncMock(return_value=True))
|
||||
dedup_spy = AsyncMock(return_value=True) # would suppress if ever consulted
|
||||
|
||||
task = _mock_task()
|
||||
with (
|
||||
patch(
|
||||
"roboco.services.notification_delivery.all_recipients_recently_notified",
|
||||
AsyncMock(return_value=False),
|
||||
),
|
||||
patch(
|
||||
"roboco.services.notification_delivery.duplicate_unacked_notification_exists",
|
||||
dedup_spy,
|
||||
),
|
||||
):
|
||||
await svc.notify_auditor_of_rework(
|
||||
task=task,
|
||||
task_id=require_uuid(task.id),
|
||||
reason="second rework on the same task",
|
||||
actor_agent_id=actor.id,
|
||||
actor_role="qa",
|
||||
)
|
||||
|
||||
dedup_spy.assert_not_called()
|
||||
added = [c for c in session.add.call_args_list if c.args]
|
||||
assert len(added) == 1, "rework ALERT must deliver despite an unacked duplicate"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_notify_auditor_of_rework_skips_when_no_auditor(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
|
||||
@@ -11,11 +11,14 @@ re-escalate on *every* sweep tick (~1min) forever. `reescalation_decision`
|
||||
(pure, in `foundation/policy/communications.py`) gates each tick behind a
|
||||
per-notification exponential schedule + a hard retry cap.
|
||||
|
||||
Double-delivery race: `_persist_and_deliver`'s 60s dedup guard is a no-op for
|
||||
`BLOCKER_ESCALATION` (not in `_LOOP_PRONE_TYPES`), so it can't backstop two
|
||||
concurrent sweep ticks racing the same stale row — a compare-and-set claim
|
||||
(`_claim_reescalation_slot`) is the real guard, exercised below by racing two
|
||||
service instances against the same row.
|
||||
Double-delivery race: neither of `_persist_and_deliver`'s dedup guards
|
||||
backstops two concurrent sweep ticks racing the same stale row — the 60s
|
||||
Redis guard is a no-op for `BLOCKER_ESCALATION` (not in `_LOOP_PRONE_TYPES`),
|
||||
and the DB purpose-dedup guard is deliberately bypassed for this call path
|
||||
(`bypass_purpose_dedup=True`) so a legitimate repeat re-escalation is never
|
||||
silently dropped. A compare-and-set claim (`_claim_reescalation_slot`) is the
|
||||
real guard, exercised below by racing two service instances against the
|
||||
same row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -35,6 +38,10 @@ from roboco.models import NotificationPriority, NotificationType
|
||||
from roboco.services.notification_delivery import NotificationDeliveryService
|
||||
from sqlalchemy import Update
|
||||
|
||||
# duplicate_unacked_notification_exists' dedup SELECT names exactly 2
|
||||
# columns (id, to_agents) — distinct from the sweep's whole-entity SELECT.
|
||||
_DEDUP_SELECT_COLUMN_COUNT = 2
|
||||
|
||||
|
||||
def _stale_notification(
|
||||
*,
|
||||
@@ -106,12 +113,27 @@ def _assign_id_on_add(obj: Any) -> None:
|
||||
|
||||
|
||||
def _session_returning(
|
||||
notifications: list[MagicMock], *, claim_succeeds: bool = True
|
||||
notifications: list[MagicMock],
|
||||
*,
|
||||
claim_succeeds: bool = True,
|
||||
dedup_rows: list[tuple[UUID, list[UUID]]] | None = None,
|
||||
) -> MagicMock:
|
||||
"""A session whose SELECT (the sweep's stale-notifications query) returns
|
||||
`notifications`; every re-escalation CAS UPDATE (`_claim_reescalation_slot`)
|
||||
reports 1 row affected — the claim wins — unless `claim_succeeds` is False,
|
||||
simulating a concurrent sweep tick that already claimed this row's slot."""
|
||||
simulating a concurrent sweep tick that already claimed this row's slot.
|
||||
|
||||
`duplicate_unacked_notification_exists` (called from `_persist_and_deliver`)
|
||||
runs its OWN SELECT — `select(NotificationTable.id, NotificationTable.to_agents)`
|
||||
— and reads it via `result.all()` directly, not `.scalars().all()` like the
|
||||
sweep query above. The two are told apart by column count (the sweep query
|
||||
selects the whole mapped entity; the dedup query selects exactly 2 columns)
|
||||
so each gets its own configured mock result — `.all()` on a bare MagicMock
|
||||
silently yields an empty iterator regardless of what's configured on
|
||||
`.scalars().all()`, which is exactly how a real "existing duplicate" could
|
||||
go unexercised by this suite. `dedup_rows` defaults to `[]` (no existing
|
||||
duplicate); pass rows to simulate one and assert the exemption still
|
||||
delivers."""
|
||||
session = MagicMock()
|
||||
session.add = MagicMock(side_effect=_assign_id_on_add)
|
||||
session.flush = AsyncMock()
|
||||
@@ -119,11 +141,18 @@ def _session_returning(
|
||||
select_result = MagicMock()
|
||||
select_result.scalars.return_value.all.return_value = notifications
|
||||
|
||||
dedup_result = MagicMock()
|
||||
dedup_result.all.return_value = dedup_rows or []
|
||||
|
||||
update_result = MagicMock()
|
||||
update_result.rowcount = 1 if claim_succeeds else 0
|
||||
|
||||
async def _execute(statement: Any, *_args: Any, **_kwargs: Any) -> MagicMock:
|
||||
return update_result if isinstance(statement, Update) else select_result
|
||||
if isinstance(statement, Update):
|
||||
return update_result
|
||||
if len(list(statement.selected_columns)) == _DEDUP_SELECT_COLUMN_COUNT:
|
||||
return dedup_result
|
||||
return select_result
|
||||
|
||||
session.execute = AsyncMock(side_effect=_execute)
|
||||
return session
|
||||
@@ -166,6 +195,43 @@ async def test_sweep_re_escalates_stale_unacked_ack_required() -> None:
|
||||
assert notif.reescalation_delivered_count == 1 # the attempt was delivered
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweep_re_escalation_not_suppressed_by_existing_unacked_duplicate() -> (
|
||||
None
|
||||
):
|
||||
"""A prior unacked re-escalation to the SAME target (same sender/type/task
|
||||
/recipient-set — exactly what `duplicate_unacked_notification_exists`
|
||||
matches on) must NOT suppress this attempt. `_re_escalate_recipient`
|
||||
passes `bypass_purpose_dedup=True`, so the DB purpose-dedup guard is
|
||||
skipped for this call path even though a matching row exists — this is
|
||||
the regression PR #742's unconditional dedup introduced."""
|
||||
recipient = _agent("be-pm")
|
||||
target = _agent("main-pm")
|
||||
notif = _stale_notification(
|
||||
requires_ack=True, acked=False, recipient_id=recipient.id
|
||||
)
|
||||
|
||||
session = _session_returning([notif], dedup_rows=[(uuid4(), [target.id])])
|
||||
svc = _svc_with_agents(session, recipient=recipient, escalation_target=target)
|
||||
|
||||
with (
|
||||
patch(
|
||||
"roboco.services.notification_delivery.all_recipients_recently_notified",
|
||||
AsyncMock(return_value=False),
|
||||
),
|
||||
patch(
|
||||
"roboco.services.notification_delivery.get_escalation_target",
|
||||
return_value="main-pm",
|
||||
),
|
||||
):
|
||||
count = await svc.sweep_expired_notifications()
|
||||
|
||||
assert count == 1
|
||||
added = [c for c in session.add.call_args_list if c.args]
|
||||
assert added, "re-escalation must still deliver despite the existing duplicate"
|
||||
assert notif.reescalation_delivered_count == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweep_does_not_re_escalate_already_acked() -> None:
|
||||
"""Ack-required + past threshold + fully acked → no re-escalation, count 0."""
|
||||
@@ -261,12 +327,15 @@ async def test_sweep_skips_re_escalation_when_no_chain_target() -> None:
|
||||
@pytest.mark.asyncio
|
||||
async def test_sweep_cas_claim_prevents_double_delivery_race() -> None:
|
||||
"""Two service instances (simulating two concurrent sweep ticks) race the
|
||||
same stale row. `_persist_and_deliver`'s 60s dedup guard cannot arbitrate
|
||||
this — BLOCKER_ESCALATION isn't a `_LOOP_PRONE_TYPES` member, so it's a
|
||||
no-op for this path. The CAS claim in `_claim_reescalation_slot` is what
|
||||
actually decides it: exactly one instance wins the guarded UPDATE and
|
||||
delivers; the loser (0 rows updated) skips delivery entirely, without
|
||||
raising."""
|
||||
same stale row. Neither of `_persist_and_deliver`'s dedup guards
|
||||
arbitrates this: the 60s Redis guard is a structural no-op for
|
||||
BLOCKER_ESCALATION (not a `_LOOP_PRONE_TYPES` member), and the DB
|
||||
purpose-dedup guard is deliberately bypassed by
|
||||
`_re_escalate_recipient` (`bypass_purpose_dedup=True`) so a legitimate
|
||||
repeat re-escalation is never silently dropped. The CAS claim in
|
||||
`_claim_reescalation_slot` is what actually decides it: exactly one
|
||||
instance wins the guarded UPDATE and delivers; the loser (0 rows
|
||||
updated) skips delivery entirely, without raising."""
|
||||
recipient = _agent("be-pm")
|
||||
target = _agent("main-pm")
|
||||
notif = _stale_notification(
|
||||
|
||||
Reference in New Issue
Block a user