[16d9a12f] Backend: close notification dedup, ACK, and count gaps (#742)

* [0d515123] fix(notification): DB purpose-dedup on _persist_and_deliver task-handoff path (#719)

Extract the shared dedup query (same sender+type+task, exact recipient-set
equality, prior still unacked) into notification_dedup.duplicate_unacked_notification_exists
and call it from both NotificationService._duplicate_unacked_exists and
NotificationDeliveryService._persist_and_deliver, closing the gap where a
retried i_am_blocked/escalate past the 60s Redis window re-created a second
unacked notification. Adds an integration test proving the suppression, and
corrects docs/map/notification.md's stale claim about the ACK path (it was
already using the transactional outbox before this task).

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>

* [ec7e3986] fix(notification): SQL COUNT aggregates for get_notification_count instead of in-memory scan (#728)

Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>

* [532b7162] Fix over-broad dedup: exempt re-escalation ladder + rework ALERTs; fix tests/docs (#744)

* [532b7162] fix(notifications): exempt re-escalation ladder + rework ALERTs from DB dedup

The DB purpose-dedup added inside NotificationDeliveryService._persist_and_deliver
was applied unconditionally, silently suppressing two paths that intentionally
re-send an identical (sender, type, task) signal: the blocker re-escalation
ladder (_re_escalate_recipient) and rework ALERTs (notify_auditor_of_rework).
Both now pass a caller-scoped bypass_purpose_dedup=True flag instead of
exempting by notification type, which would have reopened the original
retried-first-send-blocker gap. Also stamps expires_at on re-escalation rows
so the ladder can keep expiring/escalating, corrects the now-inaccurate
_maybe_reescalate docstring, fixes the sweeper test's dedup-SELECT mock blind
spot, and adds a real-db_session integration test proving two sequential
re-escalations both deliver.

* [532b7162] docs(notification): correct dedup-suppression claims for re-escalation + rework ALERT exemption

---------

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

* [4593a7d3] Fix PR-gate findings: scope DB dedup off re-escalation ladder + rework ALERTs (#780)

* [7117996c] docs(changelog): document the re-escalation ladder and rework ALERT dedup exemption fix (#777)

Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>

* [e06dc4d9] Fix CI-red quality gate + tautological test assertion on PR #780 (#798)

* [e06dc4d9] fix: regenerate stale lifecycle artifacts + replace tautological assertion

The CI 'Python quality gate' (make quality) failed on PR #780's head
because foundation-check detected lifecycle-artifact drift: the committed
panel/lib/lifecycle.json and docs/rag/lifecycle/status-transitions.md
still carried an `awaiting_pm_review -> claimed` claim transition that the
current lifecycle spec no longer emits. make gate (format/lint/mypy/xenon)
does not run foundation-check, so the reviewer could not reproduce locally.
Fix: ran `make lifecycle` to regenerate the artifacts from the spec and
commit the diff — no hand-written spec change, just stale generated output
brought current.

Second finding: tests/integration/test_notification_reescalation_dedup_exemption.py:164
had a tautological `assert target.id is not None` (target.id was uuid4() at
construction, so it could never fail). Replaced with a real DB query that
fetches all BLOCKER_ESCALATION rows for the sender/task, filters to those
addressed to target.id, and asserts exactly 2 re-escalation rows exist —
proving both _re_escalate_recipient calls delivered to the right recipient.

Verified: make gate green (format/lint/mypy/xenon); quality-fast suite
shows 7779 passed (same count as pre-change), the only failure is the
pre-existing local-only cloud_auth test (needs localhost:5432 which CI
provides); no regression in the notification dedup tests.

* [e06dc4d9] revert: restore lifecycle artifacts that prior round removed via wrong spec

The prior commit (ca84df26) regenerated panel/lib/lifecycle.json and
docs/rag/lifecycle/status-transitions.md using the MAIN checkout's
editable install (.pth → /data/workspaces/roboco-api/backend/be-dev-1)
which lacks the AWAITING_PM_REVIEW→CLAIMED transition, instead of the
worktree branch spec (lifecycle.py:250-255) which has it. This removed
the transition from both committed artifacts, introducing NEW drift that
CI's foundation-check would catch (regenerate WITH the transition from
the branch spec → git diff vs committed → FAIL).

Fix: restore the awaiting_pm_review→claimed claim transition row in
status-transitions.md and the claim_rules + transition entries in
lifecycle.json, reverting to the pre-ca84df26 state that already matched
the branch spec. The tautological assertion fix from the prior commit is
kept unchanged (QA confirmed correct).

Verified against the CORRECT (worktree) spec via PYTHONPATH override:
- make gate green (format/lint/mypy/xenon)
- pytest 15347 passed, coverage 93.83% (>80%)
- prose/vulture/deptry/imports/bandit/radon/alembic/pip-audit all pass
- foundation-check passes once committed (artifacts match branch spec)
- only failure is the local-only cloud_auth test (needs localhost:5432,
  which CI provides)

* [e06dc4d9] docs(notification): strengthen re-escalation dedup test description to reflect real assertion

---------

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

---------

Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

---------

Co-authored-by: roboco-app[bot] <302741806+roboco-app[bot]@users.noreply.github.com>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

tests/integration/test_notification_delivery_persist_dedup.py

@@ -0,0 +1,223 @@
+"""NotificationDeliveryService._persist_and_deliver DB purpose-dedup.
+
+`_persist_and_deliver` (used by the task-handoff helpers: notify_pm_of_block,
+escalate_and_notify, notify_assignee_of_unblock, etc.) used to run ONLY the
+60s Redis re-fire guard and skip DB purpose-dedup entirely — a retried
+`i_am_blocked` or a re-issued escalate past the 60s window re-created a
+second unacked notification for the same (sender, type, task, recipients).
+
+These integration tests seed an unacked notification directly (mirroring an
+already-persisted prior send), then drive `_persist_and_deliver` with a
+duplicate-shaped notification while bypassing the Redis guard (monkeypatched
+to return False, matching tests/unit/test_notification_delivery_refire.py),
+and assert no second row is created while the first stays unacked.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, cast
+from unittest.mock import AsyncMock
+from uuid import UUID, uuid4
+
+import pytest
+from roboco.db.tables import AgentTable, NotificationTable, ProjectTable, TaskTable
+from roboco.models import AgentRole, AgentStatus, NotificationPriority, NotificationType
+from roboco.models.base import TaskNature, TaskStatus, TaskType, Team
+from roboco.services.notification_delivery import get_notification_delivery_service
+from sqlalchemy import select
+
+if TYPE_CHECKING:
+    from sqlalchemy.ext.asyncio import AsyncSession
+
+
+async def _make_agent(db: AsyncSession, *, role: AgentRole) -> AgentTable:
+    agent = AgentTable(
+        id=uuid4(),
+        name=f"{role.value}-{uuid4().hex[:6]}",
+        slug=f"{role.value}-{uuid4().hex[:8]}",
+        role=role,
+        team=Team.BACKEND,
+        status=AgentStatus.ACTIVE,
+        model_config={},
+        system_prompt="agent",
+        capabilities=[],
+        permissions={},
+        metrics={},
+    )
+    db.add(agent)
+    await db.flush()
+    return agent
+
+
+async def _make_task(db: AsyncSession, *, created_by: UUID) -> TaskTable:
+    """Real Project + Task rows — notifications.related_task_id FKs to tasks."""
+    project = ProjectTable(
+        id=uuid4(),
+        name="P",
+        slug=f"p-{uuid4().hex[:8]}",
+        git_url="https://example.com/r.git",
+        assigned_cell=Team.BACKEND,
+        created_by=created_by,
+    )
+    db.add(project)
+    await db.flush()
+    task = TaskTable(
+        id=uuid4(),
+        title="Blocked task",
+        description="d",
+        acceptance_criteria=["ac"],
+        status=TaskStatus.BLOCKED,
+        priority=2,
+        task_type=TaskType.CODE,
+        nature=TaskNature.TECHNICAL,
+        project_id=project.id,
+        created_by=created_by,
+        team=Team.BACKEND,
+    )
+    db.add(task)
+    await db.flush()
+    return task
+
+
+async def _count_matching(
+    db: AsyncSession,
+    *,
+    from_agent: UUID,
+    notification_type: NotificationType,
+    related_task_id: UUID,
+) -> int:
+    result = await db.execute(
+        select(NotificationTable).where(
+            NotificationTable.from_agent == from_agent,
+            NotificationTable.type == notification_type,
+            NotificationTable.related_task_id == related_task_id,
+        )
+    )
+    return len(result.scalars().all())
+
+
+@pytest.mark.asyncio
+async def test_persist_and_deliver_suppresses_db_duplicate_past_redis_window(
+    db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """A retried blocker escalation past the 60s Redis window must not
+    create a second unacked row for the same (sender, type, task,
+    recipients) — the DB purpose-dedup gap `_persist_and_deliver` used to
+    skip entirely."""
+    sender = await _make_agent(db_session, role=AgentRole.DEVELOPER)
+    pm = await _make_agent(db_session, role=AgentRole.CELL_PM)
+    task = await _make_task(db_session, created_by=cast("UUID", sender.id))
+    task_id = cast("UUID", task.id)
+
+    # Seed the prior, still-unacked notification for this exact purpose.
+    prior = NotificationTable(
+        type=NotificationType.BLOCKER_ESCALATION,
+        priority=NotificationPriority.HIGH,
+        from_agent=cast("UUID", sender.id),
+        to_agents=[cast("UUID", pm.id)],
+        subject="ACTION REQUIRED: Blocked - task",
+        body="first send",
+        related_task_id=task_id,
+        requires_ack=True,
+        read_by=[],
+        acked_by=[],
+    )
+    db_session.add(prior)
+    await db_session.flush()
+
+    # Bypass the 60s Redis re-fire guard so the DB check is the only guard
+    # left standing — matches tests/unit/test_notification_delivery_refire.py.
+    monkeypatch.setattr(
+        "roboco.services.notification_delivery.all_recipients_recently_notified",
+        AsyncMock(return_value=False),
+    )
+
+    service = get_notification_delivery_service(db_session)
+    deliver_mock = AsyncMock()
+    monkeypatch.setattr(service, "deliver", deliver_mock)
+
+    retry = NotificationTable(
+        type=NotificationType.BLOCKER_ESCALATION,
+        priority=NotificationPriority.HIGH,
+        from_agent=cast("UUID", sender.id),
+        to_agents=[cast("UUID", pm.id)],
+        subject="ACTION REQUIRED: Blocked - task (retry)",
+        body="retried send",
+        related_task_id=task_id,
+        requires_ack=True,
+        read_by=[],
+        acked_by=[],
+    )
+
+    persisted = await service._persist_and_deliver(retry)
+
+    assert persisted is False
+    deliver_mock.assert_not_awaited()
+    count = await _count_matching(
+        db_session,
+        from_agent=cast("UUID", sender.id),
+        notification_type=NotificationType.BLOCKER_ESCALATION,
+        related_task_id=task_id,
+    )
+    assert count == 1  # only the seeded prior — the retry was suppressed
+
+
+@pytest.mark.asyncio
+async def test_persist_and_deliver_allows_distinct_recipient_set(
+    db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """A strictly-different recipient set is NOT a duplicate — overlap alone
+    must not suppress (only exact-set equality does)."""
+    sender = await _make_agent(db_session, role=AgentRole.DEVELOPER)
+    pm = await _make_agent(db_session, role=AgentRole.CELL_PM)
+    main_pm = await _make_agent(db_session, role=AgentRole.MAIN_PM)
+    task = await _make_task(db_session, created_by=cast("UUID", sender.id))
+    task_id = cast("UUID", task.id)
+
+    prior = NotificationTable(
+        type=NotificationType.BLOCKER_ESCALATION,
+        priority=NotificationPriority.HIGH,
+        from_agent=cast("UUID", sender.id),
+        to_agents=[cast("UUID", pm.id)],
+        subject="ACTION REQUIRED: Blocked - task",
+        body="first send",
+        related_task_id=task_id,
+        requires_ack=True,
+        read_by=[],
+        acked_by=[],
+    )
+    db_session.add(prior)
+    await db_session.flush()
+
+    monkeypatch.setattr(
+        "roboco.services.notification_delivery.all_recipients_recently_notified",
+        AsyncMock(return_value=False),
+    )
+
+    service = get_notification_delivery_service(db_session)
+    monkeypatch.setattr(service, "deliver", AsyncMock())
+
+    escalated = NotificationTable(
+        type=NotificationType.BLOCKER_ESCALATION,
+        priority=NotificationPriority.HIGH,
+        from_agent=cast("UUID", sender.id),
+        to_agents=[cast("UUID", pm.id), cast("UUID", main_pm.id)],
+        subject="ACTION REQUIRED: Blocked - task (escalated)",
+        body="re-escalated to main-pm too",
+        related_task_id=task_id,
+        requires_ack=True,
+        read_by=[],
+        acked_by=[],
+    )
+
+    persisted = await service._persist_and_deliver(escalated)
+
+    assert persisted is True
+    count = await _count_matching(
+        db_session,
+        from_agent=cast("UUID", sender.id),
+        notification_type=NotificationType.BLOCKER_ESCALATION,
+        related_task_id=task_id,
+    )
+    expected_rows = 2  # the prior + the distinct-recipient-set escalation
+    assert count == expected_rows

tests/integration/test_notification_reescalation_dedup_exemption.py

@@ -0,0 +1,187 @@
+"""_re_escalate_recipient must not be suppressed by its own prior attempt.
+
+PR #742's DB purpose-dedup (`duplicate_unacked_notification_exists`) closed
+the retried-first-send-blocker gap, but applied unconditionally inside
+`_persist_and_deliver` — so a SECOND re-escalation attempt at the same
+recipient (identical sender/type/task/recipient-set, and the prior attempt
+is unacked BY DEFINITION) was silently suppressed right after
+`_claim_reescalation_slot` had already burned the attempt slot, permanently
+stalling the re-escalation ladder.
+
+`_re_escalate_recipient` now passes `bypass_purpose_dedup=True` into
+`_persist_and_deliver`, so this dedup check never applies to this call path.
+This test drives `_re_escalate_recipient` twice against a real db_session,
+seeding nothing but real agent rows, and asserts both attempts persist and
+deliver a row rather than the second being silently dropped.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, cast
+from unittest.mock import AsyncMock
+from uuid import UUID, uuid4
+
+import pytest
+from roboco.db.tables import AgentTable, NotificationTable, ProjectTable, TaskTable
+from roboco.models import AgentRole, AgentStatus, NotificationPriority, NotificationType
+from roboco.models.base import TaskNature, TaskStatus, TaskType, Team
+from roboco.services.notification_delivery import get_notification_delivery_service
+from sqlalchemy import select
+
+if TYPE_CHECKING:
+    from sqlalchemy.ext.asyncio import AsyncSession
+
+
+async def _make_agent(db: AsyncSession, *, slug: str, role: AgentRole) -> AgentTable:
+    agent = AgentTable(
+        id=uuid4(),
+        name=slug,
+        slug=slug,
+        role=role,
+        team=None,
+        status=AgentStatus.ACTIVE,
+        model_config={},
+        system_prompt="agent",
+        capabilities=[],
+        permissions={},
+        metrics={},
+    )
+    db.add(agent)
+    await db.flush()
+    return agent
+
+
+async def _make_task(db: AsyncSession, *, created_by: UUID) -> TaskTable:
+    """Real Project + Task rows — notifications.related_task_id FKs to tasks."""
+    project = ProjectTable(
+        id=uuid4(),
+        name="P",
+        slug=f"p-{uuid4().hex[:8]}",
+        git_url="https://example.com/r.git",
+        assigned_cell=Team.BACKEND,
+        created_by=created_by,
+    )
+    db.add(project)
+    await db.flush()
+    task = TaskTable(
+        id=uuid4(),
+        title="Blocked task",
+        description="d",
+        acceptance_criteria=["ac"],
+        status=TaskStatus.BLOCKED,
+        priority=2,
+        task_type=TaskType.CODE,
+        nature=TaskNature.TECHNICAL,
+        project_id=project.id,
+        created_by=created_by,
+        team=Team.BACKEND,
+    )
+    db.add(task)
+    await db.flush()
+    return task
+
+
+async def _count_matching(
+    db: AsyncSession, *, from_agent: UUID, related_task_id: UUID
+) -> int:
+    result = await db.execute(
+        select(NotificationTable).where(
+            NotificationTable.from_agent == from_agent,
+            NotificationTable.type == NotificationType.BLOCKER_ESCALATION,
+            NotificationTable.related_task_id == related_task_id,
+        )
+    )
+    return len(result.scalars().all())
+
+
+@pytest.mark.asyncio
+async def test_two_sequential_due_reescalations_both_deliver(
+    db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """Two sequential re-escalation attempts at the same recipient (the shape
+    a real ladder produces on successive DUE sweep ticks) must both persist +
+    deliver — not just the first."""
+    # Random (not canonical be-pm/main-pm) slugs: this DB already carries the
+    # real seeded fleet agents, so a literal canonical slug would collide on
+    # the unique index. get_escalation_target is monkeypatched below since it
+    # keys off the real ESCALATION_CHAIN's literal canonical slugs.
+    sender = await _make_agent(
+        db_session, slug=f"dev-{uuid4().hex[:8]}", role=AgentRole.DEVELOPER
+    )
+    recipient = await _make_agent(
+        db_session, slug=f"pm-{uuid4().hex[:8]}", role=AgentRole.CELL_PM
+    )
+    target = await _make_agent(
+        db_session, slug=f"main-{uuid4().hex[:8]}", role=AgentRole.MAIN_PM
+    )
+    task = await _make_task(db_session, created_by=cast("UUID", sender.id))
+    task_id = cast("UUID", task.id)
+
+    original = NotificationTable(
+        type=NotificationType.BLOCKER_ESCALATION,
+        priority=NotificationPriority.HIGH,
+        from_agent=cast("UUID", sender.id),
+        to_agents=[cast("UUID", recipient.id)],
+        subject="ACTION REQUIRED: Blocked - task",
+        body="first send",
+        related_task_id=task_id,
+        requires_ack=True,
+        read_by=[],
+        acked_by=[],
+    )
+    db_session.add(original)
+    await db_session.flush()
+
+    monkeypatch.setattr(
+        "roboco.services.notification_delivery.all_recipients_recently_notified",
+        AsyncMock(return_value=False),
+    )
+    monkeypatch.setattr(
+        "roboco.services.notification_delivery.get_escalation_target",
+        lambda slug: target.slug if slug == recipient.slug else None,
+    )
+
+    service = get_notification_delivery_service(db_session)
+    monkeypatch.setattr(service, "deliver", AsyncMock())
+
+    first_ok = await service._re_escalate_recipient(
+        original, cast("UUID", recipient.id)
+    )
+    second_ok = await service._re_escalate_recipient(
+        original, cast("UUID", recipient.id)
+    )
+
+    assert first_ok is True
+    assert second_ok is True, (
+        "second DUE re-escalation must still deliver — it must not be "
+        "suppressed by the first, unacked, re-escalation row"
+    )
+    count = await _count_matching(
+        db_session, from_agent=cast("UUID", sender.id), related_task_id=task_id
+    )
+    expected_total_rows = 3  # the seeded original + both re-escalation attempts
+    assert count == expected_total_rows
+    # Both re-escalation rows must be addressed to target.id (the escalation
+    # target), proving the ladder re-fired to the right recipient — not a
+    # tautology on target.id which was uuid4() at construction.
+    all_rows = (
+        (
+            await db_session.execute(
+                select(NotificationTable).where(
+                    NotificationTable.from_agent == cast("UUID", sender.id),
+                    NotificationTable.type == NotificationType.BLOCKER_ESCALATION,
+                    NotificationTable.related_task_id == task_id,
+                )
+            )
+        )
+        .scalars()
+        .all()
+    )
+    re_escalation_rows = [
+        r for r in all_rows if cast("UUID", target.id) in (r.to_agents or [])
+    ]
+    expected_re_escalation_rows = 2  # one per _re_escalate_recipient call
+    assert len(re_escalation_rows) == expected_re_escalation_rows, (
+        f"expected {expected_re_escalation_rows} re-escalation rows addressed "
+        f"to target ({target.slug}), got {len(re_escalation_rows)}"
+    )

tests/integration/test_notification_system_list.py

@@ -151,6 +151,63 @@ async def test_pending_ack_only_slices_to_limit(db_session: AsyncSession) -> Non
     assert len(result) == limit
 
 
+@pytest.mark.asyncio
+async def test_get_notification_count_matches_in_memory_computation_at_scale(
+    db_session: AsyncSession,
+) -> None:
+    """SQL COUNT rewrite must match the old in-memory sum() for a large, mixed seed.
+
+    Seeds >1000 rows for one agent, mixing read/unread, acked/unacked, and
+    requires_ack true/false so all three counters exercise non-trivial values,
+    then asserts the service's result equals what the OLD Python-side
+    computation (``len(rows)`` / ``sum(agent_id not in read_by)`` /
+    ``sum(requires_ack and agent_id not in acked_by)``) would produce.
+    """
+    sender_id = await _seed_sender(db_session)
+    recipient_id = await _seed_recipient(db_session)
+    other_id = uuid4()  # noise recipient — must not affect recipient_id's counts
+
+    base = datetime(2026, 4, 1, tzinfo=UTC)
+    rows: list[NotificationTable] = []
+    seed_size = 1050
+    for i in range(seed_size):
+        read = i % 2 == 0
+        acked = i % 3 == 0
+        requires_ack = i % 5 != 0
+        n = NotificationTable(
+            type=NotificationType.REVIEW_REQUEST,
+            priority=NotificationPriority.NORMAL,
+            from_agent=sender_id,
+            to_agents=[recipient_id, other_id] if i % 7 == 0 else [recipient_id],
+            subject=f"Notification {i}",
+            body="Body text",
+            requires_ack=requires_ack,
+            acked_by=[recipient_id] if acked else [],
+            read_by=[recipient_id] if read else [],
+            timestamp=base + timedelta(seconds=i),
+        )
+        rows.append(n)
+    db_session.add_all(rows)
+    await db_session.flush()
+
+    expected_total = len(rows)
+    expected_unread = sum(1 for n in rows if recipient_id not in n.read_by)
+    expected_pending_ack = sum(
+        1 for n in rows if n.requires_ack and recipient_id not in n.acked_by
+    )
+    assert expected_unread not in (0, expected_total)
+    assert expected_pending_ack not in (0, expected_total)
+
+    service = get_notification_delivery_service(db_session)
+    counts = await service.get_notification_count(recipient_id)
+
+    assert counts == {
+        "total": expected_total,
+        "unread": expected_unread,
+        "pending_ack": expected_pending_ack,
+    }
+
+
 @pytest.mark.asyncio
 async def test_non_pending_branch_keeps_sql_limit(db_session: AsyncSession) -> None:
     """Without pending_ack_only the SQL limit still bounds the result."""