[6788ce7f] Silent bug sweep: concurrency, state integrity, engine edge-cases, panel data freshness (#638)

* [943d8c4d] Frontend data freshness and approval-queue reliability audit (#631)

* [233a8b0f] WebSocket reconnect message-loss audit and fix (#625)

* [233a8b0f] fix(panel): add REST catch-up to useNotificationStream on WS reconnect

connection.ts has no message buffering/replay, so a notification published
while the CEO bell's socket was down (disconnected/reconnecting) was lost
forever instead of merely delayed. Add a reconnect-triggered GET
/notifications?unread_only=true catch-up folded into the existing
notification_id dedup so a notification delivered both via catch-up and
live WS is never double-counted, and make clearMessages drop the held
catch-up batch too. use-a2a-live.ts and use-rate-limit-websocket.ts were
audited and already have working reconnect-triggered REST fallbacks
(verified via a2a/page.tsx, rate-limit-banner.tsx, usage-overview-panel.tsx
and their existing F083 tests) so no fix was needed there.

* [233a8b0f] docs(panel): add comprehensive WebSocket hooks reference and reconnect architecture guide

Add panel/docs/frontend/hooks.md with full API reference for useWebSocket, useNotificationStream (with new REST catch-up behavior), useAgentStream, useA2ALiveStream, and useConnectionStatus. Include examples, best practices, and testing guidance.

Add panel/docs/architecture/websocket-reconnect.md documenting the message-loss mitigation pattern: Strategy 1 (REST catch-up for events, used by useNotificationStream) and Strategy 2 (REST invalidation for state, used by A2A/rate-limit consumers), plus the dedup logic ensuring no notification is double-counted on reconnect.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>

* [d5315683] fix(frontend): add distinct toast feedback for silently-swallowed x-post and release-proposal statuses, plus regression tests for all 4 approval queues (#626)

Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>

* [cd953838] Data-hook null-guard audit and API client 429 retry-by-method fix (#630)

* [cd953838] fix(panel): gate 429 retry by HTTP method, add hook null-guard regression tests

* [cd953838] chore(conventions): waive test-fixture wrapper in hooks null-guard test

* [cd953838] docs(frontend): document API rate-limit retry behavior and null-guard audit results

Added `docs/frontend/api-rate-limiting.md` to document the 429 retry strategy: GET/PUT auto-retry, POST/PATCH/DELETE require X-Idempotency-Key header. Updated `docs/frontend/hooks.md` to confirm the data-hook null-guard audit found all hooks already have correct `enabled` guards and include a regression test suite for the board-review poll on/off behavior and enabled-guard assertions.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>

* [4534c71a] Backend concurrency, state-machine, and engine audit (#634)

* [41de844a] fix(lifecycle): sync CLAIM_RULES with runtime + clear stale claimant on PM hand-off (#627)

Two confirmed state-machine gaps found while auditing lifecycle.py,
task_lifecycle.py, the _ESCALATABLE_TO_BLOCKED bypass, and every
_REVIEW_QUEUE_STATES entry point:

- lifecycle.py's CLAIM_RULES/claim-ActionSpec/StatusTransition table
  did not grant CELL_PM/MAIN_PM re-claim of AWAITING_PM_REVIEW even
  though task.py's runtime _ROLE_CLAIM_STATUSES already granted it
  and claimed the spec agreed -- the two tables had silently drifted,
  breaking i_will_plan re-claim on an awaiting_pm_review task.

- docs_complete's _maybe_advance_to_pm_review pre-assigns a specific
  owning PM via assigned_to but left claimed_by/active_claimant_id
  pointing at the outgoing documenter, unlike every sibling transition
  into a review-queue state. A stale active_claimant_id makes
  content_actions.py's _active_claim_violation wrongly reject the
  newly-assigned PM's own content writes before it formally claims.
  Reassign claimed_by + active_claimant_id to the owning PM alongside
  assigned_to.

Adds a regression test asserting the documenter's stale claim does not
survive the docs_complete -> awaiting_pm_review hand-off.

Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>

* [0c46f666] Engine dedup race + sequencing.py edge-case audit (#628)

* [0c46f666] fix(sequencing): dedup race audit + collision-edge fallback bug

Audited the list-open-then-originate dedup pattern across six engines:
RoadmapEngine, XEngine.run_cycle, DepUpdateEngine, and CIWatchEngine each
run inside exactly one sequential orchestrator-loop asyncio task (no other
call site invokes run_cycle), so they cannot race with themselves; their
in-cycle dedup sets/keys are correctly built before any commit. SelfHealEngine
is the same shape. VideoEngine.open_video_task is genuinely different: it is
reachable from the release-publish hook, the feature-spotlight hook, and the
on-demand POST /video/request route, so two overlapping calls for the same
occasion can both pass the "no open task yet" check before either commits.
Fixed by wrapping the check+insert in a short-lived Redis mutex (reusing
HeartbeatMutex) keyed by occasion, mirroring XPostService's existing
lock pattern, with a regression test proving only one of two concurrent
calls creates a task.

Verified ReleaseExecutor's half-landed retry path (release_commit_sha):
apply_version_bumps and write_changelog_entry both run as uncommitted
working-tree edits before commit_and_push's single `git add -A` + commit,
so a bumped-version-without-changelog state can never reach origin (and
therefore can never be observed by a fresh retry clone) - confirmed correct
with a real-git-repo regression test, no fix needed.

Fixed sequencing.py's dev_task_collision_edges: the `if edges: return edges`
short-circuit dropped the same-assignee-lane fallback entirely whenever ANY
surfaced sibling pair produced a collision edge, even for a completely
unrelated same-assignee pair with no declared surface. Now the fallback
always runs, skipping only pairs the analyzer already ordered (so the two
mechanisms can never disagree on direction for the same pair).

Verified sequencing.py rule 3 (all-shared batch generates no edges): correct
by inspection (_shared_last_edges skips every pair when both are shared) and
confirmed with a regression test - no fix needed.

* [0c46f666] docs(reference): concurrency audit summary - engine races, fixes, verified patterns

---------

Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>

* [8f7f167a] Redis mutex pre-lock write audit (#629)

* [8f7f167a] Redis mutex pre-lock write audit: add cross-session regression test for XPostService.approve

Audited x_post_service.py, video_post_service.py, release_proposal.py, and
heartbeat_mutex.py for the pre-lock DB-write anti-pattern (a session write
that happens before the SET NX / HeartbeatMutex acquire returns a token,
letting a losing racer's stale write clobber a winner's committed state).

XPostService.approve, VideoPostService.approve, and
ReleaseProposalService.approve/reject already implement the correct
validate-pure-pre-lock, apply-under-lock pattern (the XPostService fix
already shipped per CHANGELOG.md: "X edited_body write deferred into the
single-flight lock (M5)"). HeartbeatMutex holds no AsyncSession at all, so
the anti-pattern is structurally inapplicable there.

Adds a genuine cross-session concurrency regression test to
test_x_post_service.py (a real second DB connection, not an in-process
mock) mirroring VideoPostService's existing cross-session test, proving a
concurrently-committed post survives and the CEO's edited body never lands
on the just-posted row.

* [8f7f167a] Remove redundant inline comments flagged by QA in cross-session regression test

Both comments restated what the surrounding docstrings already say
explicitly, per QA findings F-dbadd8f0 (line 294) and F-27ac051e (line
631) — no behavior change, tests re-verified green against a sandbox
Postgres.

* [8f7f167a] Remove inline trailing comments flagged by QA (correct file this time)

QA findings F-e6f3e6a6 and F-24189858 cited tests/unit/services/
test_x_post_service.py:294 and :631 across 5 revision rounds, but that
file never contained the flagged comment text — a repo-wide grep for
the exact quoted strings shows both comments actually live in the
mirrored tests/unit/services/test_video_post_service.py file, in its
own cross-session concurrency regression tests (the caption-edit and
tiktok-skip tests). Removed both there:
- "# externally visible to the "concurrent" session below" on the
  db_session.commit() call
- "# never attempted without credentials" on the tiktok_poster.calls
  assertion

Both restated what the surrounding docstrings/test names already say;
no behavior change. Verified with the full make quality gate against a
sandbox Postgres/Redis: 13,717 passed, 94.41% coverage, clean except
one pre-existing unrelated failure in tests/unit/api/test_cloud_auth.py
::test_login_route_parses_oauth2_form_not_query_params, which connects
to the app's default localhost:5432 Postgres (not the db_session
sandbox fixture) and is unreachable in this sandboxed environment —
structurally unrelated to the auth subsystem this task never touches.

* [8f7f167a] Redis mutex pre-lock write audit (round 7): add cross-session regression tests for reject() lock protection

Round-7 QA findings F-7eb9fbcb, F-06f39a2e, and F-4d56e49b claim
XPostService.reject(), ReleaseProposalService.reject(), and
release_executor._await_proc() lack lock protection / a CancelledError
handler — but their cited line ranges (255-267, 429-454, 241-257)
describe a pre-fix, shorter version of these functions that predates
commit fb293a787d, already on this branch. At current HEAD:

- x_post_service.py reject() (lines 275-299) acquires _LOCK_PREFIX,
  re-reads under the lock, applies markers.set_x_reject_reason() +
  CANCELLED only inside the critical section, releases in finally.
- release_proposal.py reject() (lines 460-486) does the identical
  dance with _RELEASE_LOCK_PREFIX.
- release_executor.py _await_proc() (lines 257-265) already has an
  `except asyncio.CancelledError` block that kills + reaps the child
  and re-raises, mirroring the TimeoutError handler, with an existing
  dedicated regression test
  (test_await_proc_kills_child_on_outer_cancellation).

The one genuine gap: neither reject() path had a cross-session
(real second DB connection, not an in-process mock) regression test
proving the in-lock re-read catches a concurrent approve/publish that
completes mid-lock-wait — only approve() had one. Added
test_reject_concurrent_approve_completes_during_lock_wait to both
test_x_post_service.py and test_release_proposal_status_guards.py,
mirroring the existing approve() cross-session test: a second engine
commits COMPLETED between reject's pre-lock read and lock acquisition,
and the test asserts the CANCELLED write / reject-reason marker never
lands on the just-completed row.

No production code changed — verified via 103 targeted tests green
against a sandbox Postgres/Redis, plus `make -o sync gate` clean.

* [8f7f167a] Regenerate stale lifecycle artifacts (restore auditor waive_finding)

foundation-check was the only failing gate: the committed lifecycle artifacts
were missing the auditor's waive_finding verb that the lifecycle source
defines, so make quality regenerated them and failed on the diff — nothing to
do with the mutex fix (which passes ruff/mypy/tests/coverage/bandit clean).
make lifecycle restores the drift; this is what the 8 revision rounds kept
missing.

---------

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>

* [d615f2e3] fix(tests): sync stale CLAIM_RULES pinning assertions with lifecycle.py (#635)

test_claim_rules_match_pre_gateway_table still asserted the pre-audit
two-member frozenset for CELL_PM/MAIN_PM claim rules. CLAIM_RULES in
lifecycle.py already grants both roles claim rights on
Status.AWAITING_PM_REVIEW (added by the state-machine exhaustiveness
audit) so a PM can re-claim its own review-queue task after a respawn.
Updated both assertions to include AWAITING_PM_REVIEW, matching the
actual dict. Grepped the repo for sibling stale copies of the old
literal; found none beyond this test.

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>

* [3c4e7a35] fix(quality-gate): reflow CONCURRENCY_AUDIT.md and stub occasion lock in video tests (#636)

Root cause: PR #634's CI failed at the markdown-reflow check (make quality
Makefile:285) on CONCURRENCY_AUDIT.md — a hard-wrapped audit doc left over
from the merged "Engine dedup race + sequencing.py edge-case audit" unit
(PR #628). Fixed with `make reflow-docs` (the exact remedy the CI output
itself named).

Running the full local `make quality` (with a sandbox Postgres/Redis to
get past DB-gated skips) surfaced a second real regression from that same
PR #628 unit: it added a Redis-backed HeartbeatMutex occasion lock to
VideoEngine.open_video_task, but two pre-existing test files
(tests/unit/runtime/test_video_render_loop.py and
tests/integration/test_video_routes.py) call open_video_task without
stubbing that lock, so they failed closed against the suite's
deliberately-unreachable test Redis (_no_live_redis). Fixed by applying
the same lock-stub pattern tests/unit/services/test_video_engine.py
already uses for its own occasion-lock tests: an autouse HeartbeatMutex
stand-in fixture in test_video_render_loop.py, and wrapping the two
route-level video-request tests in test_video_routes.py with the file's
existing _LOCKED patch pair (already used by every other lock-dependent
test in that file).

The one remaining local failure,
test_cloud_auth.py::test_login_route_parses_oauth2_form_not_query_params,
is a pre-existing environment gap unrelated to this branch: it needs a
real Postgres reachable at localhost:5432 (which .github/workflows/ci.yml
provides as a service container) but this dev sandbox has no such binding
— confirmed unrelated to any of the four merged audit units.

make quality now passes clean: 13729 passed, 0 regressions, 94.49% coverage.

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>

* [ddc8121f] regenerate lifecycle artifacts for awaiting_pm_review claim rules and waive_finding intent (#637)

Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>

---------

Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>

* fix(sequencing): drop lane-fallback edges that would cycle against analyzer edges

The dev-task collision fallback unioned the analyzer's authoritative edges
with same-assignee lane edges, deduping only the direct pair. A lane chain
through an unsurfaced middle sibling could still contradict an analyzer edge
transitively (the shared-last migration order inverts plain priority order),
closing a 3-cycle that made add_dependency raise ConflictError and wedged
every later delegate to that parent. Fallback edges are now accepted only
when they can't close a cycle against the edges already kept; a regression
test reproduces the exact scenario.

Also strip pre-merge cruft: remove the root CONCURRENCY_AUDIT.md working
report, delete the near-duplicate websocket-reconnect.md doc, fix the stale
a2a/page.tsx doc citation, correct the api-rate-limiting doc to state
idempotency-key retry is unimplemented, and fix two lifecycle.py comments
that referenced a guard function which never existed.

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@roboco.tech>
Co-authored-by: Frontend Documenter <fe-doc@roboco.tech>
Co-authored-by: Frontend Developer 2 <fe-dev-2@roboco.tech>
Co-authored-by: Backend Developer 2 <be-dev-2@roboco.tech>
Co-authored-by: Backend Documenter <be-doc@roboco.tech>
Co-authored-by: Backend Developer 1 <be-dev-1@roboco.tech>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
2026-07-22 08:11:28 +02:00
committed by GitHub
co-authored by Frontend Developer 1 Frontend Documenter Frontend Developer 2 Backend Developer 2 Backend Documenter Backend Developer 1 Renn F
parent a1233b2aeb
commit 17de29545a
39 changed files with 2458 additions and 89 deletions
+10 -2
View File
@@ -500,10 +500,18 @@ def test_claim_rules_match_pre_gateway_table() -> None:
{spec.Status.PENDING, spec.Status.AWAITING_DOCUMENTATION}
)
assert spec.CLAIM_RULES[spec.Role.CELL_PM] == frozenset(
{spec.Status.PENDING, spec.Status.NEEDS_REVISION}
{
spec.Status.PENDING,
spec.Status.NEEDS_REVISION,
spec.Status.AWAITING_PM_REVIEW,
}
)
assert spec.CLAIM_RULES[spec.Role.MAIN_PM] == frozenset(
{spec.Status.PENDING, spec.Status.NEEDS_REVISION}
{
spec.Status.PENDING,
spec.Status.NEEDS_REVISION,
spec.Status.AWAITING_PM_REVIEW,
}
)
+34 -4
View File
@@ -368,10 +368,22 @@ async def test_reject_records_changes_and_cancels_frees_dedup(
frees and the release manager can re-assess next cycle. The required-changes
marker stays on the cancelled row for history."""
task = await _seed_proposal(db_session)
resp = await ceo_client.post(
"/api/release/proposal/reject",
json={"required_changes": "Tighten the CHANGELOG wording for the API change."},
)
with (
patch.object(
ReleaseProposalService, "_acquire_release_lock", AsyncMock(return_value="t")
),
patch.object(
ReleaseProposalService,
"_release_release_lock",
AsyncMock(return_value=None),
),
):
resp = await ceo_client.post(
"/api/release/proposal/reject",
json={
"required_changes": "Tighten the CHANGELOG wording for the API change."
},
)
assert resp.status_code == HTTPStatus.OK
assert "Tighten the CHANGELOG" in (resp.json()["required_changes"] or "")
refreshed = await db_session.get(TaskTable, task.id)
@@ -382,6 +394,24 @@ async def test_reject_records_changes_and_cancels_frees_dedup(
assert task.id not in {t.id for t in open_proposals}
@pytest.mark.asyncio
async def test_reject_refused_while_approve_lock_held(
db_session: AsyncSession, ceo_client: AsyncClient
) -> None:
"""A concurrent approve holds the release mutex (mid ~40min execute);
reject must fail closed with 409 instead of racing an unguarded write
under it previously reject() never even attempted the lock."""
await _seed_proposal(db_session)
with patch.object(
ReleaseProposalService, "_acquire_release_lock", AsyncMock(return_value=None)
):
resp = await ceo_client.post(
"/api/release/proposal/reject",
json={"required_changes": "Tighten the CHANGELOG wording."},
)
assert resp.status_code == HTTPStatus.CONFLICT
@pytest.mark.asyncio
async def test_non_ceo_is_forbidden(db_session: AsyncSession) -> None:
await _seed_proposal(db_session)
@@ -2112,6 +2112,59 @@ async def test_docs_complete_advances_when_pr_already_created(
assert out.status == TaskStatus.AWAITING_PM_REVIEW
@pytest.mark.asyncio
async def test_docs_complete_advance_clears_stale_documenter_claim(
task_setup: dict, db_session: AsyncSession
) -> None:
"""F-audit: docs_complete's PM hand-off must not leave the outgoing
documenter as claimed_by/active_claimant_id once a specific owning PM is
assigned unlike its siblings (qa_pass/fail_qa/submit_for_qa/pr_pass/
pr_fail/request_changes), which always reassign or clear both fields
together, _maybe_advance_to_pm_review used to only update assigned_to.
A stale active_claimant_id then makes content_actions.py's
`_active_claim_violation` wrongly reject the newly-assigned PM's own
explicit-task_id note()/commit() calls before it formally claims.
"""
svc = task_setup["svc"]
pm_agent = AgentTable(
id=uuid4(),
name="PM",
slug=f"be-pm-{uuid4().hex[:8]}",
role=AgentRole.CELL_PM,
team=Team.BACKEND,
status=AgentStatus.ACTIVE,
model_config={},
system_prompt="pm",
capabilities=[],
permissions={},
metrics={},
)
db_session.add(pm_agent)
await db_session.flush()
parent = await svc.create(_req(task_setup, assigned_to=pm_agent.id))
task = await svc.create(_req(task_setup, parent_task_id=parent.id))
task.status = TaskStatus.AWAITING_DOCUMENTATION
documenter_id = task_setup["agent_id"]
task.assigned_to = documenter_id
task.claimed_by = documenter_id
task.active_claimant_id = documenter_id
task.pr_number = 1
task.pr_url = "u"
task.pr_created = True
await db_session.flush()
out = await svc.docs_complete(task.id, doc_notes="documented all flows")
assert out is not None
assert out.status == TaskStatus.AWAITING_PM_REVIEW
assert out.assigned_to == pm_agent.id
# The documenter's claim must not survive the hand-off.
assert out.claimed_by == pm_agent.id
assert out.active_claimant_id == pm_agent.id
assert out.claimed_by != documenter_id
assert out.active_claimant_id != documenter_id
# ---------------------------------------------------------------------------
# mark_pr_created edge cases
# ---------------------------------------------------------------------------
+14 -11
View File
@@ -265,15 +265,16 @@ async def test_request_video_opens_authoring_task(
project = (
await db_session.execute(select(ProjectTable).where(ProjectTable.slug == SLUG))
).scalar_one()
resp = await ceo_client.post(
"/api/video/request",
json={
"occasion": "CEO on-demand: launch teaser",
"brief": "A short teaser for the new dashboard",
"platforms": ["x", "tiktok"],
"project_id": str(project.id),
},
)
with _LOCKED[0], _LOCKED[1]:
resp = await ceo_client.post(
"/api/video/request",
json={
"occasion": "CEO on-demand: launch teaser",
"brief": "A short teaser for the new dashboard",
"platforms": ["x", "tiktok"],
"project_id": str(project.id),
},
)
assert resp.status_code == HTTPStatus.OK
body = resp.json()
assert body["status"] == "opened"
@@ -396,12 +397,14 @@ async def test_request_video_not_opened_on_duplicate_occasion(
"platforms": ["x"],
"project_id": str(project.id),
}
first = await ceo_client.post("/api/video/request", json=payload)
with _LOCKED[0], _LOCKED[1]:
first = await ceo_client.post("/api/video/request", json=payload)
assert first.status_code == HTTPStatus.OK
assert first.json()["status"] == "opened"
task_id = first.json()["task_id"]
try:
second = await ceo_client.post("/api/video/request", json=payload)
with _LOCKED[0], _LOCKED[1]:
second = await ceo_client.post("/api/video/request", json=payload)
assert second.status_code == HTTPStatus.OK
assert second.json()["status"] == "not_opened"
assert second.json()["task_id"] is None
+21 -9
View File
@@ -189,9 +189,13 @@ async def test_reject_cancels_and_records_reason(
db_session: AsyncSession, ceo_client: AsyncClient
) -> None:
task = await _seed_draft(db_session)
resp = await ceo_client.post(
f"/api/x/posts/{task.id}/reject", json={"reason": "Not our voice"}
)
with (
patch.object(XPostService, "_acquire_lock", AsyncMock(return_value="tok")),
patch.object(XPostService, "_release_lock", AsyncMock(return_value=None)),
):
resp = await ceo_client.post(
f"/api/x/posts/{task.id}/reject", json={"reason": "Not our voice"}
)
assert resp.status_code == HTTPStatus.OK
assert resp.json()["reject_reason"] == "Not our voice"
refreshed = await db_session.get(TaskTable, task.id)
@@ -204,9 +208,13 @@ async def test_history_returns_posted_and_rejected_newest_first(
db_session: AsyncSession, ceo_client: AsyncClient
) -> None:
rejected = await _seed_draft(db_session)
await ceo_client.post(
f"/api/x/posts/{rejected.id}/reject", json={"reason": "off-brand tone"}
)
with (
patch.object(XPostService, "_acquire_lock", AsyncMock(return_value="tok")),
patch.object(XPostService, "_release_lock", AsyncMock(return_value=None)),
):
await ceo_client.post(
f"/api/x/posts/{rejected.id}/reject", json={"reason": "off-brand tone"}
)
posted = await _seed_draft(db_session)
posted_project = await db_session.get(ProjectTable, posted.project_id)
with (
@@ -258,9 +266,13 @@ async def test_history_respects_limit(
) -> None:
for _ in range(3):
t = await _seed_draft(db_session)
await ceo_client.post(
f"/api/x/posts/{t.id}/reject", json={"reason": "not relevant"}
)
with (
patch.object(XPostService, "_acquire_lock", AsyncMock(return_value="tok")),
patch.object(XPostService, "_release_lock", AsyncMock(return_value=None)),
):
await ceo_client.post(
f"/api/x/posts/{t.id}/reject", json={"reason": "not relevant"}
)
resp = await ceo_client.get("/api/x/posts/history", params={"limit": HISTORY_LIMIT})
assert resp.status_code == HTTPStatus.OK
assert len(resp.json()) == HISTORY_LIMIT
@@ -28,6 +28,7 @@ from roboco.runtime.orchestrator import (
_MAX_VIDEO_RENDER_ATTEMPTS,
AgentOrchestrator,
)
from roboco.services import video_engine as video_engine_module
from roboco.services.task import VIDEO_POST_SOURCE, get_task_service
from roboco.services.video_engine import VideoEngine
from sqlalchemy import select
@@ -49,6 +50,26 @@ def _orch() -> Any:
return AgentOrchestrator.__new__(AgentOrchestrator)
class _AlwaysAcquiredMutex:
"""Stand-in for ``HeartbeatMutex``: always acquires immediately, no live
Redis required (matches the project's ``_no_live_redis`` fixture and
mirrors ``test_video_engine.py``'s identical stub)."""
def __init__(self, *_args: object, **_kwargs: object) -> None:
pass
async def acquire(self) -> str | None:
return "tok"
async def release(self, _token: str) -> None:
return None
@pytest.fixture(autouse=True)
def _stub_occasion_lock(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(video_engine_module, "HeartbeatMutex", _AlwaysAcquiredMutex)
class _FakeRenderer:
"""Records every render() call; returns a deterministic path or raises."""
+104 -1
View File
@@ -9,8 +9,9 @@ call sequence; the production git/gh ops is exercised live (CEO-gated).
from __future__ import annotations
import base64
import subprocess
from types import SimpleNamespace
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Any, cast
from unittest.mock import MagicMock
import pytest
@@ -444,6 +445,108 @@ def test_release_ci_workflow_decoupled_from_self_heal_setting(
assert _resolve_release_ci_workflow() == "ci.yml"
# --------------------------------------------------------------------------- #
# _GitReleaseOps.release_commit_sha — half-landed retry detection against a
# REAL git repo (not the fake ops above): verifies the exact worry the task
# named — that ``_current_version`` could return the bumped version while the
# changelog hasn't actually been written yet. It can't: ``apply_version_bumps``
# and ``write_changelog_entry`` both run as uncommitted working-tree edits
# BEFORE ``commit_and_push``'s single ``git add -A`` + commit, so nothing ever
# reaches origin (and therefore no fresh retry clone can ever observe it) with
# one written but not the other.
# --------------------------------------------------------------------------- #
def _git_sync(repo: Path, *args: str) -> str:
result = subprocess.run(
["git", "-C", str(repo), *args], check=True, capture_output=True, text=True
)
return result.stdout
def _init_release_repo(repo: Path, *, initial_version: str = "0.12.0") -> None:
repo.mkdir(parents=True, exist_ok=True)
_git_sync(repo, "init", "-b", "master")
_git_sync(repo, "config", "user.email", "t@example.com")
_git_sync(repo, "config", "user.name", "T")
_git_sync(repo, "config", "commit.gpgsign", "false")
(repo / "pyproject.toml").write_text(f'[project]\nversion = "{initial_version}"\n')
(repo / "CHANGELOG.md").write_text("# Changelog\n")
_git_sync(repo, "add", "-A")
_git_sync(repo, "commit", "-m", "init")
def _ops(session: object, root: Path) -> _GitReleaseOps:
ctx = _ReleaseContext(
slug="roboco-api",
prod_branch="master",
root=root,
git_url="x",
git_prefix=[],
ci_workflow="ci.yml",
env_chain=[],
)
return _GitReleaseOps(session=cast("Any", session), ctx=ctx)
@pytest.mark.asyncio
async def test_release_commit_sha_detects_a_real_half_landed_commit(
tmp_path: Path,
) -> None:
"""A genuine publish_failed retry: a prior execute already ran
``apply_version_bumps`` + ``write_changelog_entry`` + committed (both
landed in the SAME commit, exactly as ``commit_and_push`` does with one
``git add -A``). The fresh clone must detect that commit and its
changelog entry must already be present never re-bump, never skip the
changelog."""
_init_release_repo(tmp_path)
ops = _ops(MagicMock(), tmp_path)
await ops.apply_version_bumps(["pyproject.toml"], "0.13.0")
await ops.write_changelog_entry(
"## [0.13.0] - 2026-07-21\n\n### Added\n- a thing\n"
)
_git_sync(tmp_path, "add", "-A")
_git_sync(tmp_path, "commit", "-m", "chore(release): 0.13.0")
sha = await ops.release_commit_sha("0.13.0")
assert sha is not None
assert sha == _git_sync(tmp_path, "rev-parse", "HEAD").strip()
# The changelog entry landed in the SAME commit as the bump — never split.
assert "0.13.0" in (tmp_path / "pyproject.toml").read_text()
assert "a thing" in (tmp_path / "CHANGELOG.md").read_text()
@pytest.mark.asyncio
async def test_release_commit_sha_none_for_uncommitted_bump_no_false_half_landed(
tmp_path: Path,
) -> None:
"""An uncommitted version bump (e.g. a crash between apply_version_bumps
and commit_and_push) must NOT be mistaken for a half-landed release
only a matching COMMIT counts. A fresh retry clone starts from origin's
unchanged HEAD anyway (this isolates release_commit_sha's own check)."""
_init_release_repo(tmp_path)
ops = _ops(MagicMock(), tmp_path)
# Bump the working tree WITHOUT committing (write_changelog_entry never ran).
await ops.apply_version_bumps(["pyproject.toml"], "0.13.0")
sha = await ops.release_commit_sha("0.13.0")
assert sha is None # falls through to the normal bump/changelog/gate/commit path
@pytest.mark.asyncio
async def test_release_commit_sha_none_when_version_not_yet_bumped(
tmp_path: Path,
) -> None:
"""No prior attempt at all: the clone is still at the old version, so
there is nothing half-landed to detect."""
_init_release_repo(tmp_path)
ops = _ops(MagicMock(), tmp_path)
assert await ops.release_commit_sha("0.13.0") is None
# --------------------------------------------------------------------------- #
# H11: the PAT must never appear in a git subprocess argv. The release clone
# and the release push carry the token via ``-c http.extraheader=Authorization:
@@ -10,9 +10,10 @@ under a second — never relying on real wall-clock timing of the defaults.
from __future__ import annotations
import asyncio
import os
from pathlib import Path
from typing import TYPE_CHECKING, cast
from unittest.mock import AsyncMock, MagicMock
from typing import TYPE_CHECKING, Any, cast
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
@@ -337,6 +338,42 @@ async def test_clone_run_times_out_and_kills_proc(
assert proc.killed
@pytest.mark.asyncio
async def test_await_proc_kills_child_on_outer_cancellation() -> None:
"""Redis mutex pre-lock write audit sibling finding: an outer cancellation
(e.g. the release loop's own task being cancelled mid-op) throws
``CancelledError`` into ``_await_proc``'s ``wait_for``, bypassing the
``TimeoutError`` handler. Without a dedicated handler (which
``quality_gate.py``'s ``_run_one`` already has) the child is orphaned and
keeps running past the cancelled release op. ``_await_proc`` must kill +
reap it and re-raise, mirroring the already-shipped ``quality_gate.py``
pattern exactly."""
real_create_subprocess_exec = asyncio.create_subprocess_exec
spawned: dict[str, asyncio.subprocess.Process] = {}
async def _capturing_create(*args: Any, **kwargs: Any) -> Any:
proc = await real_create_subprocess_exec(*args, **kwargs)
spawned["proc"] = proc
return proc
with patch(
"roboco.services.release_executor.asyncio.create_subprocess_exec",
_capturing_create,
):
task = asyncio.ensure_future(_run(["sleep", "30"]))
while "proc" not in spawned:
await asyncio.sleep(0.01)
await asyncio.sleep(0.1) # let the child actually exec
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
proc = spawned["proc"]
assert proc.returncode is not None, "child was not reaped after cancellation"
with pytest.raises(ProcessLookupError):
os.kill(proc.pid, 0)
# ---------------------------------------------------------------------------
# Regression: the happy path still returns the real rc + decoded output.
# ---------------------------------------------------------------------------
@@ -12,6 +12,7 @@ do for their own seams.
from __future__ import annotations
import contextlib
from typing import TYPE_CHECKING, cast
from unittest.mock import AsyncMock, patch
from uuid import uuid4
@@ -19,6 +20,7 @@ from uuid import uuid4
import pytest
from roboco.db.tables import AgentTable, ProjectTable, TaskTable
from roboco.foundation import identity as _foundation
from roboco.foundation.policy.content import markers
from roboco.models.base import AgentRole, AgentStatus, TaskNature, TaskStatus, TaskType
from roboco.models.base import Team as T
from roboco.services.release_proposal import (
@@ -26,13 +28,17 @@ from roboco.services.release_proposal import (
TaskAlreadyCompletedError,
)
from roboco.services.release_readiness import ReleaseReadinessReport, report_to_dict
from roboco.services.task import RELEASE_MANAGER_SOURCE
from roboco.services.task import RELEASE_MANAGER_SOURCE, TaskService
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
async_sessionmaker,
create_async_engine,
)
if TYPE_CHECKING:
from uuid import UUID
from sqlalchemy.ext.asyncio import AsyncSession
_VERSION = "0.18.0"
@@ -170,3 +176,89 @@ async def test_reject_raises_when_already_published(db_session: AsyncSession) ->
)
await db_session.refresh(task)
assert task.status == TaskStatus.COMPLETED # untouched, never cancelled
async def _fresh_session(url: str) -> tuple[AsyncSession, AsyncEngine]:
"""A session on a brand-new engine/connection (caller disposes)."""
engine = create_async_engine(url, future=True)
factory = async_sessionmaker(
bind=engine, class_=AsyncSession, expire_on_commit=False
)
return factory(), engine
async def _dispose(session: AsyncSession, engine: AsyncEngine) -> None:
with contextlib.suppress(Exception):
await session.rollback()
await engine.dispose()
@pytest.mark.asyncio
async def test_reject_concurrent_approve_completes_during_lock_wait(
db_session: AsyncSession, _test_database_url: str
) -> None:
"""Redis mutex pre-lock write audit regression for ``reject()``: a
genuinely concurrent approve (a real second session/connection) publishes
+ commits COMPLETED in the window between reject's pre-lock read and its
lock acquisition. The in-lock re-read must see that committed state and
refuse the CANCELLED status write and required-changes marker must
never land on the just-published row, proving the fix holds across
sessions, not merely within one. Mirrors
``test_x_post_service.test_reject_concurrent_approve_completes_during_lock_wait``.
"""
task = await _seed_proposal(db_session)
task_id = cast("UUID", task.id)
await db_session.commit()
real_get = TaskService.get
injected = False
async def _get_then_inject_concurrent_publish(
self: TaskService, tid: UUID
) -> TaskTable | None:
"""Fires once, right after reject's pre-lock read — the exact window
between that read and reject's own (would-be) pre-lock write."""
nonlocal injected
result = await real_get(self, tid)
if not injected:
injected = True
other, other_engine = await _fresh_session(_test_database_url)
try:
other_task = await other.get(TaskTable, tid)
assert other_task is not None
other_task.status = TaskStatus.COMPLETED
await other.commit()
finally:
await _dispose(other, other_engine)
return result
with (
patch.object(TaskService, "get", _get_then_inject_concurrent_publish),
patch.object(
ReleaseProposalService,
"_acquire_release_lock",
AsyncMock(return_value="tok"),
),
patch.object(
ReleaseProposalService,
"_release_release_lock",
AsyncMock(return_value=None),
),
patch.object(
ReleaseProposalService, "_close_redis", AsyncMock(return_value=None)
),
pytest.raises(TaskAlreadyCompletedError),
):
await ReleaseProposalService(db_session).reject(
task_id, "needs another migration check"
)
fresh, fresh_engine = await _fresh_session(_test_database_url)
try:
final = await fresh.get(TaskTable, task_id)
assert final is not None
assert final.status == TaskStatus.COMPLETED
# The reject must never have landed on the just-published row.
assert markers.get_release_required_changes(final) is None
finally:
await _dispose(fresh, fresh_engine)
+110
View File
@@ -83,6 +83,27 @@ def test_touches_shared_runs_last() -> None:
assert plan.waves[-1] == [2] # the shared task is the final wave
def test_all_shared_batch_with_disjoint_surfaces_generates_no_edges() -> None:
# Verifying the claimed rule-3 property: when every draft in the batch
# touches_shared, ``_shared_last_edges`` skips every candidate pair (its
# inner loop continues on `other.touches_shared`), so it contributes no
# edges on its own. With disjoint file surfaces rule 1 (same-shared-status
# overlap) also contributes nothing, so the whole batch runs in one
# parallel wave — confirmed correct, no fix needed.
s = [
DraftSurface(0, 1, ["fe/app/a.tsx"], False, True),
DraftSurface(1, 1, ["fe/app/b.tsx"], False, True),
DraftSurface(2, 1, ["fe/app/c.tsx"], False, True),
]
plan = SequencingService().analyze(s, _frontend, {"frontend": 3})
assert plan.edges == []
assert plan.waves == [[0, 1, 2]]
# And rule 3 in isolation truly contributes zero edges for an all-shared
# set, regardless of overlap — it is rule 1 (same-shared-status overlap),
# not rule 3, that would serialize two OVERLAPPING shared surfaces.
assert SequencingService()._shared_last_edges(s) == []
def test_cycle_is_rejected() -> None:
with pytest.raises(SequencingError):
SequencingService()._toposort([(0, 1), (1, 0)], 2)
@@ -218,6 +239,27 @@ def _edge_set(pairs: list[tuple[object, object]]) -> set[tuple[object, object]]:
return set(pairs)
def _has_cycle(pairs: list[tuple[object, object]]) -> bool:
"""True if the (depends_on, task) edge list contains a directed cycle."""
graph: dict[object, set[object]] = {}
for dep_on, task in pairs:
graph.setdefault(dep_on, set()).add(task)
visiting: set[object] = set()
done: set[object] = set()
def _visit(node: object) -> bool:
visiting.add(node)
for nxt in graph.get(node, ()):
if nxt in visiting or (nxt not in done and _visit(nxt)):
return True
visiting.discard(node)
done.add(node)
return False
nodes = {n for pair in pairs for n in pair}
return any(n not in done and _visit(n) for n in nodes)
def test_dev_collision_disjoint_surfaces_are_parallel() -> None:
# Same project, disjoint files → no edge (the two dev tasks run together).
a, b = (
@@ -354,6 +396,74 @@ def test_dev_collision_fallback_idempotent_on_rerun() -> None:
assert dev_task_collision_edges([a, b]) == dev_task_collision_edges([a, b])
def test_dev_collision_fallback_still_applies_when_another_pair_collides() -> None:
# Regression: a `if edges: return edges` short-circuit used to drop the
# assignee-lane fallback ENTIRELY whenever ANY surfaced pair produced a
# collision edge, even for a totally unrelated same-assignee pair with no
# declared surface at all. (a, b) collide on a.py (different assignees, so
# no lane relationship between them); (c, d) share an assignee/project but
# declare no surface — they must still get lane-ordered.
a = _Sib(
uuid4(),
sequence=0,
intends_to_touch=["a.py"],
assigned_to="be-dev-1",
)
b = _Sib(
uuid4(),
sequence=1,
intends_to_touch=["a.py"],
assigned_to="be-dev-2",
)
c = _Sib(uuid4(), sequence=2, assigned_to="be-dev-3")
d = _Sib(uuid4(), sequence=3, assigned_to="be-dev-3")
edges = _edge_set(dev_task_collision_edges([a, b, c, d]))
assert edges == {(a.id, b.id), (c.id, d.id)}
def test_dev_collision_fallback_covers_unsurfaced_sibling_in_surfaced_lane() -> None:
# Same assignee/project lane mixes a surfaced sibling (touches a.py) with
# an unsurfaced one (no declared surface) and a third surfaced sibling
# that doesn't overlap the first — the analyzer alone wires nothing for
# this lane (no pair overlaps), so the fallback must still chain all three
# by (priority, sequence).
first = _Sib(uuid4(), sequence=0, assigned_to="be-dev-1", intends_to_touch=["a.py"])
bare = _Sib(uuid4(), sequence=1, assigned_to="be-dev-1")
other = _Sib(uuid4(), sequence=2, assigned_to="be-dev-1", intends_to_touch=["b.py"])
edges = dev_task_collision_edges([first, bare, other])
assert edges == [(first.id, bare.id), (bare.id, other.id)]
def test_dev_collision_fallback_never_closes_cycle_against_analyzer() -> None:
# Regression: the analyzer's shared-last migration order inverts priority
# order (s3 before s1), while the same-assignee lane fallback chains by
# priority through the unsurfaced middle sibling (s1 -> s2 -> s3). Naively
# unioning the two closed a 3-cycle s1 -> s3 -> s2 -> s1 that made
# add_dependency raise ConflictError and wedged every later delegate. The
# analyzer edge wins; the fallback edge that would cycle is dropped.
s1 = _Sib(
uuid4(),
priority=1,
sequence=0,
assigned_to="be-dev-1",
adds_migration=True,
touches_shared=True,
)
s2 = _Sib(uuid4(), priority=2, sequence=1, assigned_to="be-dev-1") # unsurfaced
s3 = _Sib(
uuid4(),
priority=3,
sequence=2,
assigned_to="be-dev-1",
adds_migration=True,
touches_shared=False,
)
edges = dev_task_collision_edges([s1, s2, s3])
assert not _has_cycle(edges)
assert (s3.id, s1.id) in edges # authoritative analyzer edge preserved
assert (s2.id, s3.id) not in edges # the cycling fallback edge is dropped
# ---------------------------------------------------------------------------
# cell_task_wave_chain_depends_on — the cell-task wave chain (edge kind 2).
# Pure glue: a new cell-task under root-subtask UT_n depends on every cell-task
+126
View File
@@ -8,6 +8,7 @@ Secretary-owned and held for the CEO. Asserted against a real Postgres DB.
from __future__ import annotations
import asyncio
from typing import TYPE_CHECKING, cast
from unittest.mock import AsyncMock
from uuid import UUID, uuid4
@@ -21,6 +22,7 @@ from roboco.models.base import AgentRole, AgentStatus, Complexity, Team
from roboco.models.base import TaskStatus as TS
from roboco.services import video_engine as video_engine_module
from roboco.services.company_goals import get_company_goals_service
from roboco.services.heartbeat_mutex import HeartbeatLockUnavailable
from roboco.services.task import VIDEO_POST_SOURCE, VIDEO_SOURCE, get_task_service
from sqlalchemy import delete, select
@@ -99,6 +101,27 @@ def _mock_local_model(monkeypatch: pytest.MonkeyPatch, reply: str | None) -> Asy
return mock
class _AlwaysAcquiredMutex:
"""Stand-in for ``HeartbeatMutex``: always acquires immediately, no live
Redis required (matches the project's ``_no_live_redis`` fixture). Used
as the default so every scenario below that isn't specifically testing
the occasion-lock behavior is unaffected by its introduction."""
def __init__(self, *_args: object, **_kwargs: object) -> None:
pass
async def acquire(self) -> str | None:
return "tok"
async def release(self, _token: str) -> None:
return None
@pytest.fixture(autouse=True)
def _stub_occasion_lock(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(video_engine_module, "HeartbeatMutex", _AlwaysAcquiredMutex)
# --------------------------------------------------------------------------- #
# open_video_task
# --------------------------------------------------------------------------- #
@@ -203,6 +226,109 @@ async def test_open_video_task_dedupes_same_occasion(
assert len(open_tasks) == ONE
@pytest.mark.asyncio
async def test_open_video_task_returns_none_when_occasion_lock_held(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A held per-occasion lock (another call already owns it) is a no-op,
not an error it opens nothing and leaves the shared session usable."""
await _seed(db_session)
_enable(monkeypatch)
class _HeldMutex:
def __init__(self, *_a: object, **_kw: object) -> None:
pass
async def acquire(self) -> str | None:
return None # another call already holds this occasion's lock
async def release(self, _token: str) -> None:
raise AssertionError("release must not be called when acquire failed")
monkeypatch.setattr(video_engine_module, "HeartbeatMutex", _HeldMutex)
engine = video_engine_module.VideoEngine(db_session)
task = await engine.open_video_task(
occasion="release v1.0.0", script="s", platforms=["x"], brief="b"
)
assert task is None
assert await get_task_service(db_session).list_open_video_posts() == []
@pytest.mark.asyncio
async def test_open_video_task_returns_none_when_lock_unavailable(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A Redis outage on the occasion lock fails closed (no-op), not a crash."""
await _seed(db_session)
_enable(monkeypatch)
class _BrokenMutex:
def __init__(self, *_a: object, **_kw: object) -> None:
pass
async def acquire(self) -> str | None:
raise HeartbeatLockUnavailable("redis down")
async def release(self, _token: str) -> None:
raise AssertionError("release must not be called when acquire failed")
monkeypatch.setattr(video_engine_module, "HeartbeatMutex", _BrokenMutex)
engine = video_engine_module.VideoEngine(db_session)
task = await engine.open_video_task(
occasion="release v1.0.0", script="s", platforms=["x"], brief="b"
)
assert task is None
assert await get_task_service(db_session).list_open_video_posts() == []
@pytest.mark.asyncio
async def test_open_video_task_concurrent_same_occasion_creates_only_one(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Regression (engine dedup race): ``open_video_task`` is reachable from
several genuinely concurrent callers for the same occasion (a double-click
on ``/video/request``, or an on-demand call racing the release hook)
unlike the other engines' single-loop ``run_cycle``, two overlapping calls
are NOT serialized by the orchestrator's own scheduling. A real
``asyncio.Lock`` stands in for the Redis SET NX mutex's mutual exclusion
(no live Redis in tests): the first caller to reach the DB's genuinely
suspending await wins the lock and creates the task; the second finds it
held and returns None immediately never both passing the dedup check."""
await _seed(db_session)
_enable(monkeypatch)
engine = video_engine_module.VideoEngine(db_session)
real_lock = asyncio.Lock()
class _RaceMutex:
def __init__(self, *_a: object, **_kw: object) -> None:
pass
async def acquire(self) -> str | None:
if real_lock.locked():
return None
await real_lock.acquire()
return "tok"
async def release(self, _token: str) -> None:
real_lock.release()
monkeypatch.setattr(video_engine_module, "HeartbeatMutex", _RaceMutex)
results = await asyncio.gather(
engine.open_video_task(
occasion="race me", script="s1", platforms=["x"], brief="b1"
),
engine.open_video_task(
occasion="race me", script="s2", platforms=["x"], brief="b2"
),
)
created = [r for r in results if r is not None]
assert len(created) == ONE
open_tasks = await get_task_service(db_session).list_open_video_posts()
assert len(open_tasks) == ONE
@pytest.mark.asyncio
async def test_open_video_task_respects_open_cap(
db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch
@@ -274,7 +274,7 @@ async def test_approve_completes_when_unconfigured_platform_is_skipped(
assert result.status == "posted"
assert result.posted == {"x": "x-vid-1"}
assert "skipped (unconfigured): tiktok" in result.detail
assert tiktok_poster.calls == [] # never attempted without credentials
assert tiktok_poster.calls == []
await db_session.refresh(task)
assert task.status == TS.COMPLETED
draft = markers.get_video_draft(task)
@@ -967,7 +967,7 @@ async def test_approve_concurrent_caption_edit_does_not_erase_a_committed_posted
commit the retry re-posted the platform (a double-post)."""
task = await _seed_video_post(db_session, platforms=["x", "tiktok"])
task_id = _id(task)
await db_session.commit() # externally visible to the "concurrent" session below
await db_session.commit()
real_get = TaskService.get
injected = False
+206 -14
View File
@@ -7,6 +7,8 @@ fixture) so approve exercises the real post + status-transition path.
from __future__ import annotations
import contextlib
from contextlib import contextmanager
from typing import TYPE_CHECKING, cast
from unittest.mock import AsyncMock, patch
from uuid import uuid4
@@ -25,7 +27,12 @@ from roboco.models.base import (
from roboco.models.base import TaskNature as TN
from roboco.models.base import TaskStatus as TS
from roboco.models.base import TaskType as TT
from roboco.services.task import X_FEATURE_SOURCE, X_POST_SOURCE, X_REPLY_SOURCE
from roboco.services.task import (
X_FEATURE_SOURCE,
X_POST_SOURCE,
X_REPLY_SOURCE,
TaskService,
)
from roboco.services.x_client import XClient, XMention, XPostResult
from roboco.services.x_post_service import (
TaskAlreadyCompletedError,
@@ -34,18 +41,34 @@ from roboco.services.x_post_service import (
XPostService,
get_x_post_service,
)
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
async_sessionmaker,
create_async_engine,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from uuid import UUID
from sqlalchemy.ext.asyncio import AsyncSession
SYSTEM_UUID = _foundation.AGENTS["system"].uuid
SECRETARY_UUID = _foundation.AGENTS["secretary-1"].uuid
ONE = 1
TWO = 2
@contextmanager
def _lock_free() -> Iterator[None]:
"""Patch XPostService's lock helpers so approve/reject exercise the real
post/cancel path without touching the (test-blocked) Redis."""
with (
patch.object(XPostService, "_acquire_lock", AsyncMock(return_value="tok")),
patch.object(XPostService, "_release_lock", AsyncMock(return_value=None)),
):
yield
class _StubClient(XClient):
def __init__(self, *, posted: bool = True, tweet_id: str = "999") -> None:
self._posted = posted
@@ -268,7 +291,7 @@ async def test_approve_no_credentials_result(db_session: AsyncSession) -> None:
assert result is not None
assert result.status == "no_credentials"
await db_session.refresh(task)
assert task.status == TS.PENDING # never advanced without credentials
assert task.status == TS.PENDING
@pytest.mark.asyncio
@@ -326,7 +349,8 @@ async def test_approve_refuses_already_rejected_draft(
refuses and never calls the X client the reproduced bug (a stale
Approve after reject re-posting)."""
task = await _seed_draft(db_session)
await _svc(db_session).reject(_id(task), "not on-brand")
with _lock_free():
await _svc(db_session).reject(_id(task), "not on-brand")
client = _StubClient()
with (
patch("roboco.services.x_post_service.build_x_client", return_value=client),
@@ -387,17 +411,39 @@ async def test_approve_unknown_task_returns_none(db_session: AsyncSession) -> No
@pytest.mark.asyncio
async def test_reject_records_reason_and_cancels(db_session: AsyncSession) -> None:
task = await _seed_draft(db_session, source=X_REPLY_SOURCE)
updated = await _svc(db_session).reject(_id(task), "Tone doesn't match our voice")
with _lock_free():
updated = await _svc(db_session).reject(
_id(task), "Tone doesn't match our voice"
)
assert updated is not None
assert updated.status == TS.CANCELLED
assert markers.get_x_reject_reason(updated) == "Tone doesn't match our voice"
@pytest.mark.asyncio
async def test_reject_refused_while_lock_held_by_concurrent_approve(
db_session: AsyncSession,
) -> None:
"""A concurrent approve holds the post lock (mid-tweet-POST); reject must
fail closed instead of racing a CANCEL under it previously reject()
never even attempted the lock, so it could commit CANCELLED to a draft a
concurrent approve was about to mark COMPLETED, or clobber the approve's
outcome depending on commit ordering."""
task = await _seed_draft(db_session)
with patch.object(XPostService, "_acquire_lock", AsyncMock(return_value=None)):
result = await _svc(db_session).reject(_id(task), "not relevant")
assert result is None
await db_session.refresh(task)
assert task.status == TS.PENDING
assert markers.get_x_reject_reason(task) is None
@pytest.mark.asyncio
async def test_list_open_posts_excludes_terminal(db_session: AsyncSession) -> None:
open_task = await _seed_draft(db_session)
rejected_task = await _seed_draft(db_session, source=X_REPLY_SOURCE)
await _svc(db_session).reject(_id(rejected_task), "not relevant")
with _lock_free():
await _svc(db_session).reject(_id(rejected_task), "not relevant")
open_posts = await _svc(db_session).list_open_posts()
ids = {t.id for t in open_posts}
assert open_task.id in ids
@@ -450,13 +496,73 @@ async def test_reject_completed_raises(db_session: AsyncSession) -> None:
await _svc(db_session).reject(_id(task), "nope")
@pytest.mark.asyncio
async def test_reject_concurrent_approve_completes_during_lock_wait(
db_session: AsyncSession, _test_database_url: str
) -> None:
"""Redis mutex pre-lock write audit regression for ``reject()``: a
genuinely concurrent approve (a real second session/connection) posts +
commits COMPLETED in the window between reject's pre-lock read and its
lock acquisition. The in-lock re-read must see that committed state and
refuse the CANCELLED status write and reject reason must never land on
the just-posted row, proving the fix holds across sessions, not merely
within one. Mirrors
``test_approve_concurrent_edit_does_not_clobber_a_committed_post``."""
task = await _seed_draft(db_session)
task_id = _id(task)
await db_session.commit()
real_get = TaskService.get
injected = False
async def _get_then_inject_concurrent_post(
self: TaskService, tid: UUID
) -> TaskTable | None:
"""Fires once, right after reject's pre-lock read — the exact window
between that read and reject's own (would-be) pre-lock write."""
nonlocal injected
result = await real_get(self, tid)
if not injected:
injected = True
other, other_engine = await _fresh_session(_test_database_url)
try:
other_task = await other.get(TaskTable, tid)
assert other_task is not None
markers.set_x_posted_tweet_id(other_task, "concurrent-999")
other_task.status = TS.COMPLETED
await other.commit()
finally:
await _dispose(other, other_engine)
return result
with (
patch.object(TaskService, "get", _get_then_inject_concurrent_post),
patch.object(XPostService, "_acquire_lock", AsyncMock(return_value="tok")),
patch.object(XPostService, "_release_lock", AsyncMock(return_value=None)),
pytest.raises(TaskAlreadyCompletedError),
):
await _svc(db_session).reject(task_id, "Tone doesn't match")
fresh, fresh_engine = await _fresh_session(_test_database_url)
try:
final = await fresh.get(TaskTable, task_id)
assert final is not None
assert final.status == TS.COMPLETED
assert markers.get_x_posted_tweet_id(final) == "concurrent-999"
# The reject must never have landed on the just-posted row.
assert markers.get_x_reject_reason(final) is None
finally:
await _dispose(fresh, fresh_engine)
@pytest.mark.asyncio
async def test_list_post_history_excludes_open_drafts(
db_session: AsyncSession,
) -> None:
open_task = await _seed_draft(db_session)
rejected_task = await _seed_draft(db_session, source=X_REPLY_SOURCE)
await _svc(db_session).reject(_id(rejected_task), "not relevant")
with _lock_free():
await _svc(db_session).reject(_id(rejected_task), "not relevant")
history = await _svc(db_session).list_post_history()
ids = {t.id for t in history}
assert rejected_task.id in ids
@@ -468,7 +574,8 @@ async def test_list_post_history_newest_acted_first(
db_session: AsyncSession,
) -> None:
rejected_task = await _seed_draft(db_session, source=X_REPLY_SOURCE)
await _svc(db_session).reject(_id(rejected_task), "not relevant")
with _lock_free():
await _svc(db_session).reject(_id(rejected_task), "not relevant")
posted_task = await _seed_draft(db_session)
client = _StubClient()
with (
@@ -495,7 +602,8 @@ async def test_list_post_history_includes_marker_fields(
):
await _svc(db_session).approve(_id(posted_task))
rejected_task = await _seed_draft(db_session, source=X_REPLY_SOURCE)
await _svc(db_session).reject(_id(rejected_task), "off-brand tone")
with _lock_free():
await _svc(db_session).reject(_id(rejected_task), "off-brand tone")
history = await _svc(db_session).list_post_history()
by_id = {t.id: t for t in history}
@@ -508,7 +616,8 @@ async def test_list_post_history_respects_limit(db_session: AsyncSession) -> Non
tasks = []
for _ in range(3):
t = await _seed_draft(db_session, source=X_REPLY_SOURCE)
await _svc(db_session).reject(_id(t), "not relevant")
with _lock_free():
await _svc(db_session).reject(_id(t), "not relevant")
tasks.append(t)
history = await _svc(db_session).list_post_history(limit=2)
assert len(history) == TWO
@@ -550,6 +659,86 @@ async def test_approve_does_not_flush_edited_body_before_lock(
assert markers.get_x_draft_body(task) == original_body
async def _fresh_session(url: str) -> tuple[AsyncSession, AsyncEngine]:
"""A session on a brand-new engine/connection (caller disposes)."""
engine = create_async_engine(url, future=True)
factory = async_sessionmaker(
bind=engine, class_=AsyncSession, expire_on_commit=False
)
return factory(), engine
async def _dispose(session: AsyncSession, engine: AsyncEngine) -> None:
with contextlib.suppress(Exception):
await session.rollback()
await engine.dispose()
@pytest.mark.asyncio
async def test_approve_concurrent_edit_does_not_clobber_a_committed_post(
db_session: AsyncSession, _test_database_url: str
) -> None:
"""Redis mutex pre-lock write audit regression: a genuinely concurrent
approve (a real second session/connection, not an in-process mock) posts
+ commits COMPLETED in the window between our pre-lock read and our lock
acquisition. The in-lock re-read must see that committed state and the
CEO's edited body must never land on the just-posted row — proving the
fix holds across sessions, not merely within one, mirroring
VideoPostService's identical cross-session regression test."""
task = await _seed_draft(db_session, body="Original")
task_id = _id(task)
await db_session.commit()
real_get = TaskService.get
injected = False
async def _get_then_inject_concurrent_post(
self: TaskService, tid: UUID
) -> TaskTable | None:
"""Fires once, right after the outer pre-lock read — the exact
window between our read and our own (would-be) pre-lock write."""
nonlocal injected
result = await real_get(self, tid)
if not injected:
injected = True
other, other_engine = await _fresh_session(_test_database_url)
try:
other_task = await other.get(TaskTable, tid)
assert other_task is not None
markers.set_x_posted_tweet_id(other_task, "concurrent-999")
other_task.status = TS.COMPLETED
await other.commit()
finally:
await _dispose(other, other_engine)
return result
client = _StubClient()
with (
patch("roboco.services.x_post_service.build_x_client", return_value=client),
patch.object(TaskService, "get", _get_then_inject_concurrent_post),
patch.object(XPostService, "_acquire_lock", AsyncMock(return_value="tok")),
patch.object(XPostService, "_release_lock", AsyncMock(return_value=None)),
):
result = await _svc(db_session).approve(task_id, "Edited body")
assert result is not None
assert result.status == "already_posted"
assert result.tweet_id == "concurrent-999"
# No double-post: the concurrently-committed tweet wins, ours never fires.
assert client.calls == []
fresh, fresh_engine = await _fresh_session(_test_database_url)
try:
final = await fresh.get(TaskTable, task_id)
assert final is not None
assert final.status == TS.COMPLETED
assert markers.get_x_posted_tweet_id(final) == "concurrent-999"
# The edit must never have landed on the just-posted row.
assert markers.get_x_draft_body(final) == "Original"
finally:
await _dispose(fresh, fresh_engine)
# --------------------------------------------------------------------------- #
# Spotlight video hook (Task 4, 2026-07-09 pipeline fixes): moved from
# authoring time (propose_feature_spotlight) to this posted-success branch so
@@ -684,9 +873,12 @@ async def test_reject_feature_spotlight_with_wants_video_opens_none(
task = await _seed_feature_draft(db_session)
video_engine = AsyncMock()
video_engine.open_video_task = AsyncMock(return_value=None)
with patch(
"roboco.services.video_engine.get_video_engine",
return_value=video_engine,
with (
patch(
"roboco.services.video_engine.get_video_engine",
return_value=video_engine,
),
_lock_free(),
):
updated = await _svc(db_session).reject(_id(task), "not on-brand")
assert updated is not None