mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Chore/all/logical gaps sweep (#286)
* release-manager: fencing-token mutex + executor/readiness hardening
Closes the release-mutex TTL race (#17, HIGH) and the remaining
release-manager gaps (#88, #89, #201, #202):
- #17: the release mutex is now acquired with a uuid4 fencing token and
released via Lua compare-and-del; a background asyncio heartbeat
compare-and-expires the TTL ~every 60s while the execute owns the lock,
so a live execute no longer expires and a crashed one auto-releases
<=3000s. A second approve after TTL expiry cannot usurp and rm -rf the
in-flight clone — the fenced first-finally keeps its lock.
- #89: a Redis outage during acquire stays fail-closed (the execute never
runs) but now returns a distinct redis_unavailable result + log so the
CEO sees the cause instead of a false already_in_progress.
- #88: commit_and_push RuntimeError is wrapped into a structured
ReleaseResult(commit_failed) instead of a 500.
- #201: first-release fallback still emits untracked version-ref files as
gaps (no longer silenced by the first-release branch).
- #202: _await_proc awaits proc.wait() after kill() so a timeout cannot
leak a zombie.
TDD: tests/unit/services/test_release_proposal_concurrency.py extends
_FakeRedis with eval/get/expire and pins the fencing/heartbeat/usurper
invariants + the redis_unavailable result.
* PM/code-task creation guard + main_pm coverage + issue carve-out
Closes the creation-time role x task_type gap (the user's explicit example)
and the main_pm delegate hole:
- New pure helper `pm_cannot_own_code(role, task_type, is_issue_resolution)`
in foundation/policy/batch.py — single source of truth. Both PM roles
(cell_pm + main_pm) coordinate; a `code` task assigned/claimed by a PM is
a structural mismatch, EXCEPT a PM taking a code task in needs_revision to
resolve review/QA issues directly (the carve-out).
- Creation-time guard: TaskService.create calls the helper (closes the
create-with-cell-PM-assignee hole the team-based check misses).
- Delegate path + spec claim gate consult the same helper.
`_validate_assignee_task_type` / `_task_type_hint_for` now key on the
Role (CELL_PM OR MAIN_PM), not the cell-PM slug set — closes the
delegate-to-main-pm-as-code hole.
- identity.role_for_uuid_or_none is None-tolerant (treats None as "not a
PM" and proceeds) so a malformed/missing assignee cannot crash the guard.
- prompter.create_task_from_draft reuses the guard at draft-create.
TDD: test_batch.py (helper matrix + carve-out), test_main_pm_code_guard.py
(main_pm coverage), test_delegate_assignee_task_type.py (delegate parity),
test_lifecycle_spec.py (claim gate: rejects PM claiming code from pending,
allows from needs_revision + PM claiming planning + dev claiming code).
* task-service: completion hooks + escalation/cancel/audit hardening
Closes the task-service cluster (#21/#98, #99, #100, #101, #103, #216; #102
verified already-covered, #217 verified already-guarded):
- #21/#98: ceo_approve now closes the work session + triggers completion
hooks before worktree removal (no-op when work_session_id is None), so a
CEO-approved task lands the same close-path as PM-completed.
- #99: apply_escalation routes through the transition validator with an
enumerated escalation exemption (_ESCALATABLE_TO_BLOCKED) instead of an
arbitrary source->BLOCKED write; BACKLOG is refused.
- #100: branchless ceo_reject awaiting_ceo_approval->pending gets a real
spec edge (ceo_reject_to_pool ActionSpec + _STATUS_TRANSITIONS entry) so
future admin-override tightening can't wedge the path.
- #101: revision_count bump is documented as the single chokepoint, with
the pre-block RESTORE path undoing it when restoring a snapshotted
needs_revision (same cycle resuming, not a new rejection).
- #103: cancel cascade surfaces non-terminal orphans instead of swallowing
the role violation.
- #216: _remove_task_worktree_on_terminal escalates recurring FS/permission
failure (audit/notify after N) instead of silent-failing forever.
- #102: pinned in test_verb_runner_midverb_invalid_state.py (committed with
the choreographer cluster) — verb-runner savepoints already surface a
concurrent mid-verb state change as INVALID_STATE.
- #217: submit_for_qa claimed_by guard verified intact.
TDD: test_task.py, test_worktree_cleanup_on_complete.py,
test_escalation_board_guard.py (#99), test_task_service_* integration,
test_lifecycle_spec.py.
* choreographer: gate-claim guards + pr-gate hardening + fail-open logging
Closes the choreographer cluster (#5/#222, #29, #30, #82, #188, #189,
#192; #157/#187 verified already-fixed/pinned; #102 pin lives here):
- #5/#222: the unchanged-PR guard's fail-open head_sha lookup now logs
(warning) on a slug-resolver/git-helper error so a regression cannot
silently turn the pr_fail re-submit loop-stopper into a no-op. Stays
fail-open (never wedges the PM).
- #29: pinned (REFUTED-with-pin) — _lane_claim_guard already returns the
error envelope without releasing the claim on a transient lookup error.
- #30: pinned (REFUTED-with-negative-pin) — a non-batch branchless main_pm
root cannot bypass the complete spec gate (is_batch_umbrella requires
batch_id set).
- #82: _post_gate_review_to_pr wraps the slug-resolution call in try/except
(mirrors _capture_pr_head_sha) so a malformed cell_map AttributeError no
longer 500s the reviewer after a committed gate transition.
- #188: _is_hand_formatted_verdict anchors the header regex to line-start,
so a quoted (> ## Summary) or inline (mid-prose) header mention no longer
false-refuses a hand-formatted verdict.
- #189: pr_fail re-captures the PR head SHA after the transition commits and
re-stamps the verdict note only when it advanced (closes the stale-SHA
false-allow loop-hole); no-advance stays a single note write.
- #192: claim_gate_review skips the dev claim guards (already_active/paused/
lane) via a new skip_dev_guards param — a pr_reviewer inspecting an
assembled PR does not start work, so the single-active-task / code-lane
invariants do not apply; the dependency guard is kept, and QA's
claim_review parity is preserved.
- #157/#187: verified in tree — pr_review-only handoff is intentionally
prior-work-worth-resuming; self_review_block wiring (reviewer != dev)
holds on assembled tasks with 4 existing pin tests.
TDD: test_choreographer_*, test_pr_gate_posts_review (#82),
test_pr_review_hand_format_guard (#188), test_submit_root_unchanged_pr_guard
(#189), test_claim_gate_review_guards (#192), test_verb_runner_midverb
_invalid_state (#102 pin).
* playbook curate: guard the gating commit against a poisoned session (#55)
The explicit `session.commit()` that gates the RAG index (commit-before-index
so an uncommitted playbook cannot land in the corpus) raised PendingRollbackError
when a prior mid-verb failure had rolled the caller's session back — 500ing the
whole curation verb instead of returning a clean envelope, and (worse) risking a
fall-through to index an uncommitted playbook. Wrap the commit: on
PendingRollbackError, log + return invalid_state with a re-fetch/retry remediate
and skip the index. The happy path still commits exactly once then indexes.
TDD: test_playbook_verbs.py — poisoned-session returns a clean invalid_state and
does NOT index; clean-session still commits once + indexes (pins no fail-closed
inversion / no double-commit).
* [chore] gateway: atomic activate merge — preserve probe_failures across re-park (#156)
activate() was a blind SET that reset probe_failures to 0, so a probe-failure
increment that just landed (or was in flight) could be wiped by a concurrent
re-park — resetting the give-up / CEO-notify count mid-episode. Route activate
through a server-side Lua merge (roboco:activate_rate_limit) that refreshes the
episode metadata (kind / activated_at / retry_after / affected_agents) while
carrying over the previous probe_failures count. Indivisible w.r.t. the
increment/reset scripts (Redis single-threads an EVAL).
#56 (notify to prompter/secretary refused) verified SAFE — the pin tests
(test_notify_rejects_prompter_recipient / _secretary_recipient /
_allows_ceo_recipient) already cover the only human notify target invariant;
no legitimate send is dropped, no code change.
* [chore] foundation/policy: spec gates + QA retry-key pin (Cluster F)
#50 sync_branch composes=() so the spec gate accepted a terminal/paused/
blocked task and the handler rebased a dead/parked branch — add a
PRECONDITION_SYNC_BRANCH_STATE (claimed/in_progress/verifying/needs_revision
only), rejection_kind=invalid_state. TDD: 28 spec tests.
#148 submit_root's prose asserts 'a Main-PM root is planning-typed, never
code' but only the creation path (main_pm_cannot_own_code) backed it — add
PRECONDITION_ROOT_NOT_CODE on the submit_root IntentSpec (defense in depth),
scoped to submit_root only so the shared submit_for_review action keeps
cell_pm+code submit_up parity. Graceful on Mock/None task_type so the
choreographer Mock-task tests don't crash. TDD: 2 spec tests.
#150 VERB_RETRY_LIMITS is keyed by the MCP-exposed names (pass/fail), not
the IntentSpec-internal pass_review/fail_review — already correct; add a
pin test so a one-sided rename can't silently drop the QA-handoff cap.
#142 main_pm_cannot_own_code/pm_cannot_own_code already normalize casing
(.lower()) — no-op, pin test test_main_pm_cannot_own_code_is_case_insensitive
already in tree.
* [chore] worksession-git: 405 merge-method fallback + non-destructive close (Cluster W)
#108 _merge_with_retry hardcoded 'squash' and raised MergeConflictError on a
405 with no method fallback — wedging the PM on an open, mergeable PR whose
repo merely had the squash button off. Add a 405 fallback to a permitted
method (via _first_allowed_merge_method, exclude='squash'), mirroring the CEO
merge_pull_request path. A 405 with no permitted fallback (or a second 405)
still falls through to the already-merged disambiguation / MergeConflictError.
TDD: 2 new tests (fallback-success, no-permitted-method-raises).
#109 close_pull_request defaulted delete_branch=True, so the choreographer
supersede path deleted a superseded PR's branch while the orchestrator
supersede path explicitly preserved it — the two disagreed, and the
destructive default ran on the 'close the dead PR' path where the branch may
still be referenced / useful for audit. Flip the default to False (opt-in
deletion) and make the choreographer caller explicit (parity with the
orchestrator). TDD: 1 new test (default preserves branch); existing
deletion-when-requested test now passes delete_branch=True explicitly.
Dispositions verified against current code (no silent drops):
- #27 REFUTED/FIXED-UNDEPLOYED: work_session.merge_pr resolves by session_id
(no global pr_number lookup); the real cross-repo collision fix
(project_id scoping on pr_merge/close_pull_request/rebase_pr_for_task/
pr_target) is already in tree + tested (test_pr_merge_scopes_task_lookup_
by_project_id, test_close_pull_request_scopes_task_lookup_by_project_id,
test_git_pr_target_scoping). Verify-only.
- #106 REFUTED: a guard exists (rev-list --count {base_ref}..{branch} == 0)
before reset --hard + base_ref falls back to default_branch; tests lock
the safety (test_create_branch_never_repoints_branch_with_real_work,
test_create_branch_does_not_reset_or_checkout_shared_clone).
- #218 BY-DESIGN: the merge_pr idempotent guard intentionally preserves the
audit trail (docstring + test_merge_pr_idempotent_on_already_completed_
preserves_audit_trail); a COMPLETED session always carries attribution
(COMPLETED only via merge_pr), so the NULL-COMPLETED case is unreachable.
- #104 BY-DESIGN: agents never merge to the repo default branch in RoboCo's
model (root→master is CEO-only); the guard is a correct CEO-only rail,
locked by test_pr_merge_into_default_branch_is_ceo_only.
* [chore] llm: surface disabled-provider downgrade + scrub probe log (#20/#3/#211)
#20/#3 resolve_for_agent silently fell through to the legacy Anthropic path
when a configured provider was disabled — indistinguishable from 'no
assignment', so the operator got no signal that spawns bypassed the
provider. Surface the bypass with a warning (graceful degradation stays the
default — a stalled spawn is worse than a routing miss) and add an opt-in
ROBOCO_ROUTING_STRICT (default-off) that fail-closes instead. Wired into the
panel Feature Flags card. TDD: 3 unit tests (warn-on-disabled, strict-raises,
no-assignment-stays-silent).
#211 probe_ollama_tags logged str(exc) raw on the generic-exception branch —
structured log could carry connection internals / stack traces. Log the
exception class name only. TDD: existing generic-branch test strengthened to
assert the log kwargs don't leak the raw text.
* [chore] support/stream/optimal/playbook/comms hardening (Cluster S)
Logical-gaps sweep, Cluster S (TDD, red→green per item):
#64 notification_delivery.acknowledge published the NOTIFICATION_ACKED bus
event directly (bypassing the outbox) — a rollback left a phantom ACK. Route
it through defer_bus_publish (after_commit), mirroring deliver.
#76 playbook.archive()/reject() stamped the archiver into approved_by/
approved_at, overwriting approval provenance (and fabricating approval for a
rejected draft). Add archived_by/archived_at (migration 053 + table + model)
and write those on archive/reject, leaving approval attribution intact.
#181 vector_store.replace_chunks wiped existing index rows even when every
chunk lacked an embedding (embedder failure). Skip the wipe when chunks is
non-empty but records is empty — preserve good rows for nothing.
#182/#183 optimal.record_learning recomputed a learn-{md5(full_content)}
tracking source that never matched the URI the plugin embedded chunks under
(roboco://learnings/{doc_id}, doc_id=lrn-{hash100}). Use the plugin's
returned doc_id so de-index/lookup-by-source finds the chunk rows.
#96/#97 transcription periodic flush only peeked ready buffers (unbounded
map growth) and ran sync callbacks on the event loop (a slow callback
blocked the flush task). Flush (remove) each ready buffer after notifying,
and offload each callback to a thread.
#212 _TEAM_SCOPED_ROLES was duplicated across communications/agents_config/
seeds. Single-source it in foundation.policy.communications; consumers
reference that object (identity-tested).
#19 stream_bus._dispatch_event re-ran already-succeeded handlers on a
recover_pending replay (duplicate side effects). Add a per-(event.id,
handler) SET-NX idempotency guard: skip on a hit, clear the key on handler
failure so a replay re-runs it, fail-open when redis is unavailable.
Dispositions (no code change): #77 approve() index-write pair asserted
BY-DESIGN; #62/#63 notification DB-dedup verified pinned; #184/#185 REFUTED;
#214 REFUTED; #215 BY-DESIGN.
* [chore] db/migrations: graph-integrity guard + conftest unreachable-DB warning (Cluster D)
Logical-gaps sweep, Cluster D (TDD + real alembic upgrade head verification):
#16/#37 add tests/unit/test_migration_graph_integrity.py — a static guard that
the alembic migration graph has exactly one head, every down_revision resolves,
every revision is reachable from a root, and no revision id is duplicated. The
suite builds its DB via Base.metadata.create_all (not alembic upgrade head), so
a forked head / dangling down_revision / duplicate id would otherwise ship
silently and break a real deploy mid-stream.
Caught a real bug in the process: migration 053's revision id
"053_playbook_archived_attribution" (33 chars) exceeded alembic's
alembic_version.version_num VARCHAR(32) — a fresh `alembic upgrade head` raised
"value too long for type character varying(32)" at the 053 stamp. Renamed to
"053_playbook_archived_attr" (26 chars). Verified end-to-end on a scratch PG:
upgrade head stamps 053, downgrade -1 returns to 052. (The pre-existing
test_every_migration_revision_id_fits_the_alembic_version_column guard is now
green too; it had been red on the 33-char id.)
#90 conftest silently pytest.skip'd every DB test when Postgres was unreachable
— a non-Docker box reported a green run of all-skips. Extract the warning into
_warn_if_pg_unavailable and fire it at import so the operator sees the DB is
down (the per-test skip path is unchanged). Test: warns when unavailable, silent
when reachable (verified under -W error::UserWarning).
Dispositions (verified against real code + a fresh alembic upgrade head, no code
change): #6 REFUTED — sa.Enum(create_type=False) at 001:119/304 does NOT break a
fresh upgrade head (001→052 applied cleanly on a scratch DB); #8 REFUTED — the
upgrade passed 030/031 (RAG chunk tables) without pgvector installed; pgvector is
a runtime concern handled by roboco/db/base.py, not a migration prerequisite;
#40 REFUTED — the `|| echo` mask was already removed and partial-schema drift
reports exit 1 (only by-design unreachable/unmigrated skips remain); #204 REFUTED
— the property walk seed IS pinned (random.Random(20260504), line 97); #205/#206
REFUTED — the smoke-trace fixture IS wired via
test_lifecycle_smoke_replay.py (8 passed); no shell smoke scripts exist in the
tree to wire; #137 BY-DESIGN — pyproject version 0.14.0 is an operational note,
no code gate.
* [chore] panel: admin-override force flag + kanban subtask_count + ws cleanup + ui-store dedupe (Cluster P)
#13: kanban admin-override into a hatch state (completed / awaiting_qa /
awaiting_pm_review) now requires an explicit force=true from the panel and
emits a dedicated task.admin_override audit row server-side; non-hatch
overrides need no force. Backend gate in tasks route + admin_set_status;
panel kanban-board sends force for hatch targets; TaskUpdate carries force.
#198: kanban service threads the real subtask_count (one grouped query) into
dev + priority-swimlane + main-pm-flat boards instead of a hardcoded 0.
#79: useWebSocket cleanup clears messages/lastMessage/state on unmount or
endpoint change so a dep-change (navigating to another stream) can't leak the
prior subscription's stale snapshot as live.
#186: disambiguate the duplicate ui-store modules -- the session/scroll store
in lib/stores renamed to useScrollRestorationStore / scroll-restoration-store
(barrel + 2 consumers updated); the sidebar/theme useUIStore in @/store is now
the sole useUIStore.
#12: verified already in-tree (release-proposal-card surfaces non-404 errors
with retry; getProposal maps only 404->null). #80 by-design (handleTransportError
already resets isSending on a no-payload SSE drop). #81 docs (streamUrl docstring
records that live-intake SSE auth is session-id-based bearer-style).
Backend: ruff+mypy clean, 278 tests green. Panel: lint+typecheck clean, 159 tests.
* orchestrator: park/reaper/readopt/a2a hardening + self-heal/ci-watch dedupe (Cluster O)
Closes the orchestrator-side logical gaps from the sweep:
- #75 a2a human-only drop surfaced: _dispatch_a2a_work logs the skip
("a2a request targets a human-only role; left as a notification for the
human (not spawned)") instead of silently dropping the target — the
CEO/secretary/prompter still see the notification; only the spawn is
suppressed. (orchestrator.py)
- #72 readopt liveness: _readopt_running_agents requires a non-stale live
claim (via _agent_holds_live_claim) and skips a zombie container so a
reaped-but-restart-readopted agent isn't double-counted as active.
- #74 shutdown drain: stop() calls _flush_respawn_tracker so the durable
respawn counter write-throughs aren't lost on a clean stop.
- #71 resolve_wait active-guard + deferred liveness: a rate_limit_lifted
WaitingRecord is only confirmed-live after a _confirm_resume_liveness
probe (deferred deletion _resume_confirm_delay=30.0), and an
already-active agent short-circuits the repark. Scoped to
rate_limit_lifted records (the only ones at risk of a false lift).
- #73 stuck-Claude kill: _maybe_kill_stuck_claude + _claude_stuck_kill_ttl
(config.claude_stuck_kill_seconds) — a live container whose heartbeat is
stale past the grace AND whose gateway probe is broken is killed+evicted,
not protected forever by the reaper's live-skip.
- #230 verified FIXED-UNDEPLOYED: _gateway_broken_past_grace already
requires N consecutive false-broken probes (not one flaky streak); no
change, test added to pin the N-consecutive invariant.
- #43 self-heal per-observation dedupe: a fingerprint collapses repeat
CEO notifications for the same CI regression.
- #44 ci_watch dedupe by (git_url, workflow): a monorepo's multiple
workflows each get their own fix task (was collapsed by git_url alone).
- #49 identity.role_for_slug_or_none None-hardening: a stale/malformed
slug resolves to None and the human-only skip falls through to the safe
"not spawnable" path instead of crashing.
- #193 strategy engine: notify the CEO on a persistent assess failure
instead of failing silently in the background loop.
TDD: test_no_spawn_human_roles (a2a skip surfaced), test_orchestrator_
shutdown_drain (#74), test_provider_overload_break (#71), test_readopt_
running_agents (#72), test_resolve_wait_repark (#71), test_stale_claim_
reaper (#73/#230), test_strategy_engine_loop (#193, new),
test_self_heal_engine (#43), test_ci_watch_engine (#44), test_identity
(#49). All red->green.
* chore: make-quality green — xenon complexity refactors + mypy test fixes + lifecycle regen
No behavior changes. Brings the tree to a fully green `make quality` (the
base branch never passed the xenon B-rank gate on several blocks; the
lifecycle artifacts had drifted from the committed ceo_reject_to_pool edge).
Xenon B-rank refactors (extract a helper; preserve semantics exactly):
- api/routes/tasks.py: _apply_forced_status_override + _StatusOverride
dataclass bundle (update_task override block).
- services/task.py: _enforce_no_pm_code_on_create (create guards) +
_escalation_diverts_to_pool (collapses the two board/advisory +
main_pm+code divert branches into one predicate).
- services/prompter.py: _coerce_pm_code_to_planning (create_task_from_draft).
- services/notification.py: _duplicate_unacked_exists (_create_notification
purpose-based dedup query + ACK_REQUIRED_BY_TYPE gate).
- services/sequencing.py: _same_assignee_lane_edges (the undeclared-surface
same-assignee lane fallback at the tail of dev_task_collision_edges).
- gateway/choreographer/_impl.py: _pm_task_type_error static helper
(_validate_assignee_task_type compound PM guard).
- gateway/choreographer/pr_gate.py: _gate_review_event_verdict +
_gate_review_body static helpers (_post_gate_review_to_pr).
mypy test fixes (no type:ignore — banned; use typing.cast with quoted
strings per TC006):
- test_task_update_completeness: TaskUpdate(acceptance_criteria=None).
- test_bus: cast("Redis", _FakeRedis()); Redis import under TYPE_CHECKING.
- test_pr_merge_concurrency: capture AsyncMocks into locals before asserting.
- test_notification_delivery_phantom: cast("UUID", to_agents[0]).
Lifecycle artifact regen (owed from Cluster T #100 — the
awaiting_ceo_approval -> pending `ceo_reject_to_pool` edge was added to the
spec in 3d633084 without regenerating the derived artifacts the
foundation-check gate diffs against): docs/rag/lifecycle/intent-verbs.md,
docs/rag/lifecycle/status-transitions.md, panel/lib/lifecycle.json.
services/kanban.py: ruff format only (collapses the _load_subtask_counts
signature that drifted unformatted from Cluster P).
* [chore] logical-gaps sweep — Cluster I (intake/product/pitch)
#57/#58 prompter: preserve a top-level product_id with a 1-cell map
(prompter.py create_task_from_draft — top-level target wins over a
redundant 1-cell map instead of dropping product_id); reject — not
silently skip — a malformed project_id in the_work cell entries
(prompter.py _draft_cell_map raises ValidationError).
#59/#159 prompter: create_task_from_draft now operates on a copy
(_copy_draft) so _validate_and_coerce_draft / _clean_list never mutate
the caller's draft dict.
#160 prompter: _resolve_owning_team consults product/board routing
before forcing MAIN_PM on a multi-cell map (product root stays Board,
product+assignee-is-board stays Board).
#83/#84 github_provisioning: create_repo is idempotent by GitHub name
— a 422 "name already exists" (orphaned repo from a rolled-back prior
approval) is fetched and reused instead of erroring; pitch re-approval
now reuses the orphaned repo end-to-end.
#196 kanban: flat main-PM board has a "coordination" column for
non-cell teams (MAIN_PM/Board) instead of dropping their cards.
#197 project update: an explicit null in the PATCH body now clears the
stored field, distinct from an absent field (leave unchanged).
ProjectService.update drops exclude_none so explicit-None applies; the
PATCH route uses ProjectUpdate.model_validate(data.model_dump(
exclude_unset=True)) to preserve the request's unset-tracking (the old
field-by-field construction marked every field set and defeated the
distinction — nulling NOT-NULL git_url).
TDD: prompter 47, github_provisioning+pitch 16, kanban+project 67,
project routes 37 — all green; ruff + mypy clean.
* [chore] logical-gaps sweep — Cluster M (mcp-servers)
#60 flow_server/do_server: the circuit-breaker substitution no longer
erases the fixable rejection — the original envelope (kind/message/
remediate) is nested as inner on a copy of the SDK's circuit_open
envelope (the SDK dict is not mutated in place). The agent still sees
WHY the verb failed, not just that the breaker tripped.
#61 flow_server/do_server: a 404 carrying a *descriptive* detail
(not FastAPI's bare default {"detail":"Not Found"}) is now a
real resource not_found, surfaced as not_found so the agent
re-fetches state — instead of a misleading "server-side wiring gap"
invalid_state. The bare default and unparseable 404s still synthesize
the wiring-gap envelope; a 404 with a real Envelope (error field)
is still surfaced as-is.
#161 flow_server/do_server: dict error.code classification now uses
an exact-code map (authoritative for the codes the handlers emit) with
a substring fallback for unknown codes. Fixes the real regression:
AUTHENTICATION_REQUIRED carries no AUTHORIZED/DENIED/PERMISSION
substring, so the old substring-only rule dropped it to invalid_state
instead of not_authorized — an auth storm attributed as a state storm.
The fallback also adds AUTH so future AUTH-prefixed codes classify.
#162 flow_server/do_server: _register_tools gains a
ROBOCO_ALLOW_FULL_TOOLSET env override (default-off) so a missing
manifest falls back to the full tool set instead of raising — a
dev/test escape hatch. Production fail-loud behaviour is unchanged.
#163 intake_server: propose_batch accepts name as well as
title (intake drafts in the wild have used both), normalizing a
name-only draft onto a copy as title (caller's dict never mutated),
and reports the dropped count + reason in the return instead of
silently vanishing malformed drafts. The empty-batch hint now names
name as an alternative.
TDD: 123 mcp_servers tests green (14 new + 2 updated); ruff + mypy clean.
* [chore] Cluster N — conventions/docs logical-gaps sweep
#33: _create_new_doc/_update_existing_doc now resolve via
_resolve_contained_path (the RAG-returned update path was not containment-
checked — an escaping source could write/overwrite outside the docs dir).
#34: _commit_doc_to_repo returns committed/skipped/failed instead of
swallowing all exceptions; surfaced on DocRef.commit_status, the write
response, and the docs MCP guidance so a failed repo commit is fail-loud.
#35: write_doc only updates the similar doc when its filename matches — a
different filename creates a new file instead of collapsing onto the
similar doc's path (the dedup-overwrite defect codified by the old tests).
#129: a custom rule scoped to a language the validator never reports (a
typo) is surfaced as a warn finding on .roboco/conventions.yml via the
runner's once-per-run validation; #32 (tsx->typescript dialect) stays
BY-DESIGN.
#130: _cache_put only swallows a UNIQUE violation (23505) as a concurrent
duplicate; a non-unique IntegrityError (FK/NOT NULL/check) is log-errored
and re-raised instead of being silently misattributed.
#132: health re-reads the live file status (a cached degraded row hid an
in-place repair at a stale head key); get_map skips cached degraded rows
and stops caching degraded so a repaired file re-derives. #134 BY-DESIGN.
#133: _DB_METHODS gains stream/stream_scalars (SQLAlchemy 2.0 streaming
constructs are data access too — a route calling them is not thin).
#199: regenerate_verb_tables._annot_str strips Annotated[...] metadata
(BeforeValidator) before rendering; regenerated verbs.md + per-role
prompts so the BeforeValidator(func=...) repr (with a memory address) no
longer leaks into agent-facing prompt text.
TDD: 206 conventions/docs tests green (incl. 5 new files / appended
cases); ruff + mypy clean.
* [chore] Cluster 16 — cross-cutting hygiene logical-gaps sweep
Disposition + fix the 10 cross-cutting-hygiene gaps, TDD. make quality green
(ruff, mypy 944 files, pytest, xenon, vulture, foundation-check, enum-parity).
FIX:
- #24 /ws/system now gated by _require_panel_token (matches every sibling
/ws/* stream); rejects a missing token in strict mode and a forged token
even in dev. (roboco/api/websocket.py)
- #25 two drifted _require_ceo implementations (orchestrator router vs release
handler) unified on a single require_ceo_role helper in deps — same 403,
same role set, accepts Role/AgentRole/"ceo". (roboco/api/deps.py,
routes/orchestrator.py, routes/release.py)
- #11 a spawn session for a delivery role (developer/qa/documenter) with no
task_id now logs an unattributed-usage warning via is_unattributed_delivery_spawn.
(roboco/runtime/orchestrator.py)
- #65 pricing returns a structured CostResult(cost_usd, unpriced, is_anthropic)
so an unpriced Anthropic model (real spend we'd undercount) is flagged
instead of silently $0; calculate_cost stays a thin float wrapper.
(roboco/billing/pricing.py, billing/__init__.py)
- #67 blocker-metrics "blocked since" reads the task.blocked audit transition
(indexed on target_id/event_type/timestamp), not updated_at — which
over-counted when a blocked task was touched for a non-blocking reason.
Falls back to updated_at/created_at only with no audit row.
(roboco/services/metrics.py)
- #94 grok refresh_if_stale uses double-checked locking (_refresh_lock +
_recheck_or_refresh) so two concurrent callers don't both POST the
single-use refresh grant and burn the credential. (grok_auth.py)
DOCS (fix the doc, behavior already correct/pinned by tests):
- #66 get_summary docstring corrected — it sums raw agent_spawn_sessions rows
(sub-day precise); daily_usage_rollups/get_today_summary can diverge for
"today" until the sweeper catches up. (roboco/services/usage.py)
- #68 DashboardStorage is a documented in-memory stub; added a test pinning
that auditor flags are lost on storage reset (persisting = a migration +
service refactor, out of scope as a half-implementation).
(tests/integration/test_dashboard_service.py)
BY-DESIGN (no code change, with file:line evidence):
- #28 dashboard reads are open to the authenticated operator (dashboard.py:36
documents this); mutating auditor routes already gate via
_require_auditor_or_ceo. Role-gating reads would break the panel (no
X-Agent-ID on dashboard reads) and CEO-token-gating the router would block
the Auditor (auditor token != CEO token). nginx is the prod boundary.
REFUTED (narrowing would reintroduce a documented hang):
- #93 the ~/.grok directory mount (vs a single auth.json file) is load-bearing
— a single-file bind mount pins the inode so the atomic tmp.replace refresh
doesn't propagate to running containers (they hang at grok's login prompt).
Already documented in grok.py:161 and locked by
test_intake_grok_mounts_subscription_auth_when_present.
Incidental gate-greening (mypy errors a stale .mypy_cache had hidden in
earlier-cluster test files; xenon refactors for the new B-threshold):
- tests/unit/test_regenerate_verb_tables.py: type the dynamic-module loader.
- tests/unit/services/test_prompter.py: annotate the draft dict as dict[str,Any].
- tests/unit/services/test_conventions_cache_put.py: _FakeOrig is a real Exception
(IntegrityError's orig arg requires BaseException).
- metrics._blocked_since_map extracted from get_blocker_metrics (complexity).
- intake_server._normalize_batch_drafts extracted from propose_batch (complexity).
* Docs update
* [bug] spawn: self-heal vanished clone + branch ref before worktree ensure (be-dev-1 fatal loop)
A vanished clone_root (disk loss / /data/workspaces wipe / manual cleanup)
fatal-looped the resume path: _ensure_worktree_before_spawn ran
`git -C <missing>` and released the claim, but the reaper-style release
preserves assigned_to + branch_name so the next dispatch is a RESUME
(create_branch never re-runs to re-clone) and the same missing clone failed
every ~30s.
- workspace.py: ensure_worktree_self_heal re-attaches a present worktree +
symlinks the shared .venv; on a missing local branch ref it fetches from
origin (create_branch pushes at claim time, so pushed work survives) and
re-creates the ref, falling back to -b origin/HEAD only when the branch
was never pushed. _fetch_branch_ref is the token-aware fetch helper.
- orchestrator.py: _ensure_worktree_before_spawn health-checks the clone
and re-clones via ensure_workspace BEFORE the worktree self-heal. Fatal
git-state (WorkspaceError) still releases the claim + aborts; transient
failures abort without releasing (a fresh claim wouldn't help and
re-cloning is destructive).
TDD: 21 new + 61 related worktree/git/cancel/cleanup tests green; ruff +
mypy clean.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This commit is contained in:
Renzo F
+5
-1
@@ -25,7 +25,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
|
||||
- **Loop-prone notifications now have a bounded re-fire guard.** `TASK_ASSIGNMENT` / `REVIEW_REQUEST` / `DOCUMENTATION_REQUEST` / `BROADCAST` can be re-fired by a coordinator PM every tick while a task sits in a state, flooding inboxes. The existing DB purpose-dedup never fires for these four (`ACK_REQUIRED_BY_TYPE` marks them `requires_ack=False`, so the dedup is gated off), and the delivery path (`_persist_and_deliver`) had no dedup at all — so a wedged task re-sent the same signal every cycle, inflating each recipient's unacked set and driving respawn churn. A 60s Redis `SET NX` window per `(type, sender, recipient, task)` now coalesces the re-fire on both creation chokepoints (`NotificationService._create_notification` and `NotificationDeliveryService._persist_and_deliver`): the first fire acquires (marks) keys for fresh recipients, subsequent fires within the window are suppressed when no recipient was fresh, and the storm converges. Fail-open: Redis unavailable → never suppress (a notification is never dropped over dedup infra). One-shot types (`KNOWLEDGE_SHARE` / `MENTION` / `A2A_REQUEST`) bypass entirely (distinct content per send, no dedup key).
|
||||
|
||||
- **A whole-codebase logic-gap audit — roughly 140 concurrency, scoping, signal, and lifecycle gaps fixed.** The dominant body of this release. The categories: **cross-repo PR scoping** — `pr_number` and `branch_name` are per-repo but were stored and looked up unscoped, so two tasks on different repos sharing a PR number could merge the wrong repo's PR or skip the org's own in-flight integration PR; every PR-merge and branch-ownership lookup is now `project_id`-scoped, and `close_pull_request` / `pr_target` make `project_id` mandatory. **Advisory locks closing TOCTOU races** — per-agent on claim, per-parent on `delegate`, per-task on `open_pr` (preventing a milestone double-emit), plus an atomic server-side Redis probe-failure counter and a single-transaction `replace_chunks` (delete+insert) closing a reindex race. **Audit-row transactionality** — status-transition audit rows and the rework counter are written in-session in the caller's transaction (the old fire-and-forget path is gone), so the audit trail can't diverge from the state change. **Signal gaps** — `pr_fail` now pushes the reviewer's issues to the owning cell PM (the re-submit loop where a PM respawned into `needs_revision` blind and re-submitted the same PR is closed), and `fail_qa` routes a `needs_revision` dev task back to the dev, never the pool. **Asyncio cleanup** — `OptimalService.close()` cancels its startup indexing task before the periodic task and the plugin clear, so it can't write against closed plugins. **Conventions standard** — the validator now times out and reaps on hang, and the gate fails closed on resolution errors (a broken standard can no longer silently disable the gate). **WebSocket** — fan-out is non-blocking with finally-disconnect, idle-timeout, and dead-socket reaping on send error. **Orchestrator runtime** — it drains its fire-and-forget background set on shutdown and stops in lifespan shutdown before closing the DB; the probe-resume loop actually revives parked agents; the grok auth token is refreshed before expiry and parked (not crash-retried) when missing. **Release executor** — every subprocess (git/make/gh/clone) is deadline-bounded and it fails closed on a git add/commit before push. Dozens more across org-memory (private-leak closures, playbook index/unindex as a post-commit step so the RAG corpus never leads the status transaction), the reaper, the provider-park/overload break, and the live-chat bridges. The full categorized tracker lives in `docs/internal` (gitignored).
|
||||
- **A whole-codebase logic-gap sweep — 230 deduped regression risks plus the PM/code-task creation guard, every one dispositioned against the live tree.** The dominant body of this release. Each item was read against the real code first (four of the prior batch's Highs had been false alarms, so the inventory was never trusted blindly), then TDD-fixed; the dispositions ran 86 FIX, 78 BY-DESIGN (intentional/documented tradeoffs, with the silent-swallow-only cases reclassified to FIX with logging added), 18 REFUTED (the cited code already guards it), and 8 DOCS. The categories: **cross-repo PR scoping** — `pr_number` and `branch_name` are per-repo but were stored and looked up unscoped, so two tasks on different repos sharing a PR number could merge the wrong repo's PR or skip the org's own in-flight integration PR; every PR-merge and branch-ownership lookup is now `project_id`-scoped, and `close_pull_request` / `pr_target` make `project_id` mandatory. **Advisory locks closing TOCTOU races** — per-agent on claim, per-parent on `delegate`, per-task on `open_pr` (preventing a milestone double-emit), plus an atomic server-side Redis probe-failure counter and a single-transaction `replace_chunks` (delete+insert) closing a reindex race. **Audit-row transactionality** — status-transition audit rows and the rework counter are written in-session in the caller's transaction (the old fire-and-forget path is gone), so the audit trail can't diverge from the state change. **Signal gaps** — `pr_fail` now pushes the reviewer's issues to the owning cell PM (the re-submit loop where a PM respawned into `needs_revision` blind and re-submitted the same PR is closed), and `fail_qa` routes a `needs_revision` dev task back to the dev, never the pool. **Asyncio cleanup** — `OptimalService.close()` cancels its startup indexing task before the periodic task and the plugin clear, so it can't write against closed plugins. **Conventions standard** — the validator now times out and reaps on hang, and the gate fails closed on resolution errors (a broken standard can no longer silently disable the gate). **WebSocket** — fan-out is non-blocking with finally-disconnect, idle-timeout, and dead-socket reaping on send error. **Orchestrator runtime** — it drains its fire-and-forget background set on shutdown and stops in lifespan shutdown before closing the DB; the probe-resume loop actually revives parked agents; the grok auth token is refreshed before expiry and parked (not crash-retried) when missing. **Release executor** — every subprocess (git/make/gh/clone) is deadline-bounded and it fails closed on a git add/commit before push. Dozens more across org-memory (private-leak closures, playbook index/unindex as a post-commit step so the RAG corpus never leads the status transaction), the reaper, the provider-park/overload break, and the live-chat bridges. The single HIGH was the release-mutex TTL race (its own bullet below). The full readjusted mapping — every gap → disposition → `file:line` → how it works now — lives in `docs/internal/how-it-works-now-2026-06-30.md` (gitignored).
|
||||
|
||||
- **The release mutex is now fenced + heartbeated (the sweep's single HIGH).** `ReleaseProposalService.approve` guarded the fail-closed `ReleaseExecutor` with a Redis `SET NX EX` lock, but the lock held a static value (no fencing token), had no heartbeat, and its TTL (~50 min) was shorter than the worst-case clone+gate+CI+publish run (~90 min). On TTL expiry a second CEO approve re-acquired and `_prepare_release_clone` `rm -rf`'d the in-flight shared clone. The lock now carries a uuid4 fencing token; release is a Lua compare-and-del that only fires when `GET == token` (a late first-finally cannot delete a usurper's lock); a background heartbeat refreshes the TTL every 60 s while execute owns it. A Redis outage is distinguished from a held lock and both stay fail-closed (`redis_unavailable` vs `already_in_progress`).
|
||||
|
||||
- **The 2026-06-27 live-run meltdown cluster — root-caused and closed.** A run hit several compounding wedges at once, each TDD-fixed and verified green: a `main_pm` assigned a `code`-typed task is a structural impossibility (a coordinator PM does no coding) and is now hard-rejected at the gate; `cell_pm_complete` resolved a merge by global `pr_number` and merged the wrong repo's PR (closed by the cross-repo `project_id` scoping above); `submit_root` re-submitted an unchanged PR into an infinite `pr_fail` loop (now hard-gated); `fail_qa` bounced a dev task to the pool instead of back to the dev; a `note(scope='handoff')` with an empty section crashed the note path and tripped a PM respawn loop; the MegaTask four-layer hierarchy (umbrella → root → cell → dev) hit a depth cap sized for three layers; and the durable respawn counter's persist raced under fire-and-forget (an atomic upsert closes it).
|
||||
|
||||
@@ -45,6 +47,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
|
||||
- **The local LLM was bumped to `glm-5.2` and the Ollama fleet defaults swapped off minimax.** The in-house RAG / hybrid-retrieval model and the default fleet model assignment move to `glm-5.2:cloud`; a stale minimax default that no longer matched the running fleet is cleared.
|
||||
|
||||
- **Agent-facing RAG docs and generated prompts readjusted to the post-fix behavior.** The per-task worktree model (F123), the `/app/.venv` is-sacred rule, the PM/code-task invariant, and the `MAX_TASK_DEPTH=4` MegaTask hierarchy are now documented in the RAG corpus (`docs/rag/architecture/workspaces.md`, `workflows/task-claiming.md`, `workflows/git-commits.md`, `workflows/task-planning.md`, `roles/developer.md`) and the generated verb/status tables, so a respawned agent resumes against current guidance instead of the pre-fix model.
|
||||
|
||||
## [0.13.0] - 2026-06-26
|
||||
|
||||
### Added
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
| Verb | Body schema |
|
||||
|------|-------------|
|
||||
| `complete` | `complete(task_id: UUID, notes: str)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, intends_to_touch: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | None = None, intends_to_touch: list[str] | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `escalate_up` | `escalate_up(task_id: UUID, reason: str)` |
|
||||
| `give_me_work` | `give_me_work()` |
|
||||
| `i_am_idle` | `i_am_idle()` |
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
| Verb | Body schema |
|
||||
|------|-------------|
|
||||
| `complete` | `complete(task_id: UUID, notes: str)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, intends_to_touch: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | None = None, intends_to_touch: list[str] | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `escalate_to_ceo` | `escalate_to_ceo(task_id: UUID, reason: str)` |
|
||||
| `escalate_up` | `escalate_up(task_id: UUID, reason: str)` |
|
||||
| `give_me_work` | `give_me_work()` |
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
| `give_me_work` | `give_me_work()` |
|
||||
| `i_am_blocked` | `i_am_blocked(task_id: UUID, reason: str, blocker_type: str | None = None, what_needed: str | None = None)` |
|
||||
| `i_am_idle` | `i_am_idle()` |
|
||||
| `pass_review` | `pass_review(task_id: UUID, notes: str, ac_verdicts: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None)` |
|
||||
| `pass_review` | `pass_review(task_id: UUID, notes: str, ac_verdicts: list[str] | None = None)` |
|
||||
| `resume` | `resume(task_id: UUID)` |
|
||||
| `unclaim` | `unclaim(task_id: UUID)` |
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ real tools live in their agent_sdk drivers, not role_config.
|
||||
| `give_me_work` | `give_me_work()` |
|
||||
| `i_am_blocked` | `i_am_blocked(task_id: UUID, reason: str, blocker_type: str | None = None, what_needed: str | None = None)` |
|
||||
| `i_am_idle` | `i_am_idle()` |
|
||||
| `pass_review` | `pass_review(task_id: UUID, notes: str, ac_verdicts: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None)` |
|
||||
| `pass_review` | `pass_review(task_id: UUID, notes: str, ac_verdicts: list[str] | None = None)` |
|
||||
| `resume` | `resume(task_id: UUID)` |
|
||||
| `unclaim` | `unclaim(task_id: UUID)` |
|
||||
|
||||
@@ -113,7 +113,7 @@ real tools live in their agent_sdk drivers, not role_config.
|
||||
| Verb | Body schema |
|
||||
|------|-------------|
|
||||
| `complete` | `complete(task_id: UUID, notes: str)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, intends_to_touch: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | None = None, intends_to_touch: list[str] | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `escalate_up` | `escalate_up(task_id: UUID, reason: str)` |
|
||||
| `give_me_work` | `give_me_work()` |
|
||||
| `i_am_idle` | `i_am_idle()` |
|
||||
@@ -151,7 +151,7 @@ real tools live in their agent_sdk drivers, not role_config.
|
||||
| Verb | Body schema |
|
||||
|------|-------------|
|
||||
| `complete` | `complete(task_id: UUID, notes: str)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, intends_to_touch: list[str] | BeforeValidator(func=<function coerce_str_list at 0x109dbdee0>, json_schema_input_type=PydanticUndefined) | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `delegate` | `delegate(parent_task_id: UUID, title: str, description: str, assigned_to: str, team: str, task_type: str, nature: str, estimated_complexity: str, acceptance_criteria: list[str], project_id: UUID | None = None, covers_parent_criteria: list[str] | None = None, intends_to_touch: list[str] | None = None, adds_migration: bool = False, touches_shared: bool = False, depends_on: list[UUID] | None = None)` |
|
||||
| `escalate_to_ceo` | `escalate_to_ceo(task_id: UUID, reason: str)` |
|
||||
| `escalate_up` | `escalate_up(task_id: UUID, reason: str)` |
|
||||
| `give_me_work` | `give_me_work()` |
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Add archived_by / archived_at to playbooks — distinct retirement attribution.
|
||||
|
||||
``archive`` (retire an APPROVED playbook) and ``reject`` (decline a DRAFT) both
|
||||
end in ARCHIVED, but they are distinct curation acts from ``approve``. Stamping
|
||||
the archiver into ``approved_by``/``approved_at`` overwrote the approval
|
||||
provenance (and fabricated approval attribution for a rejected draft that was
|
||||
never approved). These two columns record who retired it and when, leaving
|
||||
``approved_by``/``approved_at`` to record only the approval.
|
||||
|
||||
Revision ID: 053_playbook_archived_attr
|
||||
Revises: 052_task_cell_projects
|
||||
Create Date: 2026-06-30
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "053_playbook_archived_attr"
|
||||
down_revision = "052_task_cell_projects"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"playbooks",
|
||||
sa.Column("archived_by", sa.UUID(as_uuid=True), nullable=True),
|
||||
)
|
||||
op.add_column(
|
||||
"playbooks",
|
||||
sa.Column(
|
||||
"archived_at",
|
||||
sa.DateTime(timezone=True),
|
||||
nullable=True,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("playbooks", "archived_at")
|
||||
op.drop_column("playbooks", "archived_by")
|
||||
@@ -19,6 +19,12 @@ Every branch, commit, and pull request carries the task ID it belongs to, so you
|
||||
|
||||
A branch is created automatically the moment an agent claims a task, and a **work session** tracks its branch, base, commits, files changed, and pull request from claim to merge.
|
||||
|
||||
## Per-task worktrees
|
||||
|
||||
An agent's clone is a single shared checkout, but a coordinator PM legitimately holds several in-progress roots at once. So instead of one checkout that each fresh claim `git reset --hard`s onto a new branch — destroying uncommitted work on the still-active first root — **each claimed task gets its own working tree** under the clone at `{clone}/.worktrees/{task-short}/`. The clone keeps the real `.git` object store and the shared `.venv`; each worktree symlinks that `.venv` so tools resolve without a per-task re-sync. The clone's `HEAD` is never moved by a claim, so a second task (or a PM's parallel roots) never overwrites a first task's uncommitted changes.
|
||||
|
||||
Branch-by-name operations (`push`, `pull`, `merge`) run from the clone root as before; only the checkout/HEAD-moving operations (`create_branch`, `commit`, `rebase`) target the task's worktree. Exactly one active work session exists per task at a time, so a re-claim supersedes any stale prior session and re-points the worktree at the new claim. When a task reaches a terminal state (completed or cancelled), its worktree is removed best-effort; a task that bounces back through `needs_revision` keeps its worktree to keep working.
|
||||
|
||||
## Work converges up a chain
|
||||
|
||||
Each developer works in their **own clone** and opens a pull request from their branch. Those flow upward:
|
||||
|
||||
@@ -55,9 +55,45 @@ ROBOCO_WORKSPACE_CLONE_TIMEOUT=300
|
||||
3. **Branch Flexibility**: Different branches simultaneously
|
||||
4. **Clean State**: Fresh clone if needed
|
||||
|
||||
## Per-Task Worktrees (F123)
|
||||
|
||||
Your agent clone is **shared across all your tasks**, but each **claimed task** gets its own linked git worktree — a separate working directory on the same underlying clone — so two of your in-progress tasks never clobber each other on one checkout (this is what lets a coordinator PM hold several roots at once, and what stops a fresh claim from `reset --hard`-ing uncommitted work on your still-active first task).
|
||||
|
||||
```
|
||||
{workspaces_root}/{project}/{team}/{agent}/ # the clone root (shared)
|
||||
├── .git/ # the real object store (shared)
|
||||
├── .venv/ # the per-project venv (shared, agent-owned)
|
||||
├── .uv-python/ # uv's managed CPython (shared, gitignored)
|
||||
└── .worktrees/
|
||||
└── {task-short}/ # ONE per claimed task — your cwd for that task
|
||||
├── .venv -> ../../.venv # symlink to the clone-root venv
|
||||
└── [your task's checked-out branch]
|
||||
```
|
||||
|
||||
- **Claim** (`i_will_work_on` / `claim_review` / `claim_doc_task`) adds a worktree at `.worktrees/{task-id-first-8}/` and checks out the task's branch there. The clone root's HEAD is **never moved** by a claim.
|
||||
- **Your container is started with `-w` pointing at the worktree** for your current task, so `commit`, edits, and `uv run` all resolve there automatically. Spawn resolves the worktree from your `current_task_id` on every spawn (never cached), so a resume/respawn re-attaches a pruned worktree before launch.
|
||||
- **The clone-root `.venv` is shared** — each worktree's `.venv` is a symlink to `../../.venv`, so `uv run` from a worktree resolves the clone-root venv. No per-worktree re-sync.
|
||||
- **Git ops split by kind**: checkout/HEAD-moving ops (`create_branch`, `commit`, `rebase`, `checkout`) target the worktree; branch-by-name ops (`push`, `pull`, `fetch`, `pr_merge`, `diff`) run from the clone root. You never do either by hand — the verbs resolve the worktree for you.
|
||||
- **One active WorkSession per task** is enforced both in the service layer and by a DB unique index — a re-claim (pool release, reaper unclaim, escalation redirect) supersedes any prior agent's stale session for that task.
|
||||
- **Claim rollback** (a mid-claim failure) `worktree remove --force`s the worktree so a retry doesn't collide with a stale one.
|
||||
- **Terminal completion** (`complete` / `ceo_approve`) removes the assignee's worktree best-effort, so finished tasks don't accumulate. A `needs_revision` bounce keeps the worktree — you need it back. The stale-claim reaper does **not** remove the worktree; it routes the task to `pending` for a re-claim that reuses it.
|
||||
|
||||
You do not manage any of this. The verbs do. The only thing you must know: **your cwd is the worktree for your current task, not the clone root** — so relative paths and `uv run` resolve against your task's checkout.
|
||||
|
||||
## The `/app/.venv` is sacred — never retarget onto it
|
||||
|
||||
Two venv classes exist in the container:
|
||||
|
||||
- **Workspace venvs** — per-project, agent-owned, under `/data/workspaces/.../{agent}/.venv`. These are yours.
|
||||
- **`/app/.venv`** — the image-baked MCP-gateway venv. The MCP servers (`roboco-flow`, `roboco-do`, the git-readonly server) import from here. **It is sacred. If it breaks, every tool you have stops spawning.**
|
||||
|
||||
A past live incident: an agent hit a permission error on its workspace venv, followed uv's hint to run `uv run --active`, and that retargeted onto `VIRTUAL_ENV=/app/.venv` (baked globally) — uv rebuilt `/app/.venv` from a drifted lock and deleted its `bin/`, bricking every MCP server spawn fleet-wide.
|
||||
|
||||
The bash-guard hook now **blocks** `uv run --active` and any `uv run` / `uvx` against `/app` (`--project /app`, `--directory /app`, `UV_PROJECT_ENVIRONMENT=/app`, `cd /app && uv ...`). If you ever feel tempted to use `--active` or point uv at `/app`, **don't** — call `i_am_blocked(reason='workspace venv broken')` instead and let the environment be rebuilt. Bare `uv run` (cwd-relative, your workspace venv) is always fine and always what you want.
|
||||
|
||||
## No Workspace Tools — It's Automatic
|
||||
|
||||
There are **no** agent-facing workspace tools. Workspaces are created and cloned for you by the orchestrator (`WorkspaceService`) before your container starts. You never `ensure`, `clone`, or `checkout` a workspace by hand — your repo is already on disk at the path below, and the gateway verbs (`i_will_work_on`, `claim_review`, ...) check out the right branch.
|
||||
There are **no** agent-facing workspace tools. Workspaces and per-task worktrees are created for you by the orchestrator (`WorkspaceService`) before your container starts. You never `ensure`, `clone`, `checkout`, or `worktree add` by hand — your repo is already on disk, the worktree for your current task is already linked and `-w`'d as your cwd, and the gateway verbs (`i_will_work_on`, `claim_review`, ...) check out the right branch in it.
|
||||
|
||||
## Workspace Resolution
|
||||
|
||||
|
||||
@@ -231,6 +231,8 @@ Main PM opens the root→master PR and moves the root task to awaiting_pr_review
|
||||
|
||||
**Pre side effects:** create_root_pr
|
||||
|
||||
**Preconditions:** root_not_code
|
||||
|
||||
|
||||
## submit_up
|
||||
|
||||
@@ -251,7 +253,7 @@ Rebase your task's branch onto its current base THROUGH the gate (raw git is den
|
||||
|
||||
**Composes:** (no atomic actions)
|
||||
|
||||
**Preconditions:** owns_task
|
||||
**Preconditions:** owns_task, sync_branch_state
|
||||
|
||||
|
||||
## triage
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
| awaiting_ceo_approval | cancelled | cancel | cell_pm, ceo, main_pm |
|
||||
| awaiting_ceo_approval | completed | ceo_approve | ceo |
|
||||
| awaiting_ceo_approval | needs_revision | ceo_reject | ceo |
|
||||
| awaiting_ceo_approval | pending | ceo_reject_to_pool | ceo |
|
||||
| awaiting_documentation | awaiting_pm_review | docs_complete | documenter |
|
||||
| awaiting_documentation | cancelled | cancel | cell_pm, ceo, main_pm |
|
||||
| awaiting_documentation | claimed | claim | documenter |
|
||||
|
||||
@@ -67,9 +67,10 @@ There is **no** `roboco_git_commit / _push / _create_pr / _merge_pr / _checkout`
|
||||
|
||||
## Branch Discipline
|
||||
|
||||
- Branches are auto-created on `i_will_work_on()`.
|
||||
- Branches are auto-created on `i_will_work_on()`, and each claimed task gets its own **per-task worktree** (your cwd for that task). See `docs/rag/architecture/workspaces.md`.
|
||||
- Don't checkout branches by hand — call the verb on the right task.
|
||||
- A drifted clone (after a respawn/resume) is now auto-recovered onto your task branch before you commit — you normally won't see `BRANCH_MISMATCH` at all. If you still do, uncommitted changes are blocking the switch: `commit(...)` your work (or `i_am_blocked` if the changes aren't yours), then continue.
|
||||
- Never run `uv run --active` or point uv at `/app` — it bricks the MCP-gateway venv. Bare `uv run` resolves your workspace venv. If the workspace venv is broken, `i_am_blocked(reason='workspace venv broken')`.
|
||||
|
||||
## Before Submitting to QA
|
||||
|
||||
|
||||
@@ -32,9 +32,10 @@ This automatically:
|
||||
1. Prefixes the commit with `[task-id-first-8-chars]`
|
||||
2. Validates the message via `commit_validator`
|
||||
3. Stages the listed files (or everything tracked + modified if omitted)
|
||||
4. Pushes to the agent's auto-created branch
|
||||
5. Records the commit on the task (`commits[]` field on `TaskTable`)
|
||||
6. Opens a PR through the choreographer when the task transitions out of `in_progress` (no separate `create_pr` call required)
|
||||
4. Commits **inside your task worktree** (`{clone_root}/.worktrees/{task-id-first-8}/`) — your cwd, never the clone root
|
||||
5. Pushes the task's recorded branch **by name** (independent of whatever the clone happens to be checked out on)
|
||||
6. Records the commit on the task (`commits[]` field on `TaskTable`)
|
||||
7. Opens a PR through the choreographer when the task transitions out of `in_progress` (no separate `create_pr` call required)
|
||||
|
||||
## Before Committing
|
||||
|
||||
@@ -43,6 +44,8 @@ This automatically:
|
||||
3. Run type check: `uv run mypy roboco/` or `pnpm typecheck`
|
||||
4. Format code: `uv run ruff format .` or `pnpm format`
|
||||
|
||||
**Never use `uv run --active` or point uv at `/app`.** Bare `uv run` is cwd-relative and resolves your workspace venv (symlinked into the worktree) — that's always what you want. `--active` retargets onto the image-baked `/app/.venv` (the MCP-gateway venv) and rebuilds it, bricking every tool you have. The bash-guard blocks it; if you ever feel pushed toward `--active`, call `i_am_blocked(reason='workspace venv broken')` instead. See `docs/rag/architecture/workspaces.md`.
|
||||
|
||||
## After Committing
|
||||
|
||||
You don't push or create a PR yourself. The choreographer pushed the commit during `commit()`, and the PR is opened/merged as part of the lifecycle transitions:
|
||||
|
||||
@@ -25,7 +25,9 @@ claim_doc_task(task_id) # Documenter
|
||||
# - assigned_to: your agent ID
|
||||
```
|
||||
|
||||
The claim verb both claims and starts the task — there is no separate `start` call. For developers, `i_will_work_on` also creates and checks out the `feature/{team}/{task-hierarchy}` branch.
|
||||
The claim verb both claims and starts the task — there is no separate `start` call. For developers, `i_will_work_on` also creates the `feature/{team}/{task-hierarchy}` branch and **adds a dedicated per-task worktree** at `{clone_root}/.worktrees/{task-id-first-8}/`, checking out the branch there. Your container is started with that worktree as its cwd, and the clone root's HEAD is never moved by the claim — so a second claim (or a coordinator PM's many parallel roots) never overwrites your first task's uncommitted work. See `docs/rag/architecture/workspaces.md` for the worktree model.
|
||||
|
||||
Exactly one active WorkSession exists per task at a time (enforced in the service layer and by a DB unique index). A re-claim — pool release, reaper unclaim, escalation redirect — supersedes any prior agent's stale active session for that task and re-points the worktree at the new claim.
|
||||
|
||||
## Before Claiming
|
||||
|
||||
|
||||
@@ -71,6 +71,20 @@ For code subtasks, delegate each developer their **full queue up front** rather
|
||||
|
||||
Caps still apply: at most 12 subtasks per parent, and same-title duplicate subtasks are rejected.
|
||||
|
||||
## PMs Do Not Own Code Tasks
|
||||
|
||||
A PM (Cell PM or Main PM) is a coordinator — it plans and delegates, it does not write code, and it has no code verb. The role×task_type rule is enforced at **creation**, not just at delegate:
|
||||
|
||||
- A `code`-typed task **cannot be assigned to a PM** — `delegate`, `TaskService.create`, batch activation, reassign, and the claim/escalation diversion all consult the same `pm_cannot_own_code` / `main_pm_cannot_own_code` guard and reject it.
|
||||
- A Main-PM coordination **root** that is `code`-typed is rejected by `submit_root`'s `PRECONDITION_ROOT_NOT_CODE` — a Main PM can never assemble+merge a code root, because it can't have written one.
|
||||
- The one exception — "a PM may take a code task **only to resolve review issues**" (`is_issue_resolution`) — is a server-side signal the platform sets when routing a `needs_revision` code task back to its owning PM to act on concrete review issues; it is **not** something you pass from a verb. In practice no live path exercises it yet; the structural rule is: if you're a PM and you're looking at a `code` task, delegate it to a developer instead.
|
||||
|
||||
This is structural, not a hint. Before this guard, a PM assigned a code task would claim it and deadlock into a respawn loop — a coordinator with no code verb holding a code task it can neither do nor hand back. The guard makes that loop unrepresentable.
|
||||
|
||||
## Delegation Depth
|
||||
|
||||
The task hierarchy is capped at `MAX_TASK_DEPTH = 4` levels (depths 0–3). The normal 3-layer flow (Main-PM root → cell task → dev subtask) fits in 3; **MegaTask** adds one Main-PM layer on top — umbrella (depth 0) → root-subtask (1) → cell task (2) → dev subtask (3) — which is why the cap is 4, not 3. A `delegate` that would create a node at depth 4 is rejected with a clean `invalid_state` and a "create as a sibling" remediation. Don't over-nest; if you're hitting the cap, the work belongs as a sibling, not a child.
|
||||
|
||||
## Git Workflow
|
||||
|
||||
All code tasks follow the git workflow:
|
||||
|
||||
@@ -444,6 +444,14 @@
|
||||
"source": "awaiting_ceo_approval",
|
||||
"target": "needs_revision"
|
||||
},
|
||||
{
|
||||
"action": "ceo_reject_to_pool",
|
||||
"roles": [
|
||||
"ceo"
|
||||
],
|
||||
"source": "awaiting_ceo_approval",
|
||||
"target": "pending"
|
||||
},
|
||||
{
|
||||
"action": "docs_complete",
|
||||
"roles": [
|
||||
|
||||
@@ -151,7 +151,7 @@ describe("KanbanBoard — admin-override bypass confirmation (F020)", () => {
|
||||
await waitFor(() => expect(mutateAsync).toHaveBeenCalledTimes(1));
|
||||
expect(mutateAsync).toHaveBeenCalledWith({
|
||||
taskId: "t1",
|
||||
updates: { status: TaskStatus.COMPLETED },
|
||||
updates: { status: TaskStatus.COMPLETED, force: true },
|
||||
});
|
||||
});
|
||||
|
||||
@@ -184,7 +184,7 @@ describe("KanbanBoard — admin-override bypass confirmation (F020)", () => {
|
||||
await waitFor(() => expect(mutateAsync).toHaveBeenCalledTimes(1));
|
||||
expect(mutateAsync).toHaveBeenCalledWith({
|
||||
taskId: "t1",
|
||||
updates: { status: TaskStatus.CLAIMED },
|
||||
updates: { status: TaskStatus.CLAIMED, force: false },
|
||||
});
|
||||
// No bypass confirmation should ever have been surfaced.
|
||||
expect(screen.queryByText(/override & move/i)).not.toBeInTheDocument();
|
||||
|
||||
@@ -54,6 +54,16 @@ interface PendingOverride {
|
||||
skipped: string[];
|
||||
}
|
||||
|
||||
// Terminal/hatch states the server-side PATCH override guard refuses without
|
||||
// an explicit `force: true` (the bypass is deliberate + audited). A kanban
|
||||
// drag is the admin override surface, so a move into one of these must carry
|
||||
// the acknowledgement or the backend rejects with 400.
|
||||
const HATCH_OVERRIDE_STATES: ReadonlySet<TaskStatus> = new Set([
|
||||
TaskStatus.COMPLETED,
|
||||
TaskStatus.AWAITING_QA,
|
||||
TaskStatus.AWAITING_PM_REVIEW,
|
||||
]);
|
||||
|
||||
interface ColumnConfig {
|
||||
id: string;
|
||||
status: TaskStatus;
|
||||
@@ -164,7 +174,10 @@ export function KanbanBoard({
|
||||
try {
|
||||
await updateTask.mutateAsync({
|
||||
taskId,
|
||||
updates: { status: newStatus },
|
||||
updates: {
|
||||
status: newStatus,
|
||||
force: HATCH_OVERRIDE_STATES.has(newStatus),
|
||||
},
|
||||
});
|
||||
toast.success(`Task moved to ${newStatus.replace(/_/g, " ")}`);
|
||||
refetch();
|
||||
@@ -179,7 +192,10 @@ export function KanbanBoard({
|
||||
try {
|
||||
await updateTask.mutateAsync({
|
||||
taskId: task.id,
|
||||
updates: { status: newStatus },
|
||||
updates: {
|
||||
status: newStatus,
|
||||
force: HATCH_OVERRIDE_STATES.has(newStatus),
|
||||
},
|
||||
});
|
||||
toast.success(`Task moved to ${newStatus.replace(/_/g, " ")}`);
|
||||
refetch();
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
import { useEffect, useRef } from "react";
|
||||
import { usePathname, useSearchParams } from "next/navigation";
|
||||
import { useUIStore } from "@/lib/stores/ui-store";
|
||||
import { useScrollRestorationStore } from "@/lib/stores/scroll-restoration-store";
|
||||
|
||||
/**
|
||||
* Global scroll restoration component.
|
||||
@@ -12,7 +12,8 @@ import { useUIStore } from "@/lib/stores/ui-store";
|
||||
export function ScrollRestoration() {
|
||||
const pathname = usePathname();
|
||||
const searchParams = useSearchParams();
|
||||
const { setScrollPosition, getScrollPosition, setLastVisited } = useUIStore();
|
||||
const { setScrollPosition, getScrollPosition, setLastVisited } =
|
||||
useScrollRestorationStore();
|
||||
const hasRestored = useRef(false);
|
||||
const prevRouteKey = useRef<string>("");
|
||||
|
||||
|
||||
@@ -46,6 +46,8 @@ const FLAG_DESCRIPTIONS: Record<string, string> = {
|
||||
"Run the deterministic release-readiness sweep and propose a release for you to approve or reject — it never publishes without your approval, and the executor is fail-closed on a red gate.",
|
||||
org_memory_enabled:
|
||||
"Close the learn→reuse loop: distill a lesson at task completion, index journal reflections, and auto-inject similar past lessons + approved playbooks into an agent's briefing on claim.",
|
||||
routing_strict:
|
||||
"Fail-closed model routing: refuse to silently downgrade an agent to the legacy Anthropic path when its configured provider is disabled (raise instead). Off => graceful degradation with a warning.",
|
||||
};
|
||||
|
||||
export function FeatureFlagsCard() {
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { render, act } from "@testing-library/react";
|
||||
import { useEffect } from "react";
|
||||
import type {
|
||||
ConnectionState,
|
||||
WebSocketOptions,
|
||||
} from "@/lib/websocket/connection";
|
||||
|
||||
// #79: useWebSocket's effect cleanup only disconnected the socket — it left the
|
||||
// `messages` / `lastMessage` / `state` snapshot behind. On a dependency change
|
||||
// (navigating from one agent/channel stream to another) the new subscription
|
||||
// inherited the previous stream's stale buffer until a fresh frame arrived.
|
||||
// Mock the connection so the test can drive onMessage/onStateChange and observe
|
||||
// the cleanup.
|
||||
const hoisted = vi.hoisted(() => {
|
||||
const instances: MockConnection[] = [];
|
||||
class MockConnection {
|
||||
url: string;
|
||||
onMessage?: (data: unknown) => void;
|
||||
onStateChange?: (state: ConnectionState) => void;
|
||||
didConnect = false;
|
||||
didDisconnect = false;
|
||||
constructor(opts: WebSocketOptions) {
|
||||
this.url = opts.url;
|
||||
this.onMessage = opts.onMessage;
|
||||
this.onStateChange = opts.onStateChange;
|
||||
instances.push(this);
|
||||
}
|
||||
connect() {
|
||||
this.didConnect = true;
|
||||
this.onStateChange?.("connecting");
|
||||
this.onStateChange?.("connected");
|
||||
}
|
||||
disconnect() {
|
||||
this.didDisconnect = true;
|
||||
this.onStateChange?.("disconnected");
|
||||
}
|
||||
}
|
||||
return { instances, MockConnection };
|
||||
});
|
||||
|
||||
vi.mock("@/lib/websocket/connection", () => ({
|
||||
getWebSocketUrl: () => "ws://test/ws",
|
||||
WebSocketConnection: hoisted.MockConnection,
|
||||
}));
|
||||
|
||||
vi.mock("@/lib/constants", () => ({
|
||||
CEO_AGENT_ID: "00000000-0000-0000-0000-000000000001",
|
||||
STREAM_MAX_MESSAGES: 100,
|
||||
}));
|
||||
|
||||
import { useWebSocket } from "../use-websocket";
|
||||
|
||||
interface Frame {
|
||||
type: "agent.stream";
|
||||
agent_id: string;
|
||||
chunk: string;
|
||||
}
|
||||
|
||||
// Capture the hook's latest return into a shared ref object (mutation, not
|
||||
// reassignment — the react-hooks/globals rule forbids the latter in render).
|
||||
const resultRef: {
|
||||
current: ReturnType<typeof useWebSocket<Frame>> | null;
|
||||
} = { current: null };
|
||||
|
||||
function Harness({ endpoint }: { endpoint: string }) {
|
||||
const ws = useWebSocket<Frame>(endpoint, undefined, true);
|
||||
// Capture the latest return after each render via a passive effect (keeps
|
||||
// render pure — the react-hooks rules forbid mutating shared state in render).
|
||||
useEffect(() => {
|
||||
resultRef.current = ws;
|
||||
});
|
||||
return null;
|
||||
}
|
||||
|
||||
describe("useWebSocket — clears snapshot on cleanup (#79)", () => {
|
||||
beforeEach(() => {
|
||||
hoisted.instances.length = 0;
|
||||
resultRef.current = null;
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it("clears messages/lastMessage/state when the endpoint changes (no stale leak)", () => {
|
||||
// 1. Subscribe to stream A and receive a frame.
|
||||
const { rerender } = render(<Harness endpoint="/agents/a" />);
|
||||
const connA = hoisted.instances[0];
|
||||
expect(connA.didConnect).toBe(true);
|
||||
act(() => {
|
||||
connA.onMessage?.({
|
||||
type: "agent.stream",
|
||||
agent_id: "a",
|
||||
chunk: "hello",
|
||||
});
|
||||
});
|
||||
expect(resultRef.current?.messages).toHaveLength(1);
|
||||
expect(resultRef.current?.lastMessage?.chunk).toBe("hello");
|
||||
expect(resultRef.current?.isConnected).toBe(true);
|
||||
|
||||
// 2. Navigate to stream B — the cleanup for A runs, then B connects. Before
|
||||
// the fix the buffer from A survived into B's subscription.
|
||||
act(() => {
|
||||
rerender(<Harness endpoint="/agents/b" />);
|
||||
});
|
||||
expect(connA.didDisconnect).toBe(true);
|
||||
expect(hoisted.instances).toHaveLength(2);
|
||||
expect(hoisted.instances[1].url).toContain("/agents/b");
|
||||
// The stale snapshot MUST be cleared.
|
||||
expect(resultRef.current?.messages).toEqual([]);
|
||||
expect(resultRef.current?.lastMessage).toBeNull();
|
||||
});
|
||||
|
||||
it("disconnects the connection on unmount", () => {
|
||||
const { unmount } = render(<Harness endpoint="/agents/a" />);
|
||||
const conn = hoisted.instances[0];
|
||||
unmount();
|
||||
expect(conn.didDisconnect).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -8,14 +8,14 @@
|
||||
|
||||
import { useEffect, useRef } from "react";
|
||||
import { usePathname, useSearchParams } from "next/navigation";
|
||||
import { useUIStore } from "@/lib/stores/ui-store";
|
||||
import { useScrollRestorationStore } from "@/lib/stores/scroll-restoration-store";
|
||||
|
||||
export function useScrollRestoration(
|
||||
scrollContainerRef?: React.RefObject<HTMLElement>,
|
||||
) {
|
||||
const pathname = usePathname();
|
||||
const searchParams = useSearchParams();
|
||||
const { setScrollPosition, getScrollPosition } = useUIStore();
|
||||
const { setScrollPosition, getScrollPosition } = useScrollRestorationStore();
|
||||
|
||||
// Create a unique key for current route including search params
|
||||
const routeKey = `${pathname}?${searchParams.toString()}`;
|
||||
|
||||
@@ -12,8 +12,11 @@ import {
|
||||
type EscalateRequest,
|
||||
} from "@/types";
|
||||
|
||||
// Type for task updates - allows any Task field to be updated
|
||||
export type TaskUpdate = Partial<Task>;
|
||||
// Type for task updates - allows any Task field to be updated. `force` is not a
|
||||
// Task field: it acknowledges that a kanban admin-override into a hatch state
|
||||
// (completed / awaiting_qa / awaiting_pm_review) bypasses the lifecycle gate,
|
||||
// matching the server-side PATCH override guard.
|
||||
export type TaskUpdate = Partial<Task> & { force?: boolean };
|
||||
|
||||
// Query keys
|
||||
export const taskKeys = {
|
||||
|
||||
@@ -90,10 +90,16 @@ export function useWebSocket<T>(
|
||||
connectionRef.current = connection;
|
||||
connection.connect();
|
||||
|
||||
// Cleanup on unmount or when dependencies change
|
||||
// Cleanup on unmount or when dependencies change. Disconnect AND clear the
|
||||
// snapshot — otherwise a dep change (navigating to another stream) leaves
|
||||
// the prior subscription's messages/lastMessage/state visible until a fresh
|
||||
// frame arrives, surfacing another stream's stale buffer as live (#79).
|
||||
return () => {
|
||||
connection.disconnect();
|
||||
connectionRef.current = null;
|
||||
setMessages([]);
|
||||
setLastMessage(null);
|
||||
setState("disconnected");
|
||||
};
|
||||
}, [enabled, endpoint, queryString]); // Stable dependencies
|
||||
|
||||
|
||||
@@ -86,8 +86,12 @@ export const prompterLiveApi = {
|
||||
return data;
|
||||
},
|
||||
|
||||
/** SSE URL the panel opens to watch the agent. EventSource sends no headers
|
||||
* (the route is keyed by the opaque session id on the trusted network). */
|
||||
/** SSE URL the panel opens to watch the agent. EventSource cannot send custom
|
||||
* headers, so the live-intake stream carries no `X-Agent-*` auth — the route
|
||||
* is authenticated solely by the opaque, unguessable session id on the
|
||||
* trusted internal network. By design: any session-id leakage grants stream
|
||||
* access, so session ids must be treated as bearer credentials (never logged
|
||||
* client-side, never put in a shareable URL). */
|
||||
streamUrl: (sessionId: string): string =>
|
||||
`${API_URL}/prompter/live/${sessionId}/stream`,
|
||||
|
||||
|
||||
@@ -1 +1 @@
|
||||
export { useUIStore } from "./ui-store";
|
||||
export { useScrollRestorationStore } from "./scroll-restoration-store";
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
/**
|
||||
* UI State Store
|
||||
* Scroll-restoration / session-navigation store.
|
||||
*
|
||||
* Persists UI state across navigation using Zustand with sessionStorage.
|
||||
* This handles state that doesn't belong in URL params but should survive navigation.
|
||||
* Persists scroll position and last-visited-route state across navigation using
|
||||
* Zustand with sessionStorage — state that doesn't belong in URL params but
|
||||
* should survive navigation. Renamed from the generic `useUIStore` to avoid a
|
||||
* name clash with the sidebar/theme UI store in `@/store`.
|
||||
*/
|
||||
|
||||
import { create } from "zustand";
|
||||
@@ -13,7 +15,7 @@ interface ScrollPosition {
|
||||
y: number;
|
||||
}
|
||||
|
||||
interface UIState {
|
||||
interface ScrollRestorationState {
|
||||
// Scroll positions per route
|
||||
scrollPositions: Record<string, ScrollPosition>;
|
||||
|
||||
@@ -42,7 +44,7 @@ interface UIState {
|
||||
getLastVisited: (section: string) => string | undefined;
|
||||
}
|
||||
|
||||
export const useUIStore = create<UIState>()(
|
||||
export const useScrollRestorationStore = create<ScrollRestorationState>()(
|
||||
persist(
|
||||
(set, get) => ({
|
||||
scrollPositions: {},
|
||||
+4
-10
@@ -340,16 +340,10 @@ def get_pm_for_agent(agent_id: str) -> str | None:
|
||||
# Cross-cell roles (MAIN_PM, AUDITOR, CEO, board) are not subject to team_scope;
|
||||
# only cell-member roles (DEVELOPER/QA/DOCUMENTER/CELL_PM) are filtered.
|
||||
|
||||
# Cell-member roles subject to team_scope filtering. Lifted to module scope so
|
||||
# tests and downstream consumers can introspect the rule.
|
||||
_TEAM_SCOPED_ROLES: Final[frozenset[_foundation.Role]] = frozenset(
|
||||
{
|
||||
_foundation.Role.DEVELOPER,
|
||||
_foundation.Role.QA,
|
||||
_foundation.Role.DOCUMENTER,
|
||||
_foundation.Role.CELL_PM,
|
||||
}
|
||||
)
|
||||
# Cell-member roles subject to team_scope filtering. Single-sourced in
|
||||
# ``foundation.policy.communications.TEAM_SCOPED_ROLES``; re-exported here so
|
||||
# tests and downstream consumers can introspect the rule under the legacy name.
|
||||
_TEAM_SCOPED_ROLES: Final[frozenset[_foundation.Role]] = _comms.TEAM_SCOPED_ROLES
|
||||
|
||||
|
||||
def _slugs_for_role_set(
|
||||
|
||||
@@ -409,6 +409,22 @@ def require_pm_or_above(role: Any, action: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def require_ceo_role(role: Any, *, action: str = "perform this action") -> None:
|
||||
"""Raise 403 unless ``role`` is the CEO (#25 — the single CEO-check).
|
||||
|
||||
The orchestrator router-level gate (header + HMAC token) and the release
|
||||
handler-level gate (``CurrentAgentContext``) had drifted apart on the role
|
||||
comparison; both now delegate here so the CEO check has one source of
|
||||
truth. Accepts an ``AgentRole`` / ``Role`` enum or the lowercase header
|
||||
string. ``action`` tailors the 403 detail message.
|
||||
"""
|
||||
if _role_value(role) != Role.CEO:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail=f"Only the CEO may {action}",
|
||||
)
|
||||
|
||||
|
||||
def require_developer_or_above(role: Any, action: str) -> None:
|
||||
"""Raise 403 unless caller is developer-or-above."""
|
||||
if _role_value(role) not in _DEVELOPER_OR_ABOVE_ROLES:
|
||||
|
||||
@@ -97,6 +97,7 @@ async def write_doc(
|
||||
created_at=doc_ref.created_at,
|
||||
updated_by=doc_ref.updated_by,
|
||||
updated_at=doc_ref.updated_at,
|
||||
commit_status=doc_ref.commit_status,
|
||||
),
|
||||
)
|
||||
except ValidationError as e:
|
||||
|
||||
@@ -9,7 +9,12 @@ from typing import Annotated
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, status
|
||||
|
||||
from roboco.api.deps import _check_agent_auth_token, get_orchestrator, set_orchestrator
|
||||
from roboco.api.deps import (
|
||||
_check_agent_auth_token,
|
||||
get_orchestrator,
|
||||
require_ceo_role,
|
||||
set_orchestrator,
|
||||
)
|
||||
from roboco.api.schemas.orchestrator import (
|
||||
AgentStatusResponse,
|
||||
OrchestratorStatusResponse,
|
||||
@@ -17,7 +22,7 @@ from roboco.api.schemas.orchestrator import (
|
||||
SpawnAgentRequest,
|
||||
WaitingAgentResponse,
|
||||
)
|
||||
from roboco.foundation.identity import Role
|
||||
|
||||
|
||||
# Orchestrator control routes (spawn / stop / resolve-wait / mark-waiting,
|
||||
# plus the read-only status views) are operator/CEO control surfaces — any
|
||||
@@ -30,10 +35,8 @@ from roboco.foundation.identity import Role
|
||||
# the same contract as the v1 flow role guards and the do router. CEO is the
|
||||
# sole operator role; agents (developers/QA/PMs) drive the orchestrator via
|
||||
# MCP verbs, not these HTTP routes, so a developer token is correctly 403'd
|
||||
# here.
|
||||
_CEO_ROLE = Role.CEO.value
|
||||
|
||||
|
||||
# here. The CEO role check itself delegates to ``require_ceo_role`` (#25 —
|
||||
# the single source of truth shared with the release routes).
|
||||
def _require_ceo(
|
||||
x_agent_id: Annotated[str, Header(alias="X-Agent-ID")],
|
||||
x_agent_role: Annotated[str, Header(alias="X-Agent-Role")],
|
||||
@@ -43,13 +46,7 @@ def _require_ceo(
|
||||
# Bind the role header to a verified token BEFORE trusting it (same
|
||||
# defense-in-depth contract as the v1 flow role guards in _role_dep.py).
|
||||
_check_agent_auth_token(x_agent_id, x_agent_role, x_agent_team, x_agent_token)
|
||||
# ``Role`` is a StrEnum so the lowercase header string compares equal to
|
||||
# its matching member.
|
||||
if x_agent_role.lower() != _CEO_ROLE:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only the CEO/operator may control the orchestrator",
|
||||
)
|
||||
require_ceo_role(x_agent_role, action="control the orchestrator")
|
||||
|
||||
|
||||
router = APIRouter(dependencies=[Depends(_require_ceo)])
|
||||
|
||||
@@ -200,26 +200,13 @@ async def update_project(
|
||||
|
||||
require_cell_access(agent, project.assigned_cell, "update")
|
||||
|
||||
# Convert request to service model
|
||||
update_data = ProjectUpdate(
|
||||
name=data.name,
|
||||
git_url=data.git_url,
|
||||
default_branch=data.default_branch,
|
||||
protected_branches=data.protected_branches,
|
||||
assigned_cell=data.assigned_cell,
|
||||
git_token=data.git_token,
|
||||
test_command=data.test_command,
|
||||
lint_command=data.lint_command,
|
||||
format_command=data.format_command,
|
||||
typecheck_command=data.typecheck_command,
|
||||
build_command=data.build_command,
|
||||
quality_command=data.quality_command,
|
||||
ci_watch_enabled=data.ci_watch_enabled,
|
||||
ci_watch_workflow=data.ci_watch_workflow,
|
||||
dep_update_command=data.dep_update_command,
|
||||
dep_update_paths=data.dep_update_paths,
|
||||
is_active=data.is_active,
|
||||
)
|
||||
# Convert request to service model, preserving the request's own
|
||||
# unset-tracking: only fields the body actually carried are "set" on the
|
||||
# ProjectUpdate, so ProjectService.update's exclude_unset applies only the
|
||||
# body fields — and an explicit null in the body clears the stored value
|
||||
# (distinct from absent = leave unchanged) (#197). Passing every field
|
||||
# explicitly here would mark them all set and defeat that distinction.
|
||||
update_data = ProjectUpdate.model_validate(data.model_dump(exclude_unset=True))
|
||||
|
||||
updated = await service.update(cast("UUID", project.id), update_data)
|
||||
await db.commit()
|
||||
|
||||
@@ -9,7 +9,7 @@ from typing import TYPE_CHECKING, cast
|
||||
|
||||
from fastapi import APIRouter, HTTPException, status
|
||||
|
||||
from roboco.api.deps import CurrentAgentContext, DbSession
|
||||
from roboco.api.deps import CurrentAgentContext, DbSession, require_ceo_role
|
||||
from roboco.api.schemas.release import (
|
||||
ReleaseExecuteResponse,
|
||||
ReleaseGapModel,
|
||||
@@ -18,7 +18,6 @@ from roboco.api.schemas.release import (
|
||||
ReleaseReportModel,
|
||||
)
|
||||
from roboco.foundation.policy.content import markers
|
||||
from roboco.models import AgentRole
|
||||
from roboco.services.release_proposal import get_release_proposal_service
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -30,11 +29,7 @@ router = APIRouter()
|
||||
|
||||
|
||||
def _require_ceo(agent: CurrentAgentContext) -> None:
|
||||
if agent.role != AgentRole.CEO:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only the CEO may view or act on release proposals",
|
||||
)
|
||||
require_ceo_role(agent.role, action="view or act on release proposals")
|
||||
|
||||
|
||||
def _status_value(task: "TaskTable") -> str:
|
||||
|
||||
+80
-16
@@ -4,6 +4,7 @@ Task API Routes
|
||||
Full CRUD operations and lifecycle management for tasks.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Annotated, Any, cast
|
||||
from uuid import UUID
|
||||
|
||||
@@ -45,6 +46,7 @@ from roboco.api.schemas.tasks import (
|
||||
task_to_response,
|
||||
transform_update_data,
|
||||
)
|
||||
from roboco.db.tables import TaskTable
|
||||
from roboco.enforcement import get_valid_transitions
|
||||
from roboco.exceptions import GitError, TaskLifecycleError
|
||||
from roboco.foundation.policy import task_completeness as tc
|
||||
@@ -64,10 +66,11 @@ from roboco.services.notification_delivery import (
|
||||
EscalationError,
|
||||
get_notification_delivery_service,
|
||||
)
|
||||
from roboco.services.permissions import TaskAction
|
||||
from roboco.services.permissions import AgentContext, TaskAction
|
||||
from roboco.services.task import (
|
||||
SoftBlockInput,
|
||||
TaskCreateRequest,
|
||||
TaskService,
|
||||
extract_original_developer,
|
||||
get_task_service,
|
||||
)
|
||||
@@ -76,6 +79,69 @@ from roboco.utils.converters import require_uuid
|
||||
router = APIRouter()
|
||||
_logger = get_logger(__name__)
|
||||
|
||||
# #13: lifecycle-bypass hatch states — a privileged PATCH into one of these is a
|
||||
# forced override that must carry the explicit ``force`` acknowledgement flag.
|
||||
_HATCH_OVERRIDE_STATES = frozenset(
|
||||
{
|
||||
TaskStatus.COMPLETED,
|
||||
TaskStatus.AWAITING_QA,
|
||||
TaskStatus.AWAITING_PM_REVIEW,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _StatusOverride:
|
||||
"""Bundle of ``update_task`` override params (keeps the helper ≤ 5 args)."""
|
||||
|
||||
service: TaskService
|
||||
task_id: UUID
|
||||
task: TaskTable
|
||||
new_status: TaskStatus
|
||||
force: bool
|
||||
has_higher_perms: bool
|
||||
agent: AgentContext
|
||||
|
||||
|
||||
async def _apply_forced_status_override(req: _StatusOverride) -> TaskTable:
|
||||
"""Apply an audited admin status override, gating the lifecycle bypass.
|
||||
|
||||
Extracted from ``update_task`` so the route's complexity stays readable.
|
||||
Refuses a non-privileged caller, and refuses a bypass into a hatch state
|
||||
without the explicit ``force`` flag; otherwise delegates to the audited
|
||||
``admin_set_status`` and asserts the override landed.
|
||||
"""
|
||||
if req.new_status == req.task.status:
|
||||
return req.task
|
||||
if not req.has_higher_perms:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only privileged roles may override task status.",
|
||||
)
|
||||
if req.new_status in _HATCH_OVERRIDE_STATES and not req.force:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=(
|
||||
"Overriding a task into "
|
||||
f"{req.new_status.value} bypasses the lifecycle gate; pass "
|
||||
'"force": true to acknowledge the forced override.'
|
||||
),
|
||||
)
|
||||
task = await req.service.admin_set_status(
|
||||
req.task_id,
|
||||
req.new_status,
|
||||
actor_id=req.agent.agent_id,
|
||||
actor_role=getattr(req.agent, "role", None),
|
||||
force=req.force,
|
||||
)
|
||||
if not task:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail="Task status override failed unexpectedly",
|
||||
)
|
||||
return task
|
||||
|
||||
|
||||
# Minimum character count for notes fields that must be substantive
|
||||
# (QA pass notes, doc-complete notes, escalation notes). Below this the
|
||||
# note is useless for the next reader, so the transition is refused.
|
||||
@@ -892,6 +958,9 @@ async def update_task(
|
||||
# in-band transition. Pop it out of the generic field update and apply it
|
||||
# through the audited path, gated on elevated permissions.
|
||||
new_status = updates.pop("status", None)
|
||||
# #13: ``force`` is the explicit acknowledgement of the lifecycle bypass.
|
||||
# Pop it so it is never passed to TaskService.update as a field set.
|
||||
force = bool(updates.pop("force", False))
|
||||
|
||||
# Pop explicitly-set-to-None nullable fields. TaskService.update() skips
|
||||
# None values (not-None guard), so null-clear intent is re-applied directly
|
||||
@@ -909,23 +978,18 @@ async def update_task(
|
||||
# the MegaTask shape here too — a cleared parent_task_id / project_id must not
|
||||
# turn a root-subtask into an umbrella-shaped-but-targeted spoof.
|
||||
_reassert_batch_shape(task)
|
||||
if new_status is not None and new_status != task.status:
|
||||
if not has_higher_perms:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only privileged roles may override task status.",
|
||||
if new_status is not None:
|
||||
task = await _apply_forced_status_override(
|
||||
_StatusOverride(
|
||||
service=service,
|
||||
task_id=task_id,
|
||||
task=task,
|
||||
new_status=new_status,
|
||||
force=force,
|
||||
has_higher_perms=has_higher_perms,
|
||||
agent=agent,
|
||||
)
|
||||
task = await service.admin_set_status(
|
||||
task_id,
|
||||
new_status,
|
||||
actor_id=agent.agent_id,
|
||||
actor_role=getattr(agent, "role", None),
|
||||
)
|
||||
if not task:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail="Task status override failed unexpectedly",
|
||||
)
|
||||
await db.commit()
|
||||
return task_to_response(task)
|
||||
|
||||
|
||||
@@ -90,6 +90,7 @@ class DocRefResponse(BaseModel):
|
||||
created_at: str | None = None
|
||||
updated_by: str | None = None
|
||||
updated_at: str | None = None
|
||||
commit_status: str | None = None
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
|
||||
@@ -250,6 +250,13 @@ class TaskUpdate(BaseModel):
|
||||
# state with no valid in-band move), never as a free-form field set.
|
||||
status: TaskStatus | None = None
|
||||
|
||||
# #13: an explicit acknowledgement that the override pastes over the
|
||||
# lifecycle gate. Required for the terminal/final hatch states
|
||||
# (completed / awaiting_qa / awaiting_pm_review) so the bypass is deliberate
|
||||
# and audited as a forced override, not a free-form status set. Other
|
||||
# recovery overrides (e.g. blocked -> pending) do not require it.
|
||||
force: bool = False
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _reject_explicit_blank_acceptance_criteria(cls, data: Any) -> Any:
|
||||
|
||||
@@ -611,9 +611,16 @@ async def system_stream(websocket: WebSocket) -> None:
|
||||
|
||||
Carries system-level events for the control panel — currently the
|
||||
rate-limit lifecycle (``RATE_LIMIT_HIT`` / ``RATE_LIMIT_LIFTED``), bridged
|
||||
from the event bus by ``websocket_bridge``. No per-agent keying or auth:
|
||||
it's a read-only operator stream behind the panel's own access controls.
|
||||
from the event bus by ``websocket_bridge``. No per-agent keying; the
|
||||
panel/CEO token gate matches every sibling /ws/* stream (#24 — this was the
|
||||
only ungated /ws endpoint): in strict mode a missing CEO token closes with
|
||||
policy-violation, a presented-but-forged token is rejected even in dev.
|
||||
"""
|
||||
# Verify the panel/CEO token before subscribing — same gate as every other
|
||||
# /ws/* handler (#24).
|
||||
if not await _require_panel_token(websocket):
|
||||
await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
|
||||
return
|
||||
await manager.connect_system(websocket)
|
||||
|
||||
try:
|
||||
|
||||
@@ -3,6 +3,6 @@
|
||||
Provides token-cost calculation for Claude API models.
|
||||
"""
|
||||
|
||||
from roboco.billing.pricing import calculate_cost
|
||||
from roboco.billing.pricing import CostResult, calculate_cost, calculate_cost_result
|
||||
|
||||
__all__ = ["calculate_cost"]
|
||||
__all__ = ["CostResult", "calculate_cost", "calculate_cost_result"]
|
||||
|
||||
@@ -24,6 +24,8 @@ against missing pricing data.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import structlog
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
@@ -90,6 +92,24 @@ def _lookup_prices(lower: str) -> tuple[float, float, float, float] | None:
|
||||
return best_prices
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CostResult:
|
||||
"""Estimated cost plus pricing attribution (#65).
|
||||
|
||||
``cost_usd`` is ``0.0`` for both a genuinely-free non-Anthropic model (local
|
||||
inference — no per-token cost) and an unpriced Anthropic model (a Claude
|
||||
model we forgot to price — real spend we are failing to count). ``unpriced``
|
||||
distinguishes them so a caller can surface the miss instead of silently
|
||||
reporting ``$0``. ``is_anthropic`` records which family the model resolved
|
||||
to. ``calculate_cost`` returns just the ``cost_usd`` float for existing
|
||||
callers; ``calculate_cost_result`` returns the full attribution.
|
||||
"""
|
||||
|
||||
cost_usd: float
|
||||
unpriced: bool
|
||||
is_anthropic: bool
|
||||
|
||||
|
||||
def calculate_cost(
|
||||
model: str,
|
||||
tokens_input: int,
|
||||
@@ -99,11 +119,34 @@ def calculate_cost(
|
||||
) -> float:
|
||||
"""Calculate the estimated USD cost for a model invocation.
|
||||
|
||||
Thin wrapper over :func:`calculate_cost_result` returning just the USD
|
||||
float (kept for existing callers). See :func:`calculate_cost_result` for the
|
||||
provider-aware miss handling and the ``unpriced`` attribution.
|
||||
"""
|
||||
return calculate_cost_result(
|
||||
model,
|
||||
tokens_input=tokens_input,
|
||||
tokens_output=tokens_output,
|
||||
tokens_cache_read=tokens_cache_read,
|
||||
tokens_cache_write=tokens_cache_write,
|
||||
).cost_usd
|
||||
|
||||
|
||||
def calculate_cost_result(
|
||||
model: str,
|
||||
tokens_input: int,
|
||||
tokens_output: int,
|
||||
tokens_cache_read: int = 0,
|
||||
tokens_cache_write: int = 0,
|
||||
) -> CostResult:
|
||||
"""Calculate the estimated USD cost for a model invocation, with attribution.
|
||||
|
||||
Matches the model name against the known pricing table using substring
|
||||
search (longest match wins). Provider-aware (see module docstring):
|
||||
non-Anthropic models (local Ollama, Ollama Cloud) have no per-token cost
|
||||
and return 0.0 silently; an unpriced Anthropic model returns 0.0 but logs
|
||||
a warning since it represents real spend we are failing to count.
|
||||
and return ``cost_usd=0.0, unpriced=False``; an unpriced Anthropic model
|
||||
returns ``cost_usd=0.0, unpriced=True`` and logs a warning since it
|
||||
represents real spend we are failing to count.
|
||||
|
||||
Args:
|
||||
model: Model name or short alias (e.g. ``"claude-sonnet-4-6"``,
|
||||
@@ -118,24 +161,30 @@ def calculate_cost(
|
||||
``grok_cli_usage.usage_and_cost``).
|
||||
|
||||
Returns:
|
||||
Estimated cost in USD as a float. Returns 0.0 for unpriced models
|
||||
rather than raising.
|
||||
A :class:`CostResult` (``cost_usd``, ``unpriced``, ``is_anthropic``).
|
||||
``cost_usd`` is ``0.0`` rather than raising for unpriced models.
|
||||
"""
|
||||
if not model:
|
||||
return 0.0
|
||||
# An empty model name is a caller bug, not an unpriced-Anthropic miss.
|
||||
return CostResult(cost_usd=0.0, unpriced=False, is_anthropic=False)
|
||||
|
||||
lower = model.lower()
|
||||
is_anthropic = _is_anthropic_model(lower)
|
||||
|
||||
best_prices = _lookup_prices(lower)
|
||||
if best_prices is None:
|
||||
# No per-token rate. Warn only for Anthropic models (real spend we are
|
||||
# undercounting); non-Anthropic models are local/subscription-billed
|
||||
# and have no per-token cost, so an intentional 0.0 is correct.
|
||||
if _is_anthropic_model(lower):
|
||||
if is_anthropic:
|
||||
logger.warning("No pricing data found for Anthropic model", model=model)
|
||||
else:
|
||||
logger.debug("Non-Anthropic model has no per-token cost", model=model)
|
||||
return 0.0
|
||||
# Unpriced only when it is an Anthropic model we forgot to price; a
|
||||
# non-Anthropic model with no rate is intentionally free.
|
||||
return CostResult(
|
||||
cost_usd=0.0, unpriced=is_anthropic, is_anthropic=is_anthropic
|
||||
)
|
||||
|
||||
inp_price, out_price, cr_price, cw_price = best_prices
|
||||
|
||||
@@ -145,4 +194,6 @@ def calculate_cost(
|
||||
+ tokens_cache_read * cr_price / _MILLION
|
||||
+ tokens_cache_write * cw_price / _MILLION
|
||||
)
|
||||
return round(cost, 8)
|
||||
return CostResult(
|
||||
cost_usd=round(cost, 8), unpriced=False, is_anthropic=is_anthropic
|
||||
)
|
||||
|
||||
@@ -175,6 +175,18 @@ class Settings(BaseSettings):
|
||||
description="Base URL for Ollama native API (embeddings, model mgmt)",
|
||||
)
|
||||
|
||||
routing_strict: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Fail-closed model routing: when an agent has a configured "
|
||||
"model_assignment whose provider is disabled (or otherwise "
|
||||
"unroutable), raise instead of silently downgrading to the "
|
||||
"legacy Anthropic path. Off (default) => graceful degradation "
|
||||
"with a warning, so a misconfigured provider never stalls a "
|
||||
"spawn; the warning still surfaces the bypass so it isn't silent."
|
||||
),
|
||||
)
|
||||
|
||||
# ==========================================================================
|
||||
# Agent runtime toolchain matching (default-off)
|
||||
# ==========================================================================
|
||||
@@ -476,6 +488,17 @@ class Settings(BaseSettings):
|
||||
ge=1,
|
||||
description="Max self-heal fix tasks the loop may originate in one cycle.",
|
||||
)
|
||||
self_heal_notify_dedupe_seconds: int = Field(
|
||||
default=7200,
|
||||
ge=60,
|
||||
description=(
|
||||
"Per-fingerprint CEO-notify dedupe window. A regression that stays"
|
||||
" red across cycles notifies the CEO once per episode, not every"
|
||||
" tick; the dedupe key expires after this window so a regression"
|
||||
" that clears and later recurs notifies again. Fail-open: a Redis"
|
||||
" outage in the check still lets the notify through."
|
||||
),
|
||||
)
|
||||
|
||||
# Multi-repo CI-watch — generalizes the single-repo self-heal CI loop to any
|
||||
# opted-in project (per-project `ci_watch_enabled` column). Default-off;
|
||||
@@ -893,6 +916,21 @@ class Settings(BaseSettings):
|
||||
"override via ROBOCO_GROK_IDLE_KILL_SECONDS"
|
||||
),
|
||||
)
|
||||
# A non-GROK agent (Claude / Ollama-cloud / etc.) that gets stuck in a
|
||||
# non-verb loop — alive but firing no gateway verb, so its heartbeat never
|
||||
# advances and the reaper's live-container skip shields its claim forever
|
||||
# (#73). Past this MUCH longer window the orchestrator kills + evicts the
|
||||
# container so the reaper releases the task. Deliberately far beyond any
|
||||
# legit edit/test cycle (a working agent fires gateway verbs every few
|
||||
# minutes) so only a truly-stuck run trips it, never a slow-but-working one.
|
||||
claude_stuck_kill_seconds: int = Field(
|
||||
default=3600,
|
||||
ge=600,
|
||||
description=(
|
||||
"Stuck-in-non-verb-loop kill threshold for non-GROK agents "
|
||||
"(seconds); override via ROBOCO_CLAUDE_STUCK_KILL_SECONDS"
|
||||
),
|
||||
)
|
||||
# Budget kill-switch parity for GROK. Claude Code's per-agent token-budget
|
||||
# hook fires against the SDK :9000 server; the grok CLI exposes no live usage
|
||||
# hook, so the orchestrator enforces the cap by reading each live GROK
|
||||
|
||||
@@ -27,6 +27,20 @@ if TYPE_CHECKING:
|
||||
# plain ``.ts``. Map each dialect tag to the language family it belongs to.
|
||||
_DIALECT_OF: dict[str, str] = {"tsx": "typescript"}
|
||||
|
||||
# Languages the validator actually reports (runner._LANGUAGE_BY_SUFFIX). A
|
||||
# custom rule scoped to anything else is a typo — the rule would silently
|
||||
# never fire, so ``unrecognized_rule_languages`` surfaces it fail-loud (#129).
|
||||
# The tsx -> typescript dialect relation above is intentional (#32 BY-DESIGN).
|
||||
_KNOWN_LANGUAGES: frozenset[str] = frozenset({"python", "typescript", "tsx"})
|
||||
|
||||
|
||||
def unrecognized_rule_languages(rule: CustomRule) -> list[str]:
|
||||
"""Language tags on ``rule`` the validator never reports — likely a typo.
|
||||
|
||||
An unscoped rule (empty ``languages``) applies to everything and is valid.
|
||||
"""
|
||||
return [lang for lang in rule.languages if lang not in _KNOWN_LANGUAGES]
|
||||
|
||||
|
||||
def _rule_applies(rule_languages: list[str], file_language: str) -> bool:
|
||||
"""Whether a custom rule scoped to ``rule_languages`` applies to a file of
|
||||
|
||||
@@ -47,6 +47,8 @@ _DB_METHODS = frozenset(
|
||||
"execute",
|
||||
"scalar",
|
||||
"scalars",
|
||||
"stream",
|
||||
"stream_scalars",
|
||||
"add",
|
||||
"add_all",
|
||||
"merge",
|
||||
|
||||
@@ -11,7 +11,8 @@ from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from . import classify_python, classify_ts
|
||||
from .custom import check_custom
|
||||
from .custom import check_custom, unrecognized_rule_languages
|
||||
from .findings import Finding
|
||||
from .grammars import GrammarUnavailable
|
||||
from .hygiene import check_hygiene
|
||||
from .modularity import check_modularity
|
||||
@@ -20,11 +21,15 @@ from .placement import check_placement
|
||||
if TYPE_CHECKING:
|
||||
from roboco.foundation.policy.conventions.models import ConventionsStandard
|
||||
|
||||
from .findings import Finding
|
||||
from .placement import Definition
|
||||
|
||||
_LANGUAGE_BY_SUFFIX = {".py": "python", ".ts": "typescript", ".tsx": "tsx"}
|
||||
|
||||
# Surfaced on the conventions file itself — a custom rule scoped to a language
|
||||
# the validator never reports (a typo) would silently never fire (#129).
|
||||
_LANGUAGE_SCOPE_RULE = "custom_language_scope"
|
||||
_LANGUAGE_SCOPE_FILE = ".roboco/conventions.yml"
|
||||
|
||||
|
||||
class ValidatorCouldNotRun(RuntimeError):
|
||||
"""Raised when the validator cannot analyze a file (fail-loud signal)."""
|
||||
@@ -39,7 +44,7 @@ def run(
|
||||
) -> list[Finding]:
|
||||
"""Check ``files`` (repo-relative) under ``root`` against ``standard``."""
|
||||
root_path = Path(root)
|
||||
findings: list[Finding] = []
|
||||
findings: list[Finding] = list(_language_scope_findings(standard))
|
||||
for rel in files:
|
||||
language = _LANGUAGE_BY_SUFFIX.get(Path(rel).suffix)
|
||||
if language is not None:
|
||||
@@ -47,6 +52,29 @@ def run(
|
||||
return _apply_waivers(findings, standard)
|
||||
|
||||
|
||||
def _language_scope_findings(standard: ConventionsStandard) -> list[Finding]:
|
||||
"""Warn-once per custom rule scoped to a language the validator never reports."""
|
||||
out: list[Finding] = []
|
||||
for rule in standard.custom:
|
||||
for tag in unrecognized_rule_languages(rule):
|
||||
out.append(
|
||||
Finding(
|
||||
file=_LANGUAGE_SCOPE_FILE,
|
||||
line=0,
|
||||
kind=None,
|
||||
rule=_LANGUAGE_SCOPE_RULE,
|
||||
level="warn",
|
||||
message=(
|
||||
f"custom rule '{rule.id}' scopes to unknown language"
|
||||
f" '{tag}' — the validator reports {{python, typescript,"
|
||||
" tsx}}; the rule will never fire"
|
||||
),
|
||||
fix_hint="fix the language tag in .roboco/conventions.yml",
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def _check_file(
|
||||
root: Path, rel: str, language: str, standard: ConventionsStandard
|
||||
) -> list[Finding]:
|
||||
|
||||
@@ -759,6 +759,14 @@ class PlaybookTable(Base):
|
||||
approved_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
# Distinct from approval: who retired it (archive/reject) and when. Stamping
|
||||
# the archiver into approved_by overwrote the approval provenance.
|
||||
archived_by: Mapped[PyUUID | None] = mapped_column(
|
||||
UUID(as_uuid=True), nullable=True
|
||||
)
|
||||
archived_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
|
||||
|
||||
class SecretaryDirectiveTable(Base):
|
||||
|
||||
@@ -26,6 +26,11 @@ logger = structlog.get_logger()
|
||||
# Type for event handlers
|
||||
EventHandler = Callable[[Event], Coroutine[Any, Any, None]]
|
||||
|
||||
# #19: TTL on the per-(event.id, handler) "already processed" marker. Long
|
||||
# enough to span a replay window, short enough that the keyspace can't grow
|
||||
# unbounded across the fleet.
|
||||
_PROCESSED_KEY_TTL = 3600
|
||||
|
||||
|
||||
class StreamEventBus:
|
||||
"""
|
||||
@@ -299,7 +304,16 @@ class StreamEventBus:
|
||||
return all_succeeded
|
||||
|
||||
async def _dispatch_event(self, event: Event) -> bool:
|
||||
"""Run all handlers for an event; return True if all succeeded."""
|
||||
"""Run all handlers for an event; return True if all succeeded.
|
||||
|
||||
#19: ``recover_pending`` re-delivers a message whose ACK was blocked by a
|
||||
sibling handler's failure. Without an idempotency guard, a handler that
|
||||
already SUCCEEDED for that event would re-run on the replay (duplicate
|
||||
side effects). Each (event.id, handler) is marked processed via SET-NX;
|
||||
a handler whose key already exists is skipped, and a handler that raised
|
||||
clears its key so a replay re-runs it. The guard is best-effort — no
|
||||
redis (or a guard error) fails open and just runs the handler.
|
||||
"""
|
||||
handlers = self._handlers.get(event.type, [])
|
||||
if not handlers:
|
||||
return True
|
||||
@@ -309,10 +323,42 @@ class StreamEventBus:
|
||||
event_type=event.type.value,
|
||||
handler_count=len(handlers),
|
||||
)
|
||||
tasks = [handler(event) for handler in handlers]
|
||||
tasks = [self._run_handler_guarded(event, handler) for handler in handlers]
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
return self._check_handler_results(event, handlers, results)
|
||||
|
||||
async def _run_handler_guarded(self, event: Event, handler: EventHandler) -> None:
|
||||
"""Run one handler under the (event.id, handler) idempotency guard.
|
||||
|
||||
Fail-open: with no redis (or a guard error) the handler runs unguarded
|
||||
rather than being skipped — a dedup-infra outage never silently drops
|
||||
an event.
|
||||
"""
|
||||
if self._redis is None:
|
||||
await handler(event)
|
||||
return
|
||||
name = getattr(handler, "__name__", f"h{id(handler)}")
|
||||
key = f"bus:processed:{event.id}:{name}"
|
||||
try:
|
||||
acquired = await self._redis.set(key, "1", nx=True, ex=_PROCESSED_KEY_TTL)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Bus idempotency guard error; fail-open",
|
||||
error=e.__class__.__name__,
|
||||
)
|
||||
await handler(event)
|
||||
return
|
||||
if not acquired:
|
||||
logger.debug("Skipping already-processed handler on replay", key=key)
|
||||
return
|
||||
try:
|
||||
await handler(event)
|
||||
except Exception:
|
||||
# Clear the marker so a replay re-runs this handler — not a phantom success.
|
||||
with contextlib.suppress(Exception):
|
||||
await self._redis.delete(key)
|
||||
raise
|
||||
|
||||
async def _handle_message(
|
||||
self,
|
||||
stream: str,
|
||||
|
||||
@@ -303,14 +303,65 @@ def role_for_slug_or_none(slug: str) -> Role | None:
|
||||
|
||||
Returns ``None`` for an unknown/stale slug instead of raising ``KeyError``,
|
||||
so a stale assignee or notification-target slug can't crash the whole
|
||||
dispatcher tick. Callers that only need to check "is this a human-only
|
||||
role?" treat ``None`` as "not human-only" (``None in (CEO, ...)`` is False)
|
||||
and proceed without raising.
|
||||
dispatcher tick. Prefer :func:`is_human_only_role` /
|
||||
:func:`is_spawnable_agent_slug` over re-deriving the human-only set at each
|
||||
call site — a bare ``role in (CEO, ...)`` test treats a stale ``None`` slug
|
||||
as "not human" and proceeds to spawn it (the #49 layered-guard hole).
|
||||
"""
|
||||
row = AGENTS.get(slug)
|
||||
return row.role if row is not None else None
|
||||
|
||||
|
||||
_HUMAN_ONLY_ROLES: frozenset[Role] = frozenset(
|
||||
{Role.CEO, Role.PROMPTER, Role.SECRETARY}
|
||||
)
|
||||
|
||||
|
||||
def is_human_only_role(role: Role | None) -> bool:
|
||||
"""True for the CEO / prompter / secretary — the human-driven roles.
|
||||
|
||||
These are never containers. ``None`` (an unresolvable/stale slug) is False:
|
||||
a stale slug is not itself
|
||||
a human role, it is simply unknown. Use :func:`is_spawnable_agent_slug` at
|
||||
spawn-skip sites where a stale slug must also be skipped.
|
||||
"""
|
||||
return role in _HUMAN_ONLY_ROLES
|
||||
|
||||
|
||||
def is_spawnable_agent_slug(slug: str) -> bool:
|
||||
"""True only when ``slug`` resolves to a known non-human agent role.
|
||||
|
||||
A dispatcher may spawn a slug only when this is True. An unresolvable
|
||||
(stale/renamed) slug returns False so the dispatcher skips instead of
|
||||
launching a doomed — or formerly-human — container (#49). Live human-only
|
||||
roles are False; their dedicated spawn paths do not route through the
|
||||
dispatcher. Release/reaper paths that *recover* a stale-slug claim should
|
||||
NOT use this (they want to act on the stale slug, not spawn it).
|
||||
"""
|
||||
role = role_for_slug_or_none(slug)
|
||||
if role is None:
|
||||
return False
|
||||
return role not in _HUMAN_ONLY_ROLES
|
||||
|
||||
|
||||
def role_for_uuid_or_none(agent_uuid: object | None) -> Role | None:
|
||||
"""Resolve a seeded agent's ``UUID`` to its ``Role`` (None if unknown).
|
||||
|
||||
The create path carries ``assigned_to`` as a UUID, not a slug, so the
|
||||
role-vs-task_type guard needs a UUID→role lookup. Built once over the
|
||||
seeded ``AGENTS`` table; a non-seeded / stale UUID returns None (the guard
|
||||
treats None as "not a PM" and proceeds — the team-based ``main_pm`` check
|
||||
still holds the line for the main-pm team).
|
||||
"""
|
||||
if agent_uuid is None:
|
||||
return None
|
||||
key = str(agent_uuid)
|
||||
for row in AGENTS.values():
|
||||
if str(row.uuid) == key:
|
||||
return row.role
|
||||
return None
|
||||
|
||||
|
||||
def team_for_slug(slug: str) -> Team:
|
||||
"""Shorthand for `agent_for_slug(slug).team`."""
|
||||
return agent_for_slug(slug).team
|
||||
|
||||
@@ -15,6 +15,7 @@ between sites. Inputs are typed ``object | None`` because callers pass either OR
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from roboco.foundation.identity import Role
|
||||
from roboco.models.base import TaskType, Team
|
||||
|
||||
|
||||
@@ -120,6 +121,32 @@ def main_pm_cannot_own_code(*, team: object | None, task_type: object | None) ->
|
||||
the reassign/escalation diversion, and the claim guard. Accepts either ORM
|
||||
enum members or their ``.value`` strings (callers pass both shapes).
|
||||
"""
|
||||
team_value = str(getattr(team, "value", team))
|
||||
type_value = str(getattr(task_type, "value", task_type))
|
||||
team_value = str(getattr(team, "value", team)).lower()
|
||||
type_value = str(getattr(task_type, "value", task_type)).lower()
|
||||
return team_value == Team.MAIN_PM.value and type_value == TaskType.CODE.value
|
||||
|
||||
|
||||
def pm_cannot_own_code(
|
||||
*, role: object | None, task_type: object | None, is_issue_resolution: bool = False
|
||||
) -> bool:
|
||||
"""True when a PM role is being handed a ``code`` task it may not own.
|
||||
|
||||
Both PM roles (``cell_pm`` and ``main_pm``) *coordinate* — planning,
|
||||
delegation, unblocking — they have no verb to write code, so a ``code`` task
|
||||
assigned/claimed by a PM is the same structural mismatch
|
||||
``main_pm_cannot_own_code`` guards at creation time. The one deliberate
|
||||
exception is **issue resolution**: a PM may take a ``code`` task that is in
|
||||
``needs_revision`` to act on the review/QA issues directly. ``is_issue_resolution``
|
||||
is the caller's assertion of that state (a claim/reassign of a needs_revision
|
||||
code task); a freshly created or delegated subtask is never in that state, so
|
||||
its caller passes ``False``. Non-PM roles and non-code task types always pass.
|
||||
|
||||
Accepts ORM enum members or their ``.value`` strings (callers pass both).
|
||||
"""
|
||||
role_value = str(getattr(role, "value", role)).lower()
|
||||
if role_value not in (Role.CELL_PM.value, Role.MAIN_PM.value):
|
||||
return False
|
||||
type_value = str(getattr(task_type, "value", task_type)).lower()
|
||||
if type_value != TaskType.CODE.value:
|
||||
return False
|
||||
return not is_issue_resolution
|
||||
|
||||
@@ -115,6 +115,19 @@ class ChannelSpec:
|
||||
|
||||
# -- Helper sets (DRY across multiple channels) -------------------------------
|
||||
|
||||
# Cell-member roles subject to a channel's ``team_scope``. Cross-cell roles
|
||||
# (MAIN_PM, AUDITOR, CEO, board) are NOT filtered by team_scope. The single
|
||||
# source of truth — ``agents_config`` and ``seeds/initial_data`` import this
|
||||
# instead of carrying their own copies (which drifted).
|
||||
TEAM_SCOPED_ROLES: frozenset[Role] = frozenset(
|
||||
{
|
||||
Role.DEVELOPER,
|
||||
Role.QA,
|
||||
Role.DOCUMENTER,
|
||||
Role.CELL_PM,
|
||||
}
|
||||
)
|
||||
|
||||
# Roles present in every cell channel: cell members + main-pm.
|
||||
_CELL_READ: frozenset[Role] = frozenset(
|
||||
{
|
||||
|
||||
@@ -350,6 +350,19 @@ _STATUS_TRANSITIONS: tuple[StatusTransition, ...] = (
|
||||
"ceo_reject",
|
||||
frozenset({Role.CEO}),
|
||||
),
|
||||
# A branchless coordination root (product integration root / MegaTask
|
||||
# umbrella) has no developer to revise it, so CEO rejection routes it to
|
||||
# PENDING for the Main PM to re-plan — not NEEDS_REVISION (dev-claim-only,
|
||||
# would deadlock the root). The runtime applies this via the audited
|
||||
# privileged override (admin_set_status); the edge is in the spec so future
|
||||
# admin-override tightening can't wedge the path, and so the transition is
|
||||
# acknowledged as CEO-legal rather than an unsanctioned out-of-band write.
|
||||
StatusTransition(
|
||||
Status.AWAITING_CEO_APPROVAL,
|
||||
Status.PENDING,
|
||||
"ceo_reject_to_pool",
|
||||
frozenset({Role.CEO}),
|
||||
),
|
||||
# Direct PM submission for non-dev tasks
|
||||
StatusTransition(
|
||||
Status.IN_PROGRESS,
|
||||
@@ -639,6 +652,16 @@ _ATOMIC_ACTIONS: dict[str, ActionSpec] = {
|
||||
self_review_block=False,
|
||||
needs_team_match=False,
|
||||
),
|
||||
"ceo_reject_to_pool": ActionSpec(
|
||||
name="ceo_reject_to_pool",
|
||||
allowed_roles=frozenset({Role.CEO}),
|
||||
source_statuses=frozenset({Status.AWAITING_CEO_APPROVAL}),
|
||||
target_status=Status.PENDING,
|
||||
allowed_task_types=None,
|
||||
preconditions=(),
|
||||
self_review_block=False,
|
||||
needs_team_match=False,
|
||||
),
|
||||
"cancel": ActionSpec(
|
||||
name="cancel",
|
||||
allowed_roles=frozenset(_PM_ROLES | {Role.CEO}),
|
||||
@@ -991,6 +1014,72 @@ PRECONDITION_PR_OPEN_STATE = Precondition(
|
||||
)
|
||||
|
||||
|
||||
# The states from which a dev may re-sync their branch onto its base. A rebase
|
||||
# is meaningful only while the dev actively works the task and the branch is
|
||||
# live — once the task is paused / blocked / handed to QA-doc-PM-CEO review /
|
||||
# terminal, the dev is no longer the actor and a rebase runs against a branch
|
||||
# whose task is no longer theirs to move. ``sync_branch`` composes=() (no
|
||||
# composed action supplies a source-status gate), so without this precondition
|
||||
# the spec gate accepted a COMPLETED/CANCELLED/PAUSED/BLOCKED task and the
|
||||
# choreographer handler rebased a dead/parked branch (#50).
|
||||
SYNC_BRANCH_STATES: frozenset[Status] = frozenset(
|
||||
{
|
||||
Status.CLAIMED,
|
||||
Status.IN_PROGRESS,
|
||||
Status.VERIFYING,
|
||||
Status.NEEDS_REVISION,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _p_sync_branch_state(task: Any, _agent: Any, _ctx: Any) -> bool:
|
||||
"""True iff the task is in a sync_branch-eligible active dev state."""
|
||||
status = getattr(task, "status", None)
|
||||
value = status.value if isinstance(status, Status) else str(status)
|
||||
return value in {s.value for s in SYNC_BRANCH_STATES}
|
||||
|
||||
|
||||
PRECONDITION_SYNC_BRANCH_STATE = Precondition(
|
||||
key="sync_branch_state",
|
||||
check=_p_sync_branch_state,
|
||||
remediate=(
|
||||
"sync_branch is only valid while you actively work the task "
|
||||
"(claimed / in_progress / verifying / needs_revision); a paused, "
|
||||
"blocked, reviewing, or terminal task's branch is not yours to move"
|
||||
),
|
||||
missing_token="sync_branch_state",
|
||||
rejection_kind="invalid_state",
|
||||
)
|
||||
|
||||
|
||||
# submit_root's prose asserts "a Main-PM root is planning-typed, never code".
|
||||
# The creation path (``main_pm_cannot_own_code``) already blocks a code-typed
|
||||
# Main-PM root at TaskService.create / intake / approve_and_start, so such a
|
||||
# root is unreachable in production — but the spec gate must back the claim too
|
||||
# (defense in depth: a future creation-path change can't quietly make submit_root
|
||||
# accept a code root). Scoped to submit_root only, NOT the shared
|
||||
# ``submit_for_review`` action (which cell_pm+code submit_up legitimately uses).
|
||||
def _p_root_not_code(task: Any, _agent: Any, _ctx: Any) -> bool:
|
||||
tt = getattr(task, "task_type", None)
|
||||
value = (
|
||||
tt.value if isinstance(tt, TaskType) else (str(tt) if tt is not None else None)
|
||||
)
|
||||
return value != TaskType.CODE.value
|
||||
|
||||
|
||||
PRECONDITION_ROOT_NOT_CODE = Precondition(
|
||||
key="root_not_code",
|
||||
check=_p_root_not_code,
|
||||
remediate=(
|
||||
"a Main-PM root is planning-typed, never code; the root assembles the "
|
||||
"cells' merged work — a code-typed root belongs to a developer, not the "
|
||||
"Main PM. Reassign the code work to a dev and keep the root planning-typed"
|
||||
),
|
||||
missing_token="root_not_code",
|
||||
rejection_kind="invalid_state",
|
||||
)
|
||||
|
||||
|
||||
_INTENT_VERBS: dict[str, IntentSpec] = {
|
||||
# Phase 1: developer verbs
|
||||
"give_me_work": IntentSpec(
|
||||
@@ -1095,7 +1184,10 @@ _INTENT_VERBS: dict[str, IntentSpec] = {
|
||||
" by hand, commit, then sync_branch again."
|
||||
),
|
||||
composes=(), # git-only verb — no DB transition; the handler runs the git op
|
||||
extra_preconditions=(PRECONDITION_OWNERSHIP,),
|
||||
extra_preconditions=(
|
||||
PRECONDITION_OWNERSHIP,
|
||||
PRECONDITION_SYNC_BRANCH_STATE,
|
||||
),
|
||||
side_effects=(),
|
||||
next_hint=_next_hint_synced,
|
||||
),
|
||||
@@ -1362,7 +1454,7 @@ _INTENT_VERBS: dict[str, IntentSpec] = {
|
||||
" task_type-keyed — a Main-PM root is planning-typed, never code."
|
||||
),
|
||||
composes=("submit_for_review",),
|
||||
extra_preconditions=(),
|
||||
extra_preconditions=(PRECONDITION_ROOT_NOT_CODE,),
|
||||
# The root→master PR must exist before the reviewer can review it —
|
||||
# opened here (parent=master, is_root_pr=True), mirroring submit_up's
|
||||
# pre-create of the cell→root PR.
|
||||
@@ -1514,6 +1606,15 @@ def _check_claim_rules_narrow(role: Role, task: Any) -> Decision | None:
|
||||
"""
|
||||
status = Status(getattr(task, "status", ""))
|
||||
role_claim_statuses = CLAIM_RULES.get(role, frozenset())
|
||||
# PM/code invariant is NOT enforced here. A PM's only claim verb is
|
||||
# i_will_plan, and a cell/main PM planning a code-typed PARENT (to decompose
|
||||
# + delegate the code) is legitimate (bug-1: scoping pm_cannot_execute_code
|
||||
# to i_will_plan deadlocked the slice). Execution is already blocked at the
|
||||
# intent level — i_will_work_on is _DEV_ROLES only, so a PM cannot execute
|
||||
# code via the execution verb. The create/delegate guards
|
||||
# (pm_cannot_own_code) block a PM from being ASSIGNED a fresh code task; the
|
||||
# needs_revision carve-out (a PM resolving review issues directly) is
|
||||
# naturally allowed because PMs claim NEEDS_REVISION.
|
||||
if status in role_claim_statuses:
|
||||
return None
|
||||
allowed_list = sorted(s.value for s in role_claim_statuses)
|
||||
|
||||
@@ -26,6 +26,7 @@ import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import threading
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
@@ -38,6 +39,14 @@ if TYPE_CHECKING:
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
# Process-wide serialisation for the single-use refresh-token grant (#94). The
|
||||
# grok refresh token is invalidated the instant xAI issues the new one, so two
|
||||
# concurrent ``refresh_if_stale`` calls that both POST the grant would have the
|
||||
# second use the now-dead old token and burn the credential. The lock +
|
||||
# re-load-and-recheck inside it makes the loser find the winner's refreshed
|
||||
# token and return ``fresh`` instead of re-rotating.
|
||||
_refresh_lock = threading.Lock()
|
||||
|
||||
# xAI OIDC issuer; the token endpoint is ``<issuer>/oauth2/token`` (verified via
|
||||
# the issuer's ``.well-known/openid-configuration``). A per-entry ``oidc_issuer``
|
||||
# overrides it.
|
||||
@@ -271,6 +280,30 @@ def _do_refresh(
|
||||
return "refreshed"
|
||||
|
||||
|
||||
def _recheck_or_refresh(
|
||||
auth_path: Path,
|
||||
now: datetime,
|
||||
skew_seconds: int,
|
||||
post: Callable[[str, dict[str, str]], dict[str, Any]] | None,
|
||||
) -> str:
|
||||
"""Re-load + re-check staleness, then refresh — the locked body of refresh_if_stale.
|
||||
|
||||
Run inside ``_refresh_lock`` so a concurrent caller that waited on the lock
|
||||
re-reads the bundle a single-writer just refreshed and returns ``fresh``
|
||||
instead of re-POSTing the single-use refresh grant (#94).
|
||||
"""
|
||||
bundle = _load(auth_path)
|
||||
if bundle is None:
|
||||
return "missing"
|
||||
entry = _credential_entry(bundle)
|
||||
if entry is None:
|
||||
return "no_refresh_token"
|
||||
entry_key, creds = entry
|
||||
if not _is_stale(creds, now, skew_seconds):
|
||||
return "fresh"
|
||||
return _do_refresh(auth_path, bundle, entry_key, now, post or _post_token)
|
||||
|
||||
|
||||
def refresh_if_stale(
|
||||
auth_path: Path,
|
||||
*,
|
||||
@@ -291,11 +324,16 @@ def refresh_if_stale(
|
||||
entry = _credential_entry(bundle)
|
||||
if entry is None:
|
||||
return "no_refresh_token"
|
||||
entry_key, creds = entry
|
||||
_, creds = entry
|
||||
now = now or datetime.now(UTC)
|
||||
if not _is_stale(creds, now, skew_seconds):
|
||||
return "fresh"
|
||||
return _do_refresh(auth_path, bundle, entry_key, now, post or _post_token)
|
||||
# Single-use refresh token: hold the lock and re-load + re-check inside it
|
||||
# so a concurrent caller that waited on the lock finds the refreshed token
|
||||
# and returns ``fresh`` instead of re-POSTing the grant (which would use the
|
||||
# already-invalidated old token and burn the credential — #94).
|
||||
with _refresh_lock:
|
||||
return _recheck_or_refresh(auth_path, now, skew_seconds, post)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
+89
-23
@@ -61,23 +61,52 @@ _CIRCUIT_REJECTION_KINDS: frozenset[str] = frozenset(
|
||||
# actually records the attempt. Kinds not in `_CIRCUIT_REJECTION_KINDS` are
|
||||
# never forwarded (the SDK ignores unknown kinds anyway).
|
||||
#
|
||||
# Classification is substring-based so the many custom RobocoError codes
|
||||
# (A2A_ACCESS_DENIED, NO_WRITE_ACCESS, TASK_NOT_OWNED, …) land on the right
|
||||
# counted kind without an exhaustive literal map. The NOT_FOUND family returns
|
||||
# None — parity with the string-error contract that a `not_found` rejection
|
||||
# does NOT count (retrying a missing resource won't help until state changes).
|
||||
# The exact map is authoritative for the codes the handlers actually emit, so
|
||||
# a known code is never misclassified by an accidental substring — e.g.
|
||||
# AUTHENTICATION_REQUIRED carries no AUTHORIZED/DENIED/PERMISSION substring and
|
||||
# under a substring-only rule dropped to ``invalid_state`` instead of
|
||||
# ``not_authorized`` (#161). The NOT_FOUND family maps to None — parity with
|
||||
# the string-error contract that a `not_found` rejection does NOT count
|
||||
# (retrying a missing resource won't help until state changes). Unknown codes
|
||||
# fall through to a substring branch so a new RobocoError code still lands on a
|
||||
# counted kind without a map update. Mirrors flow_server.
|
||||
_DICT_ERROR_CODE_MAP: dict[str, str | None] = {
|
||||
"AUTHENTICATION_REQUIRED": "not_authorized",
|
||||
"CHANNEL_ACCESS_DENIED": "not_authorized",
|
||||
"JOURNAL_ACCESS_DENIED": "not_authorized",
|
||||
"PERMISSION_DENIED": "not_authorized",
|
||||
"INVALID_INPUT": "incomplete_input",
|
||||
"VALIDATION_ERROR": "incomplete_input",
|
||||
"NOT_FOUND": None,
|
||||
"INVALID_STATE": "invalid_state",
|
||||
"TASK_LIFECYCLE_ERROR": "invalid_state",
|
||||
"TASK_OWNERSHIP_ERROR": "invalid_state",
|
||||
"SERVICE_ERROR": "invalid_state",
|
||||
"SESSION_CLOSED": "invalid_state",
|
||||
"FETCH_FAILED": "invalid_state",
|
||||
"LIST_FAILED": "invalid_state",
|
||||
"READ_FAILED": "invalid_state",
|
||||
"SEARCH_FAILED": "invalid_state",
|
||||
"WRITE_FAILED": "invalid_state",
|
||||
}
|
||||
|
||||
|
||||
def _classify_dict_error_code(code: str) -> str | None:
|
||||
upper = code.upper()
|
||||
if upper in _DICT_ERROR_CODE_MAP:
|
||||
return _DICT_ERROR_CODE_MAP[upper]
|
||||
# Unknown code — substring fallback for forward-compat with new codes.
|
||||
if "NOT_FOUND" in upper:
|
||||
return None
|
||||
if (
|
||||
"DENIED" in upper
|
||||
or "AUTHORIZED" in upper
|
||||
or "AUTH" in upper
|
||||
or "FORBIDDEN" in upper
|
||||
or "PERMISSION" in upper
|
||||
):
|
||||
return "not_authorized"
|
||||
if upper == "INVALID_INPUT" or "VALIDATION" in upper:
|
||||
if "VALIDATION" in upper:
|
||||
return "incomplete_input"
|
||||
return "invalid_state"
|
||||
|
||||
@@ -156,15 +185,15 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
|
||||
headers=_build_headers(),
|
||||
json=body,
|
||||
)
|
||||
# A 404 here means a manifest-registered content tool has no matching
|
||||
# route on the orchestrator: every /api/v1/do/* route returns 200 with
|
||||
# an Envelope (including not_found rejections), so FastAPI's default
|
||||
# 404 body (no ``error`` field) is always a missing route, never a legit
|
||||
# Envelope. Synthesize an ``invalid_state`` Envelope so the breaker
|
||||
# counts it (via ``_classify_rejection``) and the agent gets a
|
||||
# remediation hint instead of a raw ``detail`` body. A 404 carrying a
|
||||
# real Envelope (``error`` field) is surfaced as-is. Mirrors
|
||||
# flow_server._post.
|
||||
# A 404 here usually means a manifest-registered content tool has no
|
||||
# matching route on the orchestrator: every /api/v1/do/* route returns
|
||||
# 200 with an Envelope (including not_found rejections), so FastAPI's
|
||||
# default 404 body (``{"detail": "Not Found"}``) is a missing route —
|
||||
# synthesize an ``invalid_state`` Envelope so the breaker counts it and
|
||||
# the agent gets a wiring-gap remediation hint. A 404 carrying a real
|
||||
# Envelope (``error`` field) is surfaced as-is; a 404 with a
|
||||
# *descriptive* ``detail`` (not the bare default) is a real resource
|
||||
# not_found, surfaced as ``not_found`` (#61). Mirrors flow_server._post.
|
||||
if response.status_code == _MISSING_ROUTE_STATUS:
|
||||
try:
|
||||
body_404 = response.json()
|
||||
@@ -172,6 +201,25 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
|
||||
body_404 = None
|
||||
if isinstance(body_404, dict) and "error" in body_404:
|
||||
payload_404: dict[str, Any] = body_404
|
||||
elif (
|
||||
isinstance(body_404, dict)
|
||||
and isinstance(body_404.get("detail"), str)
|
||||
and body_404["detail"] != "Not Found"
|
||||
):
|
||||
# A real HTTP 404 with a descriptive detail — a resource
|
||||
# not_found, not a missing route (#61).
|
||||
verb = _verb_from_path(path)
|
||||
payload_404 = {
|
||||
"error": "not_found",
|
||||
"message": body_404["detail"],
|
||||
"remediate": (
|
||||
f"the {verb} call targeted a resource that does not"
|
||||
f" exist (HTTP 404: {body_404['detail']}). Re-fetch"
|
||||
f" state and retry on a current id; do not retry the"
|
||||
f" same id."
|
||||
),
|
||||
"missing": [],
|
||||
}
|
||||
else:
|
||||
verb = _verb_from_path(path)
|
||||
payload_404 = {
|
||||
@@ -272,7 +320,12 @@ def _record_and_check_circuit(
|
||||
return payload
|
||||
|
||||
if status.get("open") and isinstance(status.get("circuit_envelope"), dict):
|
||||
circuit_env: dict[str, Any] = status["circuit_envelope"]
|
||||
# Copy so the SDK's envelope dict is not mutated in place. Nest the
|
||||
# original fixable rejection as ``inner`` so its kind/message/remediate
|
||||
# survive the substitution — the circuit_open envelope only says the
|
||||
# breaker tripped, not WHY the verb failed (#60). Mirrors flow_server.
|
||||
circuit_env: dict[str, Any] = dict(status["circuit_envelope"])
|
||||
circuit_env["inner"] = payload
|
||||
log.info(
|
||||
"do_server: circuit_open substituted for rejection",
|
||||
verb=verb,
|
||||
@@ -764,6 +817,8 @@ def _register_tools() -> list[str]:
|
||||
missing rather than silently exposing the full do-tool set (which
|
||||
includes ``commit`` — the role-gate would reject it server-side, but
|
||||
the agent shouldn't see it on its tool palette in the first place).
|
||||
``ROBOCO_ALLOW_FULL_TOOLSET`` is a dev/test escape hatch that registers
|
||||
the full tool set instead of raising (#162); default-off.
|
||||
|
||||
Returns the list of tool names actually registered.
|
||||
"""
|
||||
@@ -772,13 +827,24 @@ def _register_tools() -> list[str]:
|
||||
manifest_path = os.environ.get(
|
||||
"ROBOCO_TOOL_MANIFEST_PATH", "/app/tool-manifest.json"
|
||||
)
|
||||
msg = (
|
||||
f"do_server: manifest unavailable at {manifest_path};"
|
||||
f" refusing to register all-tools fallback for role"
|
||||
f" {AGENT_ROLE!r}. Check the orchestrator manifest mount."
|
||||
)
|
||||
log.error("do_server: manifest missing", role=AGENT_ROLE, path=manifest_path)
|
||||
raise RuntimeError(msg)
|
||||
if os.environ.get("ROBOCO_ALLOW_FULL_TOOLSET"):
|
||||
log.warning(
|
||||
"do_server: manifest missing — ROBOCO_ALLOW_FULL_TOOLSET set,"
|
||||
" registering the full tool set (dev/test only)",
|
||||
role=AGENT_ROLE,
|
||||
path=manifest_path,
|
||||
)
|
||||
allowed = list(_TOOLS.keys())
|
||||
else:
|
||||
msg = (
|
||||
f"do_server: manifest unavailable at {manifest_path};"
|
||||
f" refusing to register all-tools fallback for role"
|
||||
f" {AGENT_ROLE!r}. Check the orchestrator manifest mount."
|
||||
)
|
||||
log.error(
|
||||
"do_server: manifest missing", role=AGENT_ROLE, path=manifest_path
|
||||
)
|
||||
raise RuntimeError(msg)
|
||||
unknown = [verb for verb in allowed if verb not in _TOOLS]
|
||||
if unknown:
|
||||
log.warning(
|
||||
|
||||
@@ -60,10 +60,23 @@ async def _handle_write(
|
||||
"The file has been indexed in RAG and linked to the task."
|
||||
)
|
||||
|
||||
# Surface the repo-commit outcome (#34): a 'failed' commit means the doc
|
||||
# saved to /app/docs but did NOT reach the project repo — the cell PM must
|
||||
# be told so the PR does not ship without the docs.
|
||||
doc_ref = result.get("doc_ref") or {}
|
||||
commit_status = doc_ref.get("commit_status") if isinstance(doc_ref, dict) else None
|
||||
if commit_status == "failed":
|
||||
guidance += (
|
||||
" WARNING: the doc could NOT be committed to the project repo — tell"
|
||||
" the cell PM so the docs are not missing from the PR."
|
||||
)
|
||||
elif commit_status == "skipped":
|
||||
guidance += " (saved to /app/docs only — no task branch to commit onto yet)."
|
||||
|
||||
return {
|
||||
"status": status,
|
||||
"path": path,
|
||||
"doc_ref": result.get("doc_ref"),
|
||||
"doc_ref": doc_ref,
|
||||
"is_update": is_update,
|
||||
"guidance": guidance,
|
||||
}
|
||||
|
||||
+98
-27
@@ -77,23 +77,52 @@ _CIRCUIT_REJECTION_KINDS: frozenset[str] = frozenset(
|
||||
# actually records the attempt. Kinds not in `_CIRCUIT_REJECTION_KINDS` are
|
||||
# never forwarded (the SDK ignores unknown kinds anyway).
|
||||
#
|
||||
# Classification is substring-based so the many custom RobocoError codes
|
||||
# (A2A_ACCESS_DENIED, NO_WRITE_ACCESS, TASK_NOT_OWNED, …) land on the right
|
||||
# counted kind without an exhaustive literal map. The NOT_FOUND family returns
|
||||
# None — parity with the string-error contract that a `not_found` rejection
|
||||
# does NOT count (retrying a missing resource won't help until state changes).
|
||||
# The exact map is authoritative for the codes the handlers actually emit, so
|
||||
# a known code is never misclassified by an accidental substring — e.g.
|
||||
# AUTHENTICATION_REQUIRED carries no AUTHORIZED/DENIED/PERMISSION substring and
|
||||
# under a substring-only rule dropped to ``invalid_state`` instead of
|
||||
# ``not_authorized`` (#161). The NOT_FOUND family maps to None — parity with
|
||||
# the string-error contract that a `not_found` rejection does NOT count
|
||||
# (retrying a missing resource won't help until state changes). Unknown codes
|
||||
# fall through to a substring branch so a new RobocoError code still lands on a
|
||||
# counted kind without a map update.
|
||||
_DICT_ERROR_CODE_MAP: dict[str, str | None] = {
|
||||
"AUTHENTICATION_REQUIRED": "not_authorized",
|
||||
"CHANNEL_ACCESS_DENIED": "not_authorized",
|
||||
"JOURNAL_ACCESS_DENIED": "not_authorized",
|
||||
"PERMISSION_DENIED": "not_authorized",
|
||||
"INVALID_INPUT": "incomplete_input",
|
||||
"VALIDATION_ERROR": "incomplete_input",
|
||||
"NOT_FOUND": None,
|
||||
"INVALID_STATE": "invalid_state",
|
||||
"TASK_LIFECYCLE_ERROR": "invalid_state",
|
||||
"TASK_OWNERSHIP_ERROR": "invalid_state",
|
||||
"SERVICE_ERROR": "invalid_state",
|
||||
"SESSION_CLOSED": "invalid_state",
|
||||
"FETCH_FAILED": "invalid_state",
|
||||
"LIST_FAILED": "invalid_state",
|
||||
"READ_FAILED": "invalid_state",
|
||||
"SEARCH_FAILED": "invalid_state",
|
||||
"WRITE_FAILED": "invalid_state",
|
||||
}
|
||||
|
||||
|
||||
def _classify_dict_error_code(code: str) -> str | None:
|
||||
upper = code.upper()
|
||||
if upper in _DICT_ERROR_CODE_MAP:
|
||||
return _DICT_ERROR_CODE_MAP[upper]
|
||||
# Unknown code — substring fallback for forward-compat with new codes.
|
||||
if "NOT_FOUND" in upper:
|
||||
return None
|
||||
if (
|
||||
"DENIED" in upper
|
||||
or "AUTHORIZED" in upper
|
||||
or "AUTH" in upper
|
||||
or "FORBIDDEN" in upper
|
||||
or "PERMISSION" in upper
|
||||
):
|
||||
return "not_authorized"
|
||||
if upper == "INVALID_INPUT" or "VALIDATION" in upper:
|
||||
if "VALIDATION" in upper:
|
||||
return "incomplete_input"
|
||||
return "invalid_state"
|
||||
|
||||
@@ -176,15 +205,18 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
|
||||
headers=_build_headers(),
|
||||
json=body,
|
||||
)
|
||||
# A 404 here means a manifest-registered verb has no matching route on
|
||||
# the orchestrator: every gateway route returns 200 with an Envelope
|
||||
# (including not_found rejections), so FastAPI's default 404 body (no
|
||||
# ``error`` field) is always a missing route, never a legit Envelope.
|
||||
# Synthesize an ``invalid_state`` Envelope so the breaker counts it
|
||||
# (via ``_classify_rejection``) and the agent gets a remediation hint
|
||||
# instead of a raw ``detail`` body. A 404 carrying a real Envelope (an
|
||||
# ``error`` field — e.g. a proxy re-status a 200 rejection to 404) is
|
||||
# surfaced as-is.
|
||||
# A 404 here usually means a manifest-registered verb has no matching
|
||||
# route on the orchestrator: every gateway route returns 200 with an
|
||||
# Envelope (including not_found rejections), so FastAPI's default 404
|
||||
# body (``{"detail": "Not Found"}``, no ``error`` field) is a missing
|
||||
# route — synthesize an ``invalid_state`` Envelope so the breaker
|
||||
# counts it and the agent gets a wiring-gap remediation hint. Two other
|
||||
# 404 shapes are surfaced more accurately (#61): a 404 carrying a real
|
||||
# Envelope (an ``error`` field — e.g. a proxy re-status a 200 rejection
|
||||
# to 404) is surfaced as-is; a 404 carrying a *descriptive* ``detail``
|
||||
# (not the bare default) is a real resource not_found, not a missing
|
||||
# route — surface it as ``not_found`` so the agent re-fetches state
|
||||
# instead of being told the route is unwired.
|
||||
if response.status_code == _MISSING_ROUTE_STATUS:
|
||||
try:
|
||||
body_404 = response.json()
|
||||
@@ -194,6 +226,25 @@ def _post(path: str, body: dict[str, Any]) -> dict[str, Any]:
|
||||
# Real Envelope rejection surfaced under a 404 status —
|
||||
# surface it as-is so the agent sees the real kind/remediate.
|
||||
payload_404: dict[str, Any] = body_404
|
||||
elif (
|
||||
isinstance(body_404, dict)
|
||||
and isinstance(body_404.get("detail"), str)
|
||||
and body_404["detail"] != "Not Found"
|
||||
):
|
||||
# A real HTTP 404 with a descriptive detail — a resource
|
||||
# not_found, not a missing route (#61).
|
||||
verb = _verb_from_path(path)
|
||||
payload_404 = {
|
||||
"error": "not_found",
|
||||
"message": body_404["detail"],
|
||||
"remediate": (
|
||||
f"the {verb} call targeted a resource that does not"
|
||||
f" exist (HTTP 404: {body_404['detail']}). Re-fetch"
|
||||
f" the task state (give_me_work / resume) and retry on"
|
||||
f" a current id; do not retry the same id."
|
||||
),
|
||||
"missing": [],
|
||||
}
|
||||
else:
|
||||
verb = _verb_from_path(path)
|
||||
payload_404 = {
|
||||
@@ -300,7 +351,13 @@ def _record_and_check_circuit(
|
||||
return payload
|
||||
|
||||
if status.get("open") and isinstance(status.get("circuit_envelope"), dict):
|
||||
circuit_env: dict[str, Any] = status["circuit_envelope"]
|
||||
# Copy so the SDK's envelope dict is not mutated in place (the SDK may
|
||||
# reuse it across calls). Nest the original fixable rejection as
|
||||
# ``inner`` so its kind/message/remediate survive the substitution —
|
||||
# the circuit_open envelope only says the breaker tripped, not WHY the
|
||||
# verb failed, and the agent still needs the underlying hint (#60).
|
||||
circuit_env: dict[str, Any] = dict(status["circuit_envelope"])
|
||||
circuit_env["inner"] = payload
|
||||
log.info(
|
||||
"flow_server: circuit_open substituted for rejection",
|
||||
verb=verb,
|
||||
@@ -811,7 +868,10 @@ def _register_tools() -> list[str]:
|
||||
The manifest is the role-authoritative tool list. Falling back to
|
||||
all-verbs registration (the previous behaviour) caused PMs to see
|
||||
developer/QA verbs and call them at wrong URLs (404s). We now refuse
|
||||
to start without the manifest.
|
||||
to start without the manifest — unless ``ROBOCO_ALLOW_FULL_TOOLSET`` is
|
||||
set, a dev/test escape hatch that registers the full tool set instead of
|
||||
raising so the server modules import without a hand-written manifest
|
||||
(#162). Default-off so production behaviour is unchanged.
|
||||
|
||||
Returns the list of verb names actually registered.
|
||||
"""
|
||||
@@ -820,16 +880,27 @@ def _register_tools() -> list[str]:
|
||||
manifest_path = os.environ.get(
|
||||
"ROBOCO_TOOL_MANIFEST_PATH", "/app/tool-manifest.json"
|
||||
)
|
||||
msg = (
|
||||
f"flow_server: manifest unavailable at {manifest_path};"
|
||||
f" refusing to register all-verbs fallback (would let"
|
||||
f" {AGENT_ROLE!r} call off-role verbs at wrong URLs)."
|
||||
f" Check that the orchestrator wrote the manifest to its"
|
||||
f" /app/manifests/ directory and that the agent container"
|
||||
f" has the bind-mount."
|
||||
)
|
||||
log.error("flow_server: manifest missing", role=AGENT_ROLE, path=manifest_path)
|
||||
raise RuntimeError(msg)
|
||||
if os.environ.get("ROBOCO_ALLOW_FULL_TOOLSET"):
|
||||
log.warning(
|
||||
"flow_server: manifest missing — ROBOCO_ALLOW_FULL_TOOLSET set,"
|
||||
" registering the full tool set (dev/test only)",
|
||||
role=AGENT_ROLE,
|
||||
path=manifest_path,
|
||||
)
|
||||
allowed = list(_TOOLS.keys())
|
||||
else:
|
||||
msg = (
|
||||
f"flow_server: manifest unavailable at {manifest_path};"
|
||||
f" refusing to register all-verbs fallback (would let"
|
||||
f" {AGENT_ROLE!r} call off-role verbs at wrong URLs)."
|
||||
f" Check that the orchestrator wrote the manifest to its"
|
||||
f" /app/manifests/ directory and that the agent container"
|
||||
f" has the bind-mount."
|
||||
)
|
||||
log.error(
|
||||
"flow_server: manifest missing", role=AGENT_ROLE, path=manifest_path
|
||||
)
|
||||
raise RuntimeError(msg)
|
||||
public = [_INTENT_TO_PUBLIC.get(verb, verb) for verb in allowed]
|
||||
unknown = [verb for verb in public if verb not in _TOOLS]
|
||||
if unknown:
|
||||
|
||||
+44
-10
@@ -123,6 +123,39 @@ async def propose_draft(draft: dict[str, Any]) -> str:
|
||||
return f"Could not submit the draft to the panel: {detail}"
|
||||
|
||||
|
||||
def _draft_title(d: dict[str, Any]) -> str | None:
|
||||
"""A draft is well-formed when it carries a string ``title`` OR ``name``."""
|
||||
t = d.get("title")
|
||||
if isinstance(t, str):
|
||||
return t
|
||||
n = d.get("name")
|
||||
if isinstance(n, str):
|
||||
return n
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_batch_drafts(
|
||||
raw: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
"""Filter + normalize MegaTask drafts for the panel relay (#163).
|
||||
|
||||
A draft without a string ``title``/``name`` is dropped and counted. A
|
||||
``name``-only draft is normalized onto a copy as ``title`` (the caller's
|
||||
dict is never mutated). Returns ``(well_formed, dropped_count)``.
|
||||
"""
|
||||
well_formed: list[dict[str, Any]] = []
|
||||
for d in raw:
|
||||
draft_title = _draft_title(d)
|
||||
if draft_title is None:
|
||||
continue
|
||||
if isinstance(d.get("title"), str):
|
||||
well_formed.append(d)
|
||||
else:
|
||||
# name-only — normalize onto a copy so the relay sees a ``title``.
|
||||
well_formed.append({**d, "title": draft_title})
|
||||
return well_formed, len(raw) - len(well_formed)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def propose_batch(drafts: list[dict[str, Any]], title: str = "") -> str:
|
||||
"""Submit a MegaTask — SEVERAL task drafts at once — for the human to confirm.
|
||||
@@ -144,25 +177,26 @@ async def propose_batch(drafts: list[dict[str, Any]], title: str = "") -> str:
|
||||
"No live session id (ROBOCO_PROMPTER_SESSION_ID) — cannot surface the "
|
||||
"MegaTask."
|
||||
)
|
||||
# Drop malformed entries (a draft needs a string title) and refuse to POST an
|
||||
# empty batch — otherwise it would silently vanish on the panel side, telling
|
||||
# the agent it succeeded while nothing appears.
|
||||
raw = drafts or []
|
||||
well_formed = [
|
||||
d for d in raw if isinstance(d, dict) and isinstance(d.get("title"), str)
|
||||
]
|
||||
# Malformed entries are dropped and counted; an empty batch is refused
|
||||
# rather than silently vanishing on the panel side (#163).
|
||||
well_formed, dropped = _normalize_batch_drafts(drafts or [])
|
||||
if not well_formed:
|
||||
return (
|
||||
"That MegaTask had no well-formed task drafts — give each a title and "
|
||||
"project_id and call propose_batch again."
|
||||
"That MegaTask had no well-formed task drafts — give each a title"
|
||||
" (or name) and a project_id and call propose_batch again."
|
||||
)
|
||||
payload = {
|
||||
"drafts": well_formed,
|
||||
"title": title,
|
||||
"dropped": len(raw) - len(well_formed),
|
||||
"dropped": dropped,
|
||||
}
|
||||
result = await post_batch(session_id, payload)
|
||||
if result.get("ok"):
|
||||
if dropped:
|
||||
return (
|
||||
"MegaTask submitted — the human can review it in the panel."
|
||||
f" {dropped} draft(s) were dropped for missing a title/name."
|
||||
)
|
||||
return "MegaTask submitted — the human can review it in the panel."
|
||||
detail = result.get("detail") or result.get("error") or "unknown error"
|
||||
return f"Could not submit the MegaTask to the panel: {detail}"
|
||||
|
||||
@@ -33,6 +33,8 @@ class Playbook(RobocoBase):
|
||||
approved_by: UUID | None = None
|
||||
created_at: datetime | None = None
|
||||
approved_at: datetime | None = None
|
||||
archived_by: UUID | None = None
|
||||
archived_at: datetime | None = None
|
||||
|
||||
|
||||
class PlaybookCreate(RobocoBase):
|
||||
|
||||
@@ -58,6 +58,14 @@ class DocRef(RobocoBase):
|
||||
updated_at: str | None = Field(
|
||||
default=None, description="ISO timestamp of last update"
|
||||
)
|
||||
commit_status: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Whether the doc reached the project repo: 'committed', 'skipped' "
|
||||
"(no task branch — saved to /app/docs only), or 'failed' (git "
|
||||
"hiccup — tell the cell PM)"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class ProgressUpdate(RobocoBase):
|
||||
|
||||
+375
-56
@@ -46,7 +46,12 @@ from roboco.agents_config import (
|
||||
)
|
||||
from roboco.config import settings
|
||||
from roboco.foundation import identity as _foundation
|
||||
from roboco.foundation.identity import CELL_TEAMS, Role, role_for_slug_or_none
|
||||
from roboco.foundation.identity import (
|
||||
CELL_TEAMS,
|
||||
is_human_only_role,
|
||||
is_spawnable_agent_slug,
|
||||
role_for_slug_or_none,
|
||||
)
|
||||
from roboco.foundation.policy.agent_loop import DEFAULT_BUDGET as _AGENT_LOOP_BUDGET
|
||||
from roboco.foundation.policy.batch import is_branchless_coordination
|
||||
from roboco.models import AgentRole, Team
|
||||
@@ -150,6 +155,11 @@ def _system_api_headers() -> dict[str, str]:
|
||||
|
||||
# Consecutive failed recovery probes before the CEO is notified once per episode.
|
||||
_CEO_NOTIFY_THRESHOLD = 10
|
||||
# Consecutive strategy-engine cycle failures before the CEO is notified once
|
||||
# per failure episode (#193). Mirrors _CEO_NOTIFY_THRESHOLD so a persistently
|
||||
# failing assess() (bad DB / goals row) surfaces instead of silently producing
|
||||
# nothing every tick.
|
||||
_STRATEGY_FAIL_CEO_NOTIFY_THRESHOLD = 10
|
||||
# Persistent-probe-failure escape hatch (F094): if the recovery probe keeps
|
||||
# failing past this threshold, the probe endpoint itself is the problem (a
|
||||
# misconfigured URL, a removed API key, a network partition to the probe host)
|
||||
@@ -359,6 +369,19 @@ class _IntakeRunSpec:
|
||||
model: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class _StrategyLoopState:
|
||||
"""Consecutive-failure tracking for ``_strategy_engine_loop`` (#193).
|
||||
|
||||
``failures`` counts consecutive cycle exceptions; ``notified`` gates the
|
||||
one-CEO-alert-per-episode. Both reset on the first success so a fresh
|
||||
failure episode re-notifies.
|
||||
"""
|
||||
|
||||
failures: int = 0
|
||||
notified: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class _SecretaryRunSpec:
|
||||
"""Inputs for ``_build_secretary_run_cmd`` (mirrors ``_IntakeRunSpec``).
|
||||
@@ -382,6 +405,23 @@ class _SecretaryRunSpec:
|
||||
model: str = ""
|
||||
|
||||
|
||||
# Roles that always work a concrete task — a spawn row with ``task_id IS NULL``
|
||||
# for one of these is an unattributed-cost bug (the usage rollup can't tie the
|
||||
# spend to a task). Intake (prompter), secretary, auditor, and PMs legitimately
|
||||
# spawn taskless, so they are NOT flagged (#11).
|
||||
_TASKLESS_SPAWN_SUSPECT_ROLES = frozenset({"developer", "qa", "documenter"})
|
||||
|
||||
|
||||
def is_unattributed_delivery_spawn(role: str, task_id: str | None) -> bool:
|
||||
"""True when a delivery-role spawn carries no ``task_id`` (#11).
|
||||
|
||||
The role string comes from ``get_agent_role`` (lowercase); the comparison is
|
||||
case-insensitive for safety. Used by ``_record_spawn_session`` to warn on
|
||||
unattributed usage without noise from the intentional taskless roles.
|
||||
"""
|
||||
return task_id is None and role.lower() in _TASKLESS_SPAWN_SUSPECT_ROLES
|
||||
|
||||
|
||||
def _read_project_slug(task: dict[str, Any]) -> str | None:
|
||||
"""Extract project slug from a task payload shape-tolerantly."""
|
||||
slug = task.get("project_slug")
|
||||
@@ -843,6 +883,10 @@ class AgentOrchestrator:
|
||||
# a broken-but-alive agent (see _maybe_recover_broken_gateway).
|
||||
self._gateway_broken_since: dict[str, datetime] = {}
|
||||
self._waiting_records: dict[str, WaitingRecord] = {}
|
||||
# #71: a resumed agent's WaitingRecord is torn down only once liveness is
|
||||
# confirmed (not on a bare launch) — a container that launches then dies
|
||||
# immediately would otherwise strand its task until the reaper's TTL.
|
||||
self._resume_confirm_delay: float = 30.0
|
||||
self._health_task: asyncio.Task | None = None
|
||||
self._dispatcher_task: asyncio.Task | None = None
|
||||
self._sweeper_task: asyncio.Task | None = None
|
||||
@@ -954,6 +998,10 @@ class AgentOrchestrator:
|
||||
# killed + evicted so the reaper can release its task; see
|
||||
# _maybe_kill_wedged_grok.
|
||||
self._grok_idle_kill_ttl: int = settings.grok_idle_kill_seconds
|
||||
# #73: a non-GROK agent stuck in a non-verb loop (alive, no heartbeat
|
||||
# advance) is killed past this longer window so the reaper can release
|
||||
# its task; see _maybe_kill_stuck_claude.
|
||||
self._claude_stuck_kill_ttl: int = settings.claude_stuck_kill_seconds
|
||||
# Cost ceiling (USD) before a live GROK container is killed — the budget
|
||||
# kill-switch parity (the grok CLI exposes no live usage hook). 0 disables.
|
||||
# See _enforce_grok_cost_budget.
|
||||
@@ -1054,6 +1102,35 @@ class AgentOrchestrator:
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await asyncio.gather(*pending, return_exceptions=True)
|
||||
|
||||
async def _flush_respawn_tracker(self) -> None:
|
||||
"""Persist the full in-memory PM-respawn snapshot before the process exits.
|
||||
|
||||
Fire-and-forget persists (``_schedule_respawn_persist``) are bounded by
|
||||
the shutdown drain deadline; one cancelled by that deadline leaves the
|
||||
durable count lagging the in-memory counter, so the next restart
|
||||
re-burns the strike threshold against a still-wedged task — the exact
|
||||
re-burn the durable counter exists to stop (#74). Called from ``stop()``
|
||||
AFTER the bounded drain so it is the last writer (no further gate
|
||||
mutations fire once the agents and loops are down) and unbounded (a
|
||||
short upsert must not be dropped on the shutdown path). Best-effort: a
|
||||
row that fails to persist is logged and skipped, never crashing shutdown
|
||||
— the in-memory value is gone either way once the process exits.
|
||||
"""
|
||||
if not self._pm_respawn_tracker:
|
||||
return
|
||||
for agent_slug, task_id in list(self._pm_respawn_tracker.keys()):
|
||||
record = self._pm_respawn_tracker.get((agent_slug, task_id))
|
||||
if record is None:
|
||||
continue
|
||||
try:
|
||||
await self._persist_respawn_record(agent_slug, task_id, dict(record))
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"shutdown respawn-tracker flush failed for one row; continuing",
|
||||
agent_id=agent_slug,
|
||||
task_id=task_id,
|
||||
)
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop the orchestrator and all agents."""
|
||||
if getattr(self, "_stopped", False):
|
||||
@@ -1104,6 +1181,13 @@ class AgentOrchestrator:
|
||||
# stuck task can't hang shutdown — it is cancelled past the deadline.
|
||||
await self._drain_bg_tasks()
|
||||
|
||||
# #74: flush the authoritative in-memory respawn snapshot AFTER the
|
||||
# bounded drain so a deadline-cancelled persist can't leave the durable
|
||||
# count lagging the in-memory counter (and re-burning the strike
|
||||
# threshold on the next restart). Unbounded — a short upsert must not be
|
||||
# dropped on the shutdown path.
|
||||
await self._flush_respawn_tracker()
|
||||
|
||||
self._stopped = True
|
||||
logger.info("Orchestrator stopped")
|
||||
|
||||
@@ -2010,17 +2094,23 @@ class AgentOrchestrator:
|
||||
|
||||
The container launches with ``-w`` at the worktree; a pruned/evicted
|
||||
worktree (reaper, disk pressure, manual cleanup while the agent was
|
||||
down) would start the agent in a missing directory. Idempotent —
|
||||
``ensure_worktree_for_resume`` is a no-op when the worktree is present
|
||||
and re-adds it (no ``-b``) from the surviving branch ref when pruned.
|
||||
No-op for branchless / no-task spawns (no worktree).
|
||||
down) — or a vanished clone root (disk loss, a redeploy that wiped
|
||||
``/data/workspaces``) — would start the agent in a missing directory.
|
||||
Idempotent: a present worktree is a no-op; a pruned worktree is re-added
|
||||
from the surviving branch ref; a missing clone is re-cloned and the
|
||||
branch ref recovered from origin (``create_branch`` pushes at claim
|
||||
time) so the pushed work survives. No-op for branchless / no-task spawns.
|
||||
|
||||
A fatal git-state failure (``WorkspaceError`` — the branch ref is gone,
|
||||
so the worktree can't be re-added) releases the claim and aborts the
|
||||
spawn so the next claim rebuilds the worktree via ``create_branch``
|
||||
rather than launching the container at a missing ``-w`` path. A
|
||||
transient failure (DB/other) aborts without releasing — the next tick
|
||||
retries the same claim.
|
||||
The reaper-style claim release preserves ownership + ``branch_name``, so
|
||||
a re-dispatch is a RESUME, not a fresh claim — ``create_branch`` never
|
||||
re-runs to re-clone. Without the clone self-heal a vanished clone_root
|
||||
fatal-looped every tick (``git -C <missing>`` -> release -> re-dispatch
|
||||
into the same missing clone). A fatal git-state failure
|
||||
(``WorkspaceError`` — the clone won't re-clone, the token is missing,
|
||||
or the branch ref is unrecoverable) releases the claim and aborts so
|
||||
the next dispatch retries the rebuild, never launching the container at
|
||||
a missing ``-w``. A transient failure (DB/other) aborts without
|
||||
releasing — the next tick retries the same claim.
|
||||
"""
|
||||
if not (git_context and git_context.task_short_id and git_context.branch_name):
|
||||
return
|
||||
@@ -2035,15 +2125,24 @@ class AgentOrchestrator:
|
||||
|
||||
try:
|
||||
async with get_db_context() as db:
|
||||
await WorkspaceService(db).ensure_worktree_for_resume(
|
||||
clone_root, worktree, git_context.branch_name
|
||||
ws = WorkspaceService(db)
|
||||
# Heal a vanished/unhealthy clone first. The reaper-style claim
|
||||
# release preserves ownership + branch_name, so a re-dispatch is
|
||||
# a RESUME, not a fresh claim — create_branch never re-runs to
|
||||
# re-clone, and ensure_worktree_for_resume would ``git -C`` a
|
||||
# missing directory and fatal-loop every tick. Skipped on a
|
||||
# healthy clone (no new fetch overhead on the common resume).
|
||||
if not WorkspaceService._is_workspace_healthy(clone_root):
|
||||
await ws.ensure_workspace(project_slug, agent_id)
|
||||
await ws.ensure_worktree_self_heal(
|
||||
clone_root, worktree, git_context.branch_name, project_slug
|
||||
)
|
||||
except WorkspaceError as e:
|
||||
# Fatal git state: the branch ref is gone, so the worktree cannot be
|
||||
# re-added here. Release the claim so the next claim rebuilds the
|
||||
# worktree via create_branch, and abort before docker run -w lands
|
||||
# on a missing path. The release is best-effort (suppressed) so a
|
||||
# release failure never masks the fatal error.
|
||||
# Fatal git state (clone won't re-clone, token missing, branch ref
|
||||
# unrecoverable): release the claim so the next dispatch can retry
|
||||
# the rebuild, and abort before docker run -w lands on a missing
|
||||
# path. The release is best-effort (suppressed) so a release
|
||||
# failure never masks the fatal error.
|
||||
logger.error(
|
||||
"worktree ensure failed (fatal); releasing claim for rebuild",
|
||||
agent_id=agent_id,
|
||||
@@ -2184,7 +2283,7 @@ class AgentOrchestrator:
|
||||
# through spawn_agent (see the _spawn_intake_container note at the top
|
||||
# of this file).
|
||||
_role = role_for_slug_or_none(agent_id)
|
||||
if _role in (Role.CEO, Role.PROMPTER, Role.SECRETARY):
|
||||
if is_human_only_role(_role):
|
||||
logger.error(
|
||||
"spawn_agent refused for human-only role — dispatchers must never"
|
||||
" spawn the CEO / prompter / secretary; these are human-driven",
|
||||
@@ -4823,6 +4922,17 @@ class AgentOrchestrator:
|
||||
team = get_agent_team(agent_slug) or "backend"
|
||||
role = get_agent_role(agent_slug) or "developer"
|
||||
|
||||
# A delivery-role spawn with no task_id is unattributed usage (#11) —
|
||||
# the rollup can't tie the spend to a task. Intake/secretary/PM spawns
|
||||
# legitimately carry no task and are not flagged.
|
||||
if is_unattributed_delivery_spawn(role, task_id):
|
||||
logger.warning(
|
||||
"Spawn session has no task_id for a delivery role — "
|
||||
"unattributed usage",
|
||||
agent_slug=agent_slug,
|
||||
role=role,
|
||||
)
|
||||
|
||||
session_id = _uuid4()
|
||||
session_factory = get_session_factory()
|
||||
async with session_factory() as db:
|
||||
@@ -5621,6 +5731,11 @@ class AgentOrchestrator:
|
||||
if agent_id not in self._waiting_records:
|
||||
return None
|
||||
|
||||
# #71: a lingering record (a prior resume whose liveness confirmation
|
||||
# hasn't torn it down yet) must not double-spawn an already-active agent.
|
||||
if self._is_agent_active(agent_id):
|
||||
return None
|
||||
|
||||
record = self._waiting_records[agent_id]
|
||||
|
||||
# Generate resume prompt
|
||||
@@ -5660,10 +5775,45 @@ class AgentOrchestrator:
|
||||
# Spawn bailed without launching (provider re-parked). Keep the record
|
||||
# so the probe-resume loop re-attempts on the next clear.
|
||||
return instance
|
||||
if record.waiting_for == "rate_limit_lifted":
|
||||
# #71: don't tear down the record on a bare launch — a container that
|
||||
# launches then dies immediately would orphan the task until the
|
||||
# reaper's TTL. Keep the record past the launch and confirm liveness
|
||||
# in the background; if the container dies the probe-resume orphan
|
||||
# fallback re-resumes within a tick instead of waiting the full TTL.
|
||||
self._schedule_bg(self._confirm_resume_liveness(agent_id))
|
||||
return instance
|
||||
del self._waiting_records[agent_id]
|
||||
await self._delete_waiting_record(agent_id)
|
||||
return instance
|
||||
|
||||
async def _confirm_resume_liveness(self, agent_id: str) -> None:
|
||||
"""Tear down a resumed agent's WaitingRecord once it is confirmed alive.
|
||||
|
||||
A container that launches then dies immediately must not strand its task
|
||||
until the reaper's TTL: the record is kept past the launch (``resolve_wait``
|
||||
schedules this) and deleted only once the agent is still active past a
|
||||
short confirmation window. If the container died, the record survives so
|
||||
the probe-resume orphan fallback re-resumes on the next tick (#71). The
|
||||
confirmation reads ``_is_agent_active`` — the same signal the spawn gate
|
||||
trusts — so a container the health loop has marked dead keeps its record.
|
||||
Best-effort: a delete error is swallowed (the in-memory record is gone
|
||||
either way once the process exits, and the orphan fallback is in-memory).
|
||||
"""
|
||||
if agent_id not in self._waiting_records:
|
||||
return
|
||||
await asyncio.sleep(self._resume_confirm_delay)
|
||||
if not self._is_agent_active(agent_id):
|
||||
return # container died — keep the record for the orphan fallback
|
||||
del self._waiting_records[agent_id]
|
||||
try:
|
||||
await self._delete_waiting_record(agent_id)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"resume-liveness confirm failed to delete the durable record",
|
||||
agent_id=agent_id,
|
||||
)
|
||||
|
||||
def _generate_resume_prompt(
|
||||
self,
|
||||
record: WaitingRecord,
|
||||
@@ -6362,23 +6512,67 @@ Start by:
|
||||
|
||||
Dormant by default — returns immediately unless ``strategy_engine_enabled``
|
||||
is set, so it adds zero behaviour to a standard deployment. Notify-only;
|
||||
it never spends or builds.
|
||||
it never spends or builds. A persistently failing cycle surfaces to the
|
||||
CEO once per failure episode (#193) instead of silently logging forever.
|
||||
"""
|
||||
if not settings.strategy_engine_enabled:
|
||||
return
|
||||
from roboco.db import get_db_context
|
||||
from roboco.services.strategy_engine import get_strategy_engine
|
||||
|
||||
state = self._new_strategy_loop_state()
|
||||
interval = settings.strategy_engine_interval_seconds
|
||||
while self._running:
|
||||
try:
|
||||
await asyncio.sleep(interval)
|
||||
async with get_db_context() as db:
|
||||
await get_strategy_engine(db).run_cycle()
|
||||
await self._strategy_engine_cycle(state)
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception:
|
||||
logger.exception("strategy engine cycle failed")
|
||||
|
||||
@staticmethod
|
||||
def _new_strategy_loop_state() -> _StrategyLoopState:
|
||||
return _StrategyLoopState()
|
||||
|
||||
async def _strategy_engine_cycle(self, state: _StrategyLoopState) -> None:
|
||||
"""Run one strategy-engine pass; track consecutive failures (#193).
|
||||
|
||||
On success the failure state resets (a fresh failure episode later
|
||||
re-notifies). On a non-cancel failure, count it and notify the CEO once
|
||||
per episode past ``_STRATEGY_FAIL_CEO_NOTIFY_THRESHOLD``.
|
||||
"""
|
||||
from roboco.db import get_db_context
|
||||
from roboco.services.strategy_engine import get_strategy_engine
|
||||
|
||||
try:
|
||||
async with get_db_context() as db:
|
||||
await get_strategy_engine(db).run_cycle()
|
||||
state.failures = 0
|
||||
state.notified = False
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
logger.exception("strategy engine cycle failed")
|
||||
state.failures += 1
|
||||
if (
|
||||
state.failures >= _STRATEGY_FAIL_CEO_NOTIFY_THRESHOLD
|
||||
and not state.notified
|
||||
):
|
||||
state.notified = True
|
||||
await self._notify_strategy_engine_failure(state.failures)
|
||||
|
||||
async def _notify_strategy_engine_failure(self, fail_count: int) -> None:
|
||||
"""Send one CEO alert that the strategy engine is persistently failing."""
|
||||
try:
|
||||
from roboco.services.notification import NotificationService
|
||||
|
||||
await NotificationService().send_ack_notification(
|
||||
from_agent="system",
|
||||
to_agent="ceo",
|
||||
body=(
|
||||
"[strategy engine] persistently failing: the last "
|
||||
f"{fail_count} cycles raised and produced no "
|
||||
"observations. Check the orchestrator logs."
|
||||
),
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("strategy engine failure-notify dropped")
|
||||
|
||||
async def _external_pr_poll_loop(self) -> None:
|
||||
"""Engine 3: discover inbound PRs and open review tasks.
|
||||
@@ -8544,6 +8738,47 @@ Start now: evidence(task_id="{task_id}")
|
||||
record = records.get(slug)
|
||||
return record is not None and record.waiting_for == "rate_limit_lifted"
|
||||
|
||||
async def _agent_holds_live_claim(self, slug: str) -> bool | None:
|
||||
"""Whether ``slug`` currently owns a non-terminal task.
|
||||
|
||||
Used by ``_readopt_running_agents`` to tell a still-useful running
|
||||
container (the agent is mid-task) from a zombie left over after a prior
|
||||
orchestrator released the claim: registering a zombie ACTIVE would block
|
||||
the spawn gate from re-dispatching that slug until the stale container is
|
||||
eventually noticed (#72). Returns True when the slug owns a non-terminal
|
||||
task, False when it owns nothing (zombie), and None on a lookup error
|
||||
(indeterminate — the caller falls back to today's register behaviour so a
|
||||
startup DB hiccup can't regress the cold-start double-spawn protection).
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from roboco.db.base import get_db_context
|
||||
from roboco.db.tables import TaskTable
|
||||
from roboco.models.base import TaskStatus
|
||||
|
||||
agent_uuid = AGENT_UUIDS.get(slug)
|
||||
if agent_uuid is None:
|
||||
return False # unknown slug owns nothing by definition
|
||||
try:
|
||||
async with get_db_context() as db:
|
||||
result = await db.execute(
|
||||
select(TaskTable.id)
|
||||
.where(
|
||||
TaskTable.assigned_to == agent_uuid,
|
||||
TaskTable.status.notin_(
|
||||
(TaskStatus.COMPLETED, TaskStatus.CANCELLED)
|
||||
),
|
||||
)
|
||||
.limit(1)
|
||||
)
|
||||
return result.first() is not None
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"readopt live-claim lookup failed; falling back to register",
|
||||
slug=slug,
|
||||
)
|
||||
return None
|
||||
|
||||
async def _readopt_running_agents(self) -> int:
|
||||
"""Re-adopt still-running agent containers into ``_instances`` at startup.
|
||||
|
||||
@@ -8554,11 +8789,14 @@ Start now: evidence(task_id="{task_id}")
|
||||
inactive and can double-spawn it onto work its forgotten-but-running
|
||||
container is already doing. Probe each known agent slug's container (the
|
||||
same ``docker inspect`` the reaper uses) and register a minimal ACTIVE
|
||||
instance for any that is running and not already tracked, so both the
|
||||
reaper's live-skip and the spawn gate see the live agent immediately.
|
||||
Inert when nothing is running (degrades to today's cold start) and
|
||||
best-effort: a probe error leaves that slot untracked (the reaper's own
|
||||
fallback still covers it). Returns the number re-adopted.
|
||||
instance for any that is running, not already tracked, AND still holds a
|
||||
live (non-terminal) claim — a running container whose claim a prior
|
||||
orchestrator already released is a zombie and is skipped so it can't
|
||||
block the spawn gate from re-dispatching that slug (#72). Inert when
|
||||
nothing is running (degrades to today's cold start) and best-effort: a
|
||||
probe or claim-lookup error leaves that slot untracked / falls back to
|
||||
registering (the reaper's own fallback still covers it). Returns the
|
||||
number re-adopted.
|
||||
"""
|
||||
readopted = 0
|
||||
for slug in AGENT_IMAGES:
|
||||
@@ -8582,6 +8820,18 @@ Start now: evidence(task_id="{task_id}")
|
||||
container_id = await self._resolve_container_id(f"roboco-agent-{slug}")
|
||||
except Exception:
|
||||
container_id = None
|
||||
# #72: a running container whose slug no longer holds a live claim is
|
||||
# a zombie from a prior orchestrator that released the claim — skip it
|
||||
# so it can't block re-dispatch of the slug. ``None`` (lookup error)
|
||||
# falls back to registering: a startup DB hiccup must not regress the
|
||||
# cold-start double-spawn protection this readopt exists to provide.
|
||||
holds_claim = await self._agent_holds_live_claim(slug)
|
||||
if holds_claim is False:
|
||||
logger.info(
|
||||
"readopt skipped a running zombie container (no live claim)",
|
||||
slug=slug,
|
||||
)
|
||||
continue
|
||||
self._instances[slug] = AgentInstance(
|
||||
agent_id=slug,
|
||||
state=AgentState.ACTIVE,
|
||||
@@ -8686,6 +8936,70 @@ Start now: evidence(task_id="{task_id}")
|
||||
)
|
||||
return True
|
||||
|
||||
def _stuck_claude_slug(
|
||||
self, task: Any, last_heartbeat: "datetime | None"
|
||||
) -> str | None:
|
||||
"""Slug of an ACTIVE non-GROK container holding ``task``, stuck past the TTL.
|
||||
|
||||
The reaper's live-container skip shields a quiet agent during a long
|
||||
edit/test cycle — correct for a working agent (it fires gateway verbs
|
||||
every few minutes, advancing its heartbeat). A non-GROK agent stuck in a
|
||||
non-verb loop is ACTIVE yet silent, so the skip would protect its claim
|
||||
forever (#73). Returns the slug only for a non-GROK ACTIVE instance whose
|
||||
heartbeat has been stale longer than ``claude_stuck_kill_seconds`` — a
|
||||
recent heartbeat, no owner, a GROK provider (handled by the wedged-grok
|
||||
path), or a non-ACTIVE instance all yield ``None``.
|
||||
"""
|
||||
from roboco.models.base import ModelProvider
|
||||
|
||||
cutoff = datetime.now(UTC) - timedelta(
|
||||
seconds=getattr(self, "_claude_stuck_kill_ttl", 3600)
|
||||
)
|
||||
if last_heartbeat is not None and last_heartbeat >= cutoff:
|
||||
return None
|
||||
owner = getattr(task, "assigned_to", None) or getattr(task, "claimed_by", None)
|
||||
if not owner:
|
||||
return None
|
||||
slug = self._resolve_agent_slug(str(owner))
|
||||
instance = (getattr(self, "_instances", None) or {}).get(slug)
|
||||
config = getattr(instance, "config", None)
|
||||
is_active_non_grok = (
|
||||
instance is not None
|
||||
and instance.state == AgentState.ACTIVE
|
||||
and config is not None
|
||||
and config.provider_type != ModelProvider.GROK.value
|
||||
)
|
||||
return slug if is_active_non_grok else None
|
||||
|
||||
async def _maybe_kill_stuck_claude(
|
||||
self, task: Any, last_heartbeat: "datetime | None"
|
||||
) -> bool:
|
||||
"""Kill + evict a stuck non-GROK container so the reaper frees its task.
|
||||
|
||||
On a kill the container is removed and dropped from ``_instances``.
|
||||
Returns True only when a container was actually killed; see
|
||||
:meth:`_stuck_claude_slug` for the eligibility rule (#73).
|
||||
"""
|
||||
slug = self._stuck_claude_slug(task, last_heartbeat)
|
||||
if slug is None:
|
||||
return False
|
||||
try:
|
||||
await self._remove_container(f"roboco-agent-{slug}")
|
||||
except Exception as exc:
|
||||
logger.error(
|
||||
"stuck-claude kill failed; will retry next tick",
|
||||
agent_id=slug,
|
||||
error=str(exc),
|
||||
)
|
||||
return False
|
||||
self._instances.pop(slug, None)
|
||||
logger.warning(
|
||||
"stuck non-grok container killed and evicted",
|
||||
agent_id=slug,
|
||||
task_id=str(getattr(task, "id", "")),
|
||||
)
|
||||
return True
|
||||
|
||||
async def _maybe_recover_broken_gateway(self, task: Any) -> bool:
|
||||
"""Kill + evict a live agent whose gateway is broken past the grace window.
|
||||
|
||||
@@ -8764,6 +9078,7 @@ Start now: evidence(task_id="{task_id}")
|
||||
return (
|
||||
live
|
||||
and not await self._maybe_kill_wedged_grok(t, ts)
|
||||
and not await self._maybe_kill_stuck_claude(t, ts)
|
||||
and not await self._maybe_recover_broken_gateway(t)
|
||||
)
|
||||
|
||||
@@ -10321,14 +10636,12 @@ Never `commit`, never write code, never run `git`. PMs coordinate.
|
||||
# Human-only roles (CEO / prompter / secretary) are never
|
||||
# containers — there is no reviewer agent to respawn. Leave
|
||||
# the task for the human (the CEO approves via the panel).
|
||||
# Mirrors the spawn_agent human-role guard; a skip here keeps
|
||||
# a mis-assigned human task from aborting this dispatcher's
|
||||
# whole tick (the chokepoint would otherwise raise).
|
||||
if role_for_slug_or_none(assigned_slug) in (
|
||||
Role.CEO,
|
||||
Role.PROMPTER,
|
||||
Role.SECRETARY,
|
||||
):
|
||||
# A stale/ex-human slug is also skipped: is_spawnable_agent_slug
|
||||
# is False for it, so a renamed secretary slug can't slip past
|
||||
# the layered guard to a doomed spawn (#49). Mirrors the
|
||||
# spawn_agent human-role guard; a skip here keeps a mis-assigned
|
||||
# human task from aborting this dispatcher's whole tick.
|
||||
if not is_spawnable_agent_slug(assigned_slug):
|
||||
continue
|
||||
if self._is_agent_active(assigned_slug):
|
||||
continue
|
||||
@@ -10499,13 +10812,11 @@ Never `commit`, never write code, never run `git`. PMs coordinate.
|
||||
# Human-only roles (CEO / prompter / secretary) are never containers —
|
||||
# there is no agent to respawn. Leave the task as-is for the human to
|
||||
# act on through the panel; do NOT release it to pending (that would
|
||||
# re-route a human-owned task to a PM). See spawn_agent's human-role
|
||||
# guard for the structural backstop.
|
||||
if role_for_slug_or_none(agent_slug) in (
|
||||
Role.CEO,
|
||||
Role.PROMPTER,
|
||||
Role.SECRETARY,
|
||||
):
|
||||
# re-route a human-owned task to a PM). A stale slug (None role) is NOT
|
||||
# skipped here — a stale-slug claim SHOULD be released to pending so a
|
||||
# real agent can reclaim it (recovery, not spawning). See spawn_agent's
|
||||
# human-role guard for the structural backstop.
|
||||
if is_human_only_role(role_for_slug_or_none(agent_slug)):
|
||||
return None
|
||||
# The assignee is running, and on THIS task — healthy.
|
||||
instance = self._instances.get(agent_slug)
|
||||
@@ -10924,15 +11235,23 @@ Never `commit`, never write code, never run `git`. PMs coordinate.
|
||||
# dispatched — the CEO is the human operator and intake/
|
||||
# secretary are human-driven chats with their own launch
|
||||
# paths. Spawning a container for one is a trust violation
|
||||
# (the system acting as the human CEO). The CEO being a
|
||||
# notification target (board-review handoff, escalation, etc.)
|
||||
# is expected; it is NOT a spawn signal. Skip — the
|
||||
# notification stays for the human to read in the panel.
|
||||
if role_for_slug_or_none(agent_slug) in (
|
||||
Role.CEO,
|
||||
Role.PROMPTER,
|
||||
Role.SECRETARY,
|
||||
):
|
||||
# (the system acting as the human CEO). A stale/ex-human slug
|
||||
# is skipped too (is_spawnable_agent_slug is False for it) so
|
||||
# a renamed secretary slug can't slip past to a spawn (#49).
|
||||
# The CEO being a notification target (board-review handoff,
|
||||
# escalation, etc.) is expected; it is NOT a spawn signal.
|
||||
# Skip — the notification stays for the human to read. #75:
|
||||
# surface the skip for a human-only target (vs a silent stale
|
||||
# slug) so an a2a expecting a human-side action (a CEO sign-off
|
||||
# relay) is visible in the dispatch log, not silently dropped.
|
||||
if is_human_only_role(role_for_slug_or_none(agent_slug)):
|
||||
logger.info(
|
||||
"a2a request targets a human-only role; left as a "
|
||||
"notification for the human (not spawned)",
|
||||
target_slug=agent_slug,
|
||||
)
|
||||
continue
|
||||
if not is_spawnable_agent_slug(agent_slug):
|
||||
continue
|
||||
|
||||
if self._is_agent_active(agent_slug):
|
||||
|
||||
@@ -173,18 +173,10 @@ DEFAULT_AGENTS: list[dict[str, Any]] = _build_default_agents()
|
||||
CEO_AGENT_ID = AGENT_UUIDS["ceo"]
|
||||
|
||||
# Cell-member roles subject to a channel's team_scope. Cross-cell roles
|
||||
# (MAIN_PM, AUDITOR, CEO, board) are NOT filtered by team_scope. Mirrors the
|
||||
# rule in agents_config._TEAM_SCOPED_ROLES; duplicated here to avoid a
|
||||
# circular import (agents_config already imports AGENT_UUIDS from this
|
||||
# module).
|
||||
_TEAM_SCOPED_ROLES: frozenset[_foundation.Role] = frozenset(
|
||||
{
|
||||
_foundation.Role.DEVELOPER,
|
||||
_foundation.Role.QA,
|
||||
_foundation.Role.DOCUMENTER,
|
||||
_foundation.Role.CELL_PM,
|
||||
}
|
||||
)
|
||||
# (MAIN_PM, AUDITOR, CEO, board) are NOT filtered by team_scope. Single-sourced
|
||||
# in ``foundation.policy.communications.TEAM_SCOPED_ROLES``; aliased here for
|
||||
# the seed helpers below.
|
||||
_TEAM_SCOPED_ROLES: frozenset[_foundation.Role] = _comms.TEAM_SCOPED_ROLES
|
||||
|
||||
|
||||
def _slugs_for_role_set(
|
||||
|
||||
@@ -9,9 +9,10 @@ project's delivery lifecycle and STOPS. Like self-heal it is conservative:
|
||||
* **Never self-deploys** — it only OPENS a fix task; the fix still ships through
|
||||
the normal gates (dev -> QA -> PR review -> the CEO's merge). The engine never
|
||||
starts / approves / merges / deploys.
|
||||
* **Bounded + deduped per repo** — at most one open ci_watch task per repo
|
||||
(keyed on ``git_url``, so a monorepo's several cell-projects share one fix
|
||||
task), plus per-cycle and rolling open-task caps.
|
||||
* **Bounded + deduped per (repo, workflow)** — at most one open ci_watch task
|
||||
per ``(git_url, effective workflow)``: a same-workflow monorepo (several
|
||||
cell-projects on one repo) shares one fix task, but two RED workflows of one
|
||||
repo each get their own (#44). Plus per-cycle and rolling open-task caps.
|
||||
|
||||
Reuses the hardened per-project CI lookup via ``MultiProjectCITelemetrySource``;
|
||||
the single-repo self-heal path is untouched.
|
||||
@@ -136,12 +137,24 @@ class CiWatchEngine(BaseService):
|
||||
async def _should_open(self, task_svc: TaskService, project: Any) -> bool:
|
||||
"""True when ``project`` resolves and has no open ci_watch task yet.
|
||||
|
||||
Dedupe is per ``git_url`` so a monorepo (several cell-projects, one repo)
|
||||
gets a single open fix task, not one per cell-project.
|
||||
Dedupe is per ``(git_url, effective workflow)`` so a monorepo with
|
||||
several cell-projects on ONE repo still gets a single open fix task per
|
||||
workflow — a same-workflow monorepo collapses to one task, but two RED
|
||||
workflows of one repo each get their own (#44). The effective workflow is
|
||||
``ci_watch_workflow`` falling back to the configured default.
|
||||
"""
|
||||
if project is None or getattr(project, "id", None) is None:
|
||||
return False
|
||||
existing = await task_svc.list_open_ci_watch_tasks(git_url=project.git_url)
|
||||
workflow = (
|
||||
str(
|
||||
getattr(project, "ci_watch_workflow", None)
|
||||
or settings.ci_watch_default_workflow
|
||||
).strip()
|
||||
or None
|
||||
)
|
||||
existing = await task_svc.list_open_ci_watch_tasks(
|
||||
git_url=project.git_url, workflow=workflow
|
||||
)
|
||||
return not existing
|
||||
|
||||
async def _open_fix_task(
|
||||
|
||||
@@ -35,6 +35,7 @@ from roboco.services.base import BaseService
|
||||
from roboco.services.git import CONVENTIONS_SCAFFOLD_BRANCH, get_git_service
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from roboco.db.tables import ProjectTable
|
||||
@@ -61,6 +62,23 @@ class ConventionsHealth:
|
||||
last_ok_sha: str | None
|
||||
|
||||
|
||||
def _is_unique_violation(exc: IntegrityError) -> bool:
|
||||
"""Whether ``exc`` is a UNIQUE constraint violation (SQLSTATE 23505).
|
||||
|
||||
asyncpg exposes ``sqlstate`` on the wrapped error; psycopg exposes
|
||||
``pgcode`` / ``sqlstate``. The class-name fallback covers a driver whose
|
||||
orig lacks a code attribute. Anything else (FK / NOT NULL / check) is a
|
||||
real bug, not a benign concurrent duplicate (#130).
|
||||
"""
|
||||
orig = getattr(exc, "orig", None)
|
||||
if orig is None:
|
||||
return False
|
||||
code = getattr(orig, "sqlstate", None) or getattr(orig, "pgcode", None)
|
||||
if code == "23505":
|
||||
return True
|
||||
return "UniqueViolation" in type(orig).__name__
|
||||
|
||||
|
||||
class ConventionsService(BaseService):
|
||||
"""Cache, render, scaffold, and restore a project's conventions standard."""
|
||||
|
||||
@@ -71,18 +89,23 @@ class ConventionsService(BaseService):
|
||||
pid = self._pid(project)
|
||||
root, head = self._resolve(project, workspace)
|
||||
cached = await self._cache_get(pid, head)
|
||||
if cached is not None:
|
||||
# A cached ``degraded`` row is not trusted: a degraded file may have
|
||||
# been repaired in place at the same (stale) head key, and serving the
|
||||
# cached last-good map would hide the repair. Re-derive instead (#132).
|
||||
if cached is not None and cached.status != "degraded":
|
||||
return ConventionsStandard.model_validate(cached.effective_map)
|
||||
|
||||
file_standard, status = self._read_committed_standard(root)
|
||||
if status == "degraded":
|
||||
last_good = await self._latest_ok_map(pid)
|
||||
if last_good is not None:
|
||||
await self._cache_put(pid, head, last_good, status)
|
||||
# Not cached: a degraded row is unstable (the file may be
|
||||
# repaired in place), so never pin it — re-derive next call.
|
||||
return last_good
|
||||
|
||||
mapping = effective_map(self._derive(root), file_standard)
|
||||
await self._cache_put(pid, head, mapping, status)
|
||||
if status != "degraded":
|
||||
await self._cache_put(pid, head, mapping, status)
|
||||
return mapping
|
||||
|
||||
async def baseline_constraints(
|
||||
@@ -183,13 +206,23 @@ class ConventionsService(BaseService):
|
||||
async def health(
|
||||
self, project: ProjectTable, *, workspace: Path | None = None
|
||||
) -> ConventionsHealth:
|
||||
"""Report the standard's status at HEAD + the last-good commit SHA."""
|
||||
"""Report the standard's status at HEAD + the last-good commit SHA.
|
||||
|
||||
The status is the LIVE file state, not a cached row: a cached
|
||||
``degraded`` can hide an in-place repair at the same (stale) head key
|
||||
(#132). The map scan is expensive (cached); a single file parse is
|
||||
cheap (re-read). ``unknown`` is reserved for a project with no
|
||||
resolvable workspace at all.
|
||||
"""
|
||||
pid = self._pid(project)
|
||||
_root, head = self._resolve(project, workspace)
|
||||
current = await self._cache_get(pid, head)
|
||||
root, head = self._resolve(project, workspace)
|
||||
if root is None:
|
||||
status = "unknown"
|
||||
else:
|
||||
_file_standard, status = self._read_committed_standard(root)
|
||||
last_ok = await self._latest_ok_row(pid)
|
||||
return ConventionsHealth(
|
||||
status=current.status if current is not None else "unknown",
|
||||
status=status,
|
||||
head_sha=head,
|
||||
last_ok_sha=last_ok.commit_sha if last_ok is not None else None,
|
||||
)
|
||||
@@ -415,7 +448,21 @@ class ConventionsService(BaseService):
|
||||
status=status,
|
||||
)
|
||||
)
|
||||
except IntegrityError:
|
||||
except IntegrityError as exc:
|
||||
# Only a UNIQUE violation (23505) is the benign concurrent-duplicate
|
||||
# case the savepoint is for. A FK / NOT NULL / check violation is a
|
||||
# real bug — silently misattributing it as "concurrent put" would
|
||||
# hide the failure (#130), so log-error and re-raise instead. The
|
||||
# savepoint was rolled back (only the failed insert), leaving the
|
||||
# outer task-create transaction usable.
|
||||
if not _is_unique_violation(exc):
|
||||
self.log.error(
|
||||
"conventions cache insert failed (non-unique integrity error)",
|
||||
project_id=str(project_id),
|
||||
commit_sha=commit_sha,
|
||||
error=str(exc),
|
||||
)
|
||||
raise
|
||||
self.log.debug(
|
||||
"conventions cache row already present (concurrent put)",
|
||||
project_id=str(project_id),
|
||||
|
||||
+37
-14
@@ -218,8 +218,11 @@ class DocsService(BaseService):
|
||||
# 4. Search RAG for similar existing documentation (by content, not just title)
|
||||
existing_path = await self._find_similar_doc(req.title, req.content, team)
|
||||
|
||||
if existing_path:
|
||||
# UPDATE existing doc instead of creating new
|
||||
if existing_path and Path(existing_path).name == req.filename:
|
||||
# UPDATE the similar doc — but only when the agent is writing the
|
||||
# SAME filename. A different filename means the agent intends a new
|
||||
# file; collapsing it onto the similar doc's path would overwrite
|
||||
# an unrelated file (#35).
|
||||
result = await self._update_existing_doc(
|
||||
agent_id=agent_id,
|
||||
existing_path=existing_path,
|
||||
@@ -227,7 +230,8 @@ class DocsService(BaseService):
|
||||
doc_type=doc_type,
|
||||
)
|
||||
else:
|
||||
# 5. No similar doc found - create new
|
||||
# 5. No similar doc found (or the similar doc has a different
|
||||
# filename) — create new.
|
||||
result = await self._create_new_doc(
|
||||
agent_id=agent_id,
|
||||
team=team,
|
||||
@@ -238,18 +242,26 @@ class DocsService(BaseService):
|
||||
# 6. Persist the doc into the project's repo and commit it onto the task
|
||||
# branch, so the documenter's output actually lands in the repository via
|
||||
# the open PR — not only in the /app/docs knowledge store. Best-effort: a
|
||||
# documenter without a cloned workspace / task branch still succeeds.
|
||||
await self._commit_doc_to_repo(agent_id, req, doc_type)
|
||||
return result
|
||||
# documenter without a cloned workspace / task branch still succeeds, but
|
||||
# the outcome is surfaced on the doc_ref (#34) so a failed commit is not
|
||||
# silently swallowed.
|
||||
commit_status = await self._commit_doc_to_repo(agent_id, req, doc_type)
|
||||
rel_path, doc_ref, is_update = result
|
||||
doc_ref.commit_status = commit_status
|
||||
return rel_path, doc_ref, is_update
|
||||
|
||||
async def _commit_doc_to_repo(
|
||||
self, agent_id: str, req: WriteDocInput, doc_type: str
|
||||
) -> None:
|
||||
) -> str:
|
||||
"""Write the doc into the project's workspace clone and commit it onto
|
||||
the task branch, so it persists to the repository through the open PR.
|
||||
|
||||
Best-effort: any failure (no task branch yet, workspace not cloned, a git
|
||||
hiccup) is logged and swallowed — it must never fail the doc write.
|
||||
Best-effort: a failure (no task branch yet, workspace not cloned, a git
|
||||
hiccup) never fails the doc write, but the outcome is returned so the
|
||||
caller can surface it (#34) — ``committed`` (landed on the branch),
|
||||
``skipped`` (no branch / workspace / actor to commit onto — the doc
|
||||
still saved to /app/docs), or ``failed`` (a git hiccup — tell the cell
|
||||
PM the doc did not reach the repo).
|
||||
"""
|
||||
from roboco.services.git import get_git_service
|
||||
from roboco.services.project import get_project_service
|
||||
@@ -260,15 +272,15 @@ class DocsService(BaseService):
|
||||
)
|
||||
task = result.scalar_one_or_none()
|
||||
if task is None or not task.branch_name or task.project_id is None:
|
||||
return
|
||||
return "skipped"
|
||||
project = await get_project_service(self.session).get(
|
||||
UUID(str(task.project_id))
|
||||
)
|
||||
if project is None:
|
||||
return
|
||||
return "skipped"
|
||||
actor = self._agent_uuid(agent_id)
|
||||
if actor is None:
|
||||
return
|
||||
return "skipped"
|
||||
|
||||
git = get_git_service(self.session)
|
||||
workspace = await git.get_workspace(project.slug, actor)
|
||||
@@ -292,13 +304,18 @@ class DocsService(BaseService):
|
||||
task_id=str(req.task_id),
|
||||
path=rel_path,
|
||||
)
|
||||
return "committed"
|
||||
except Exception as exc:
|
||||
# Fail-loud (#34): a git hiccup is surfaced as ``failed`` (logged),
|
||||
# not silently swallowed — the agent can tell the cell PM the doc
|
||||
# did not reach the repo. The /app/docs write already succeeded.
|
||||
self.log.warning(
|
||||
"Could not commit documentation to repo (non-fatal)",
|
||||
agent_id=agent_id,
|
||||
task_id=str(req.task_id),
|
||||
error=str(exc),
|
||||
)
|
||||
return "failed"
|
||||
|
||||
@staticmethod
|
||||
def _agent_uuid(agent_id: str) -> UUID | None:
|
||||
@@ -390,7 +407,10 @@ class DocsService(BaseService):
|
||||
else:
|
||||
rel_path = f"{team_path}/{req.filename}"
|
||||
|
||||
full_path = DOCS_BASE_PATH / rel_path
|
||||
# Containment-check the built path (#33): a bare ``DOCS_BASE_PATH /
|
||||
# rel_path`` would silently escape on a malformed team path. The
|
||||
# filename is already validated, but the resolved path is asserted too.
|
||||
full_path = _resolve_contained_path(DOCS_BASE_PATH, rel_path)
|
||||
|
||||
self.log.info(
|
||||
"Creating new documentation",
|
||||
@@ -428,7 +448,10 @@ class DocsService(BaseService):
|
||||
doc_type: str,
|
||||
) -> tuple[str, DocRef, bool]:
|
||||
"""Update an existing documentation file."""
|
||||
full_path = DOCS_BASE_PATH / existing_path
|
||||
# The update path comes from RAG ``source`` — containment-check it
|
||||
# (#33): a ``source`` that escapes the docs dir must be refused, not
|
||||
# written / overwritten.
|
||||
full_path = _resolve_contained_path(DOCS_BASE_PATH, existing_path)
|
||||
|
||||
self.log.info(
|
||||
"Updating existing documentation (RAG dedup)",
|
||||
|
||||
@@ -919,6 +919,7 @@ class Choreographer:
|
||||
agent_id: UUID,
|
||||
task: Any,
|
||||
role_str: str | None = None,
|
||||
skip_dev_guards: bool = False,
|
||||
) -> Envelope | None:
|
||||
"""Run concurrency-invariant claim guards. Returns rejection or None.
|
||||
|
||||
@@ -934,9 +935,16 @@ class Choreographer:
|
||||
guard ``unmet_dependency`` still applies to everyone. A ``None`` role
|
||||
keeps the full guards (safe default for non-PM callers).
|
||||
|
||||
``skip_dev_guards`` skips the dev-only guards (``already_active`` /
|
||||
``paused`` / ``_lane_claim_guard``) for a non-transitioning inspection
|
||||
claim — a pr_reviewer claiming an awaiting_pr_review gate task does not
|
||||
start work, so the single-active-task / code-lane invariants that gate a
|
||||
developer starting a code task do not apply (the dependency guard still
|
||||
runs). See ``claim_gate_review`` (#192).
|
||||
|
||||
Pre-gateway location: _helpers.py:124-204.
|
||||
"""
|
||||
if role_str not in self._COORDINATOR_ROLES:
|
||||
if not skip_dev_guards and role_str not in self._COORDINATOR_ROLES:
|
||||
in_progress = await self.task.list_in_progress_for_agent(agent_id)
|
||||
if guard := already_active_guard(in_progress, task.id):
|
||||
return guard
|
||||
@@ -972,6 +980,8 @@ class Choreographer:
|
||||
# unless the task is currently claimed/in_progress.
|
||||
await self.task.release_dependency_blocked_claim(task.id)
|
||||
return guard
|
||||
if skip_dev_guards:
|
||||
return None
|
||||
return await self._lane_claim_guard(task)
|
||||
|
||||
async def _lane_claim_guard(self, task: Any) -> Envelope | None:
|
||||
@@ -4680,15 +4690,19 @@ class Choreographer:
|
||||
)
|
||||
return None
|
||||
|
||||
_CELL_PM_SLUGS: ClassVar[frozenset[str]] = frozenset({"be-pm", "fe-pm", "ux-pm"})
|
||||
|
||||
@staticmethod
|
||||
def _validate_assignee_task_type(assigned_to: str, task_type: str) -> str | None:
|
||||
"""Reject role-vs-type misclassifications.
|
||||
|
||||
Rules:
|
||||
- delegating to a Cell PM requires
|
||||
``task_type='planning'``. Cell PMs decompose; they don't execute.
|
||||
- delegating to a PM (Cell PM OR Main PM) requires
|
||||
``task_type='planning'``. PMs decompose/coordinate; they don't
|
||||
execute. A freshly delegated subtask is never in ``needs_revision``,
|
||||
so the issue-resolution carve-out (a PM taking a code task to resolve
|
||||
review issues) does not apply here — that lives at the claim gate
|
||||
(``lifecycle._check_claim_rules_narrow`` via ``pm_cannot_own_code``).
|
||||
Main PM was previously omitted (only the cell-PM slug set was
|
||||
checked), leaving a delegate-to-main-pm-as-code hole.
|
||||
- (2026-05-11 smoke): delegating to a Developer requires
|
||||
``task_type in {'code', 'documentation', 'research'}``. Devs
|
||||
implement. Planning/design/administrative belong to PMs/board.
|
||||
@@ -4707,13 +4721,11 @@ class Choreographer:
|
||||
to review PRs of code changes).
|
||||
- Delegating to a Documenter requires ``task_type='documentation'``.
|
||||
"""
|
||||
pm_err = Choreographer._pm_task_type_error(assigned_to, task_type)
|
||||
if pm_err is not None:
|
||||
return pm_err
|
||||
from roboco.foundation.identity import AGENTS, Role, Team
|
||||
|
||||
if assigned_to in Choreographer._CELL_PM_SLUGS and task_type != "planning":
|
||||
return (
|
||||
f"task_type={task_type!r} is invalid for assignee {assigned_to!r}: "
|
||||
f"Cell PMs own planning tasks, not code/documentation/etc."
|
||||
)
|
||||
agent = AGENTS.get(assigned_to)
|
||||
if agent is None:
|
||||
return None
|
||||
@@ -4736,6 +4748,28 @@ class Choreographer:
|
||||
)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _pm_task_type_error(assigned_to: str, task_type: str) -> str | None:
|
||||
"""Reject a code/non-planning task_type delegated to a PM (cell or main).
|
||||
|
||||
Extracted from ``_validate_assignee_task_type`` so the compound PM guard
|
||||
doesn't inflate the dispatcher's complexity. A freshly delegated subtask
|
||||
is never in ``needs_revision``, so the issue-resolution carve-out does
|
||||
not apply here (it lives at the claim gate).
|
||||
"""
|
||||
from roboco.foundation.identity import AGENTS, Role
|
||||
|
||||
agent = AGENTS.get(assigned_to)
|
||||
if agent is None or agent.role not in (Role.CELL_PM, Role.MAIN_PM):
|
||||
return None
|
||||
if task_type == "planning":
|
||||
return None
|
||||
return (
|
||||
f"task_type={task_type!r} is invalid for assignee"
|
||||
f" {assigned_to!r}: PMs own planning tasks, not"
|
||||
f" code/documentation/etc."
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _developer_task_type_error(
|
||||
assigned_to: str, is_ux_dev: bool, task_type: str
|
||||
@@ -4767,13 +4801,19 @@ class Choreographer:
|
||||
"""
|
||||
from roboco.foundation.identity import AGENTS, Role, Team
|
||||
|
||||
if assigned_to in Choreographer._CELL_PM_SLUGS:
|
||||
agent = AGENTS.get(assigned_to)
|
||||
if agent is not None and agent.role is Role.MAIN_PM:
|
||||
return (
|
||||
"Main PMs own PLANNING tasks — they coordinate across cells and "
|
||||
"delegate execution. Pass task_type='planning' when delegating to "
|
||||
"the Main PM; route code work to a developer via the cell PM."
|
||||
)
|
||||
if agent is not None and agent.role is Role.CELL_PM:
|
||||
return (
|
||||
"Cell PMs (be-pm/fe-pm/ux-pm) own PLANNING tasks — they "
|
||||
"decompose the slice and delegate code work to devs. Pass "
|
||||
"task_type='planning' when delegating to a Cell PM."
|
||||
)
|
||||
agent = AGENTS.get(assigned_to)
|
||||
if agent is not None and agent.role is Role.DEVELOPER:
|
||||
if agent.team is Team.UX_UI:
|
||||
return (
|
||||
@@ -6013,6 +6053,10 @@ class Choreographer:
|
||||
"already present in the base via a sibling PR that merged "
|
||||
"first. Completing the task without a redundant merge."
|
||||
),
|
||||
# Preserve the superseded branch (audit / reference), matching
|
||||
# the orchestrator supersede path. close_pull_request defaults
|
||||
# to non-destructive now; explicit here so the two paths agree.
|
||||
delete_branch=False,
|
||||
actor_agent_id=pm_agent_id,
|
||||
project_id=cast("UUID", t.project_id),
|
||||
)
|
||||
@@ -6218,7 +6262,15 @@ class Choreographer:
|
||||
return None
|
||||
sha = await self.git.get_pr_head_sha(slug, int(pr_number))
|
||||
return sha if isinstance(sha, str) else None
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
# Fail-open (never wedge the PM on a lookup error), but log so a
|
||||
# regression in the slug resolver / git helper doesn't silently
|
||||
# turn the pr_fail re-submit loop-stopper into a no-op (#5).
|
||||
logger.warning(
|
||||
"unchanged_pr_guard head_sha lookup failed (fail-open)",
|
||||
pr_number=pr_number,
|
||||
error=str(exc),
|
||||
)
|
||||
return None
|
||||
|
||||
async def _submit_up_unchanged_pr_guard(
|
||||
|
||||
@@ -112,6 +112,7 @@ class ChoreographerHelpers:
|
||||
agent_id: UUID,
|
||||
task: Any,
|
||||
role_str: str | None = None,
|
||||
skip_dev_guards: bool = False,
|
||||
) -> Envelope | None:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
@@ -81,7 +81,11 @@ class PRGateMixin(_Base):
|
||||
task_id=task_id,
|
||||
verb="claim_gate_review",
|
||||
)
|
||||
guard = await self._run_claim_guards(agent_id=reviewer_agent_id, task=t)
|
||||
guard = await self._run_claim_guards(
|
||||
agent_id=reviewer_agent_id,
|
||||
task=t,
|
||||
skip_dev_guards=True,
|
||||
)
|
||||
if guard:
|
||||
guard.with_introspection(task=t, role=role_str)
|
||||
return await self._emit_rejection(
|
||||
@@ -228,19 +232,55 @@ class PRGateMixin(_Base):
|
||||
|
||||
async def _record_gate_verdict_for(
|
||||
self, verb: str, t: Any, notes: str, *, issues: tuple[str, ...]
|
||||
) -> None:
|
||||
) -> str | None:
|
||||
"""Author the canonical pr_review verdict note before the transition.
|
||||
|
||||
On pr_fail also stamp the assembled PR's head SHA so the next submit_root
|
||||
can structurally refuse to re-submit the unchanged root (the 2026-06-27
|
||||
infinite pr_fail re-submit loop). Best-effort: a capture failure leaves
|
||||
head_sha absent and submit_root fails open rather than wedging the PM.
|
||||
|
||||
Returns the captured head_sha for pr_fail (None for pr_pass) so the caller
|
||||
can re-capture after the transition commits and re-stamp if the PR head
|
||||
advanced in between (#189 staleness).
|
||||
"""
|
||||
if verb == "pr_fail":
|
||||
head_sha = await self._capture_pr_head_sha(t)
|
||||
self._record_gate_verdict(t, verb, notes, issues=issues, head_sha=head_sha)
|
||||
else:
|
||||
self._record_gate_verdict(t, verb, notes, issues=issues)
|
||||
return head_sha
|
||||
self._record_gate_verdict(t, verb, notes, issues=issues)
|
||||
return None
|
||||
|
||||
async def _re_stamp_pr_fail_head_sha_if_advanced(
|
||||
self,
|
||||
t: Any,
|
||||
notes: str,
|
||||
*,
|
||||
issues: tuple[str, ...],
|
||||
pre_sha: str | None,
|
||||
) -> None:
|
||||
"""Re-capture the PR head SHA after the transition commits and re-stamp
|
||||
the verdict note when it advanced past the pre-transition capture (#189).
|
||||
|
||||
The pre-transition capture can go stale if cell work lands on the root
|
||||
branch between that capture and the commit; a stale recorded SHA makes
|
||||
``submit_root`` false-allow an unchanged re-submit and re-opens the
|
||||
pr_fail loop. Re-stamping only on a real advance keeps the no-advance
|
||||
case to a single note write. Best-effort: a re-capture failure leaves
|
||||
the pre-transition SHA in place (the fail-open direction).
|
||||
"""
|
||||
try:
|
||||
post_sha = await self._capture_pr_head_sha(t)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"pr_fail head-sha re-capture failed (keeping pre-transition sha)",
|
||||
task_id=str(getattr(t, "id", "")),
|
||||
)
|
||||
return
|
||||
if post_sha is not None and post_sha != pre_sha:
|
||||
self._record_gate_verdict(
|
||||
t, "pr_fail", notes, issues=issues, head_sha=post_sha
|
||||
)
|
||||
|
||||
async def _post_gate_review(
|
||||
self, t: Any, agent: Any, role_str: str, verb: str, notes: str
|
||||
@@ -319,7 +359,7 @@ class PRGateMixin(_Base):
|
||||
# Author the canonical pr_review verdict note BEFORE the transition so it
|
||||
# is persisted by the same commit (mirrors post_pr_review) and stays in
|
||||
# lock-step with the decision (pr_fail overwrites an earlier pr_pass).
|
||||
await self._record_gate_verdict_for(verb, t, notes, issues=issues)
|
||||
pre_sha = await self._record_gate_verdict_for(verb, t, notes, issues=issues)
|
||||
runner = self._verb_runner()
|
||||
try:
|
||||
t = await runner.run_intent(verb, t, agent, spec_ctx)
|
||||
@@ -356,6 +396,18 @@ class PRGateMixin(_Base):
|
||||
task_id=task_id,
|
||||
verb=verb,
|
||||
)
|
||||
# pr_fail: re-capture the PR head SHA AFTER the transition commits. The
|
||||
# pre-transition capture (in _record_gate_verdict_for) can go stale if
|
||||
# cell work lands on the root branch between that capture and the commit
|
||||
# — a stale recorded SHA would make submit_root false-allow an unchanged
|
||||
# re-submit (current head vs an older recorded head ⇒ "different") and
|
||||
# re-open the pr_fail loop. Re-stamp the note only when the head actually
|
||||
# advanced, so the no-advance case stays a single note write. Best-effort:
|
||||
# a re-capture failure leaves the pre-transition SHA in place (fail-open).
|
||||
if verb == "pr_fail":
|
||||
await self._re_stamp_pr_fail_head_sha_if_advanced(
|
||||
t, notes, issues=issues, pre_sha=pre_sha
|
||||
)
|
||||
# Post the gate verdict on the PR itself (best-effort, after the DB
|
||||
# transition — a GitHub failure must not roll back the gate decision).
|
||||
await self._post_gate_review(t, agent, role_str, verb, notes)
|
||||
@@ -499,6 +551,36 @@ class PRGateMixin(_Base):
|
||||
)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _gate_review_event_verdict(verb: str, is_root: bool) -> tuple[str, str]:
|
||||
"""Map the gate verb to a (review event, verdict label) pair.
|
||||
|
||||
``pr_pass`` → APPROVE, ``pr_fail`` → REQUEST_CHANGES — except on the
|
||||
root→master PR (``is_root``), which always gets a plain COMMENT so the
|
||||
gate never leaves an approval that could satisfy branch protection (only
|
||||
the CEO merges master) nor a blocking review that could impede that merge.
|
||||
"""
|
||||
if verb == "pr_pass":
|
||||
return "COMMENT" if is_root else "APPROVE", "PASSED ✅"
|
||||
return "COMMENT" if is_root else "REQUEST_CHANGES", "CHANGES REQUESTED 🔴"
|
||||
|
||||
@staticmethod
|
||||
def _gate_review_body(
|
||||
verdict: str, reviewer_slug: str, notes: str, is_root: bool
|
||||
) -> str:
|
||||
"""Render the gate-review comment body posted to the assembled PR."""
|
||||
body_lines = [
|
||||
f"## In-path PR-review gate — {verdict}",
|
||||
"",
|
||||
f"Reviewed by **{reviewer_slug}** (RoboCo PR reviewer). Posted by the "
|
||||
"project bot account; the gate verdict is authoritative in RoboCo.",
|
||||
"",
|
||||
(notes or "").strip() or "_(no additional notes)_",
|
||||
]
|
||||
if is_root:
|
||||
body_lines += ["", "_Only the CEO merges this PR into `master`._"]
|
||||
return "\n".join(body_lines)
|
||||
|
||||
async def _post_gate_review_to_pr(
|
||||
self, t: Any, verb: str, reviewer_slug: str, notes: str
|
||||
) -> None:
|
||||
@@ -512,31 +594,22 @@ class PRGateMixin(_Base):
|
||||
For the org's own PRs ``git.post_pr_review`` already downgrades a
|
||||
forbidden self-review to a COMMENT, so the verdict lands regardless.
|
||||
"""
|
||||
slug = await self._project_slug_for(t)
|
||||
try:
|
||||
slug = await self._project_slug_for(t)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"gate review PR post: slug resolve failed",
|
||||
task_id=str(getattr(t, "id", "")),
|
||||
)
|
||||
return
|
||||
pr_number = getattr(t, "pr_number", None)
|
||||
if not slug or not pr_number:
|
||||
return
|
||||
is_root = getattr(t, "parent_task_id", None) is None
|
||||
if verb == "pr_pass":
|
||||
event = "COMMENT" if is_root else "APPROVE"
|
||||
verdict = "PASSED ✅"
|
||||
else:
|
||||
event = "COMMENT" if is_root else "REQUEST_CHANGES"
|
||||
verdict = "CHANGES REQUESTED 🔴"
|
||||
body_lines = [
|
||||
f"## In-path PR-review gate — {verdict}",
|
||||
"",
|
||||
f"Reviewed by **{reviewer_slug}** (RoboCo PR reviewer). Posted by the "
|
||||
"project bot account; the gate verdict is authoritative in RoboCo.",
|
||||
"",
|
||||
(notes or "").strip() or "_(no additional notes)_",
|
||||
]
|
||||
if is_root:
|
||||
body_lines += ["", "_Only the CEO merges this PR into `master`._"]
|
||||
event, verdict = self._gate_review_event_verdict(verb, is_root)
|
||||
body = self._gate_review_body(verdict, reviewer_slug, notes, is_root)
|
||||
try:
|
||||
await self.git.post_pr_review(
|
||||
slug, int(pr_number), "\n".join(body_lines), event=event
|
||||
)
|
||||
await self.git.post_pr_review(slug, int(pr_number), body, event=event)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"gate review PR post failed", task_id=str(getattr(t, "id", ""))
|
||||
|
||||
@@ -170,10 +170,17 @@ class PRReviewerMixin(_Base):
|
||||
plus the ones a hand-formatter reaches for (``## Summary`` / ``## Issues``
|
||||
/ ``## Verdict``). A real one-paragraph summary does not contain ``## ``
|
||||
headers, so the prose word "summary" never trips this.
|
||||
|
||||
The header must sit at the START of a line (a real markdown header the
|
||||
reviewer authored) — a header quoted from the PR itself (``> ## Summary``)
|
||||
or named mid-prose (``the ## Summary section``) is a citation, not a
|
||||
hand-formatted verdict, and must not trip the guard.
|
||||
"""
|
||||
import re
|
||||
|
||||
lowered = (body or "").lower()
|
||||
return any(
|
||||
header in lowered
|
||||
re.search(rf"^[ \t]*{re.escape(header)}", lowered, re.MULTILINE)
|
||||
for header in ("## summary", "## issues", "## verdict", "## findings")
|
||||
)
|
||||
|
||||
|
||||
@@ -843,8 +843,31 @@ class ContentActions:
|
||||
# if this transaction rolled back — a divergence agents surface in
|
||||
# briefings. ``get_db`` commits the session again after the route returns
|
||||
# (a no-op on the now-clean transaction); this explicit commit is what
|
||||
# gates the index.
|
||||
await self.task.session.commit()
|
||||
# gates the index. A poisoned session (a prior mid-verb failure rolled it
|
||||
# back -> PendingRollbackError) must NOT 500 the curation verb NOR fall
|
||||
# through to index an uncommitted playbook: surface a clean invalid_state
|
||||
# and skip the index. See #55.
|
||||
from sqlalchemy.exc import PendingRollbackError
|
||||
|
||||
try:
|
||||
await self.task.session.commit()
|
||||
except PendingRollbackError:
|
||||
logger.warning(
|
||||
"playbook curate: gating commit on a rolled-back session",
|
||||
action=action,
|
||||
playbook_id=str(playbook_id),
|
||||
)
|
||||
return Envelope.invalid_state(
|
||||
message=(
|
||||
"the DB session was rolled back by a prior failure; the "
|
||||
"playbook status change was not committed"
|
||||
),
|
||||
remediate=(
|
||||
"re-fetch the playbook's current status and re-try the "
|
||||
"curation verb"
|
||||
),
|
||||
context_briefing={"playbook_id": str(playbook_id)},
|
||||
)
|
||||
if action == "approve":
|
||||
await svc.index_approved(playbook)
|
||||
else:
|
||||
|
||||
@@ -19,13 +19,14 @@ from roboco.config import settings
|
||||
|
||||
# Server-side atomic read-modify-write scripts. Redis single-threads a Lua
|
||||
# ``EVAL``, so the GET → decode → mutate → SET inside one script is indivisible:
|
||||
# a concurrent ``activate()`` (a re-park) is serialized entirely before or after
|
||||
# the script, never interleaved between the script's GET and SET. Without this
|
||||
# the counter update was a non-atomic ``get_state`` → mutate → ``set`` in Python,
|
||||
# so a re-park's fresh episode blob (``probe_failures: 0`` + fresh
|
||||
# ``activated_at`` / ``retry_after`` / ``affected_agents`` / ``kind``) could be
|
||||
# clobbered by the stale increment writing back the OLD blob — un-resetting the
|
||||
# counter and overwriting the fresh episode metadata. The scripts mutate ONLY
|
||||
# a concurrent op is serialized entirely before or after the script, never
|
||||
# interleaved between the script's GET and SET. Without this the counter update
|
||||
# was a non-atomic ``get_state`` → mutate → ``set`` in Python, so a re-park's
|
||||
# fresh episode blob could be clobbered by the stale increment writing back the
|
||||
# OLD blob. ``activate`` is itself a Lua merge (not a blind SET): it refreshes
|
||||
# the episode metadata but carries over the previous ``probe_failures`` count,
|
||||
# so a re-park can no longer wipe an in-flight increment (resetting the give-up
|
||||
# / CEO-notify count mid-episode). The counter scripts mutate ONLY
|
||||
# ``probe_failures`` so every other episode field survives the bump.
|
||||
_INCREMENT_PROBE_FAILURES = """\
|
||||
-- roboco:increment_probe_failures
|
||||
@@ -57,6 +58,32 @@ state['probe_failures'] = 0
|
||||
redis.call('SET', key, cjson.encode(state))
|
||||
"""
|
||||
|
||||
# activate merges: it refreshes the episode metadata (kind / activated_at /
|
||||
# retry_after / affected_agents) but carries over the previous probe_failures
|
||||
# count. A blind SET here (the old impl) reset probe_failures to 0, so a
|
||||
# probe-failure increment that just landed — or was in flight — could be wiped
|
||||
# by a concurrent re-park, resetting the give-up / CEO-notify count mid-episode.
|
||||
# Like increment/reset, the read-merge-write runs server-side as one atomic Lua
|
||||
# EVAL, so it is indivisible w.r.t. the counter scripts.
|
||||
_ACTIVATE_RATE_LIMIT = """\
|
||||
-- roboco:activate_rate_limit
|
||||
local key = KEYS[1]
|
||||
local fresh = ARGV[1]
|
||||
local raw = redis.call('GET', key)
|
||||
if raw then
|
||||
local prev = cjson.decode(raw)
|
||||
local old_pf = prev['probe_failures']
|
||||
if old_pf ~= nil then
|
||||
local new_state = cjson.decode(fresh)
|
||||
new_state['probe_failures'] = old_pf
|
||||
redis.call('SET', key, cjson.encode(new_state))
|
||||
return old_pf
|
||||
end
|
||||
end
|
||||
redis.call('SET', key, fresh)
|
||||
return 0
|
||||
"""
|
||||
|
||||
|
||||
class RateLimitStateTracker:
|
||||
"""Track rate-limit state for a single AI provider in Redis.
|
||||
@@ -131,7 +158,9 @@ class RateLimitStateTracker:
|
||||
"affected_agents": affected_agents or [],
|
||||
"probe_failures": 0,
|
||||
}
|
||||
await r.set(self._key(), json.dumps(state))
|
||||
# Atomic merge (see _ACTIVATE_RATE_LIMIT): the previous probe_failures
|
||||
# count is carried over so a re-park cannot wipe an in-flight increment.
|
||||
await r.eval(_ACTIVATE_RATE_LIMIT, 1, self._key(), json.dumps(state))
|
||||
|
||||
async def clear(self) -> None:
|
||||
"""Remove rate-limit state for this provider."""
|
||||
|
||||
+35
-3
@@ -199,6 +199,12 @@ _HTTP_NOT_FOUND = 404
|
||||
# a concurrent sibling-subtask merge updated the target branch and our local
|
||||
# refs are stale. `pr_merge` re-syncs and retries exactly once on this code.
|
||||
_HTTP_CONFLICT = 409
|
||||
# GitHub returns 405 when the repo's settings disallow the requested merge
|
||||
# method (e.g. "Squash merges are not allowed on this repository" with the
|
||||
# squash button off) — distinct from a 405 on an already-merged PR. The agent
|
||||
# `_merge_with_retry` falls back to a permitted method on this code, mirroring
|
||||
# the CEO `merge_pull_request` path.
|
||||
_HTTP_METHOD_NOT_ALLOWED = 405
|
||||
|
||||
# --- Self-heal CI signal -------------------------------------------------
|
||||
# Pull a WINDOW of recent completed runs (not just the single newest) so the
|
||||
@@ -3553,7 +3559,8 @@ class GitService(BaseService):
|
||||
target: str
|
||||
|
||||
async def _merge_with_retry(self, ctx: GitService._MergeContext) -> Any:
|
||||
"""Single-retry merge: on 409 (race), sync target then retry once."""
|
||||
"""Single-retry merge: on 409 (race) sync target then retry; on 405
|
||||
(repo disallows the merge method) fall back to a permitted method."""
|
||||
resp = await self._call_merge_api(
|
||||
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token, "squash"
|
||||
)
|
||||
@@ -3565,6 +3572,28 @@ class GitService(BaseService):
|
||||
resp = await self._call_merge_api(
|
||||
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token, "squash"
|
||||
)
|
||||
if resp.status_code == _HTTP_METHOD_NOT_ALLOWED:
|
||||
# The repo's settings disallow squash (the button is off). Try a
|
||||
# method the repo permits before treating this as a conflict —
|
||||
# mirrors the CEO merge_pull_request 405 fallback so a repo's
|
||||
# merge-button config can't wedge the PM on an open, mergeable PR.
|
||||
# A 405 with no permitted fallback (or a second 405) falls through
|
||||
# to the already-merged disambiguation / MergeConflictError below.
|
||||
fallback = await self._first_allowed_merge_method(
|
||||
ctx.owner, ctx.repo, ctx.git_token, exclude="squash"
|
||||
)
|
||||
if fallback and fallback != "squash":
|
||||
self.log.info(
|
||||
"Merge method refused by repo; retrying with a permitted one",
|
||||
requested="squash",
|
||||
fallback=fallback,
|
||||
owner=ctx.owner,
|
||||
repo=ctx.repo,
|
||||
pr=ctx.pr_number,
|
||||
)
|
||||
resp = await self._call_merge_api(
|
||||
ctx.owner, ctx.repo, ctx.pr_number, ctx.git_token, fallback
|
||||
)
|
||||
if not resp.is_success:
|
||||
# A merge PUT on an ALREADY-MERGED PR returns the same 405 as a
|
||||
# genuine "not mergeable" conflict. An already-merged PR (a prior
|
||||
@@ -3943,14 +3972,17 @@ class GitService(BaseService):
|
||||
*,
|
||||
project_id: UUID,
|
||||
comment: str | None = None,
|
||||
delete_branch: bool = True,
|
||||
delete_branch: bool = False,
|
||||
actor_agent_id: UUID | None = None,
|
||||
) -> None:
|
||||
"""Close PR ``pr_number`` on GitHub, optionally with an explanatory comment.
|
||||
|
||||
Used to retire a PR whose work is already in the base (superseded) so a
|
||||
wedged task can complete without a merge — the "close the dead PR"
|
||||
action agents had no verb for. Best-effort branch cleanup on close.
|
||||
action agents had no verb for. Branch deletion is opt-in
|
||||
(``delete_branch=False`` by default): a superseded PR's branch may still
|
||||
be referenced or useful for audit, so close does not destroy it unless
|
||||
the caller explicitly asks — matching the orchestrator supersede path.
|
||||
|
||||
``pr_number`` alone is ambiguous across projects (GitHub numbers PRs
|
||||
per-repo, but ``tasks.pr_number`` stores the bare integer with no repo
|
||||
|
||||
@@ -37,6 +37,11 @@ class ProvisionedRepo:
|
||||
html_url: str
|
||||
|
||||
|
||||
# GitHub's "name already exists on this account" reply on a duplicate repo
|
||||
# create — the orphaned-repo signal treated idempotently (#83/#84).
|
||||
_GITHUB_REPO_EXISTS_STATUS = 422
|
||||
|
||||
|
||||
class GitHubProvisioningService:
|
||||
"""Create private repos in the configured org via the GitHub REST API."""
|
||||
|
||||
@@ -76,7 +81,15 @@ class GitHubProvisioningService:
|
||||
async def create_repo(
|
||||
self, name: str, description: str = "", *, private: bool = True
|
||||
) -> ProvisionedRepo:
|
||||
"""Create ``org/name`` (auto-initialised so it is immediately cloneable)."""
|
||||
"""Create ``org/name`` (auto-initialised so it is immediately cloneable).
|
||||
|
||||
Idempotent by GitHub name: if a prior partially-rolled-back approval left
|
||||
``org/name`` on GitHub (the DB transaction rolled back but the repo did
|
||||
not), GitHub replies 422 ``name already exists on this account``. Instead
|
||||
of erroring and orphaning the re-approval, fetch and return the existing
|
||||
repo so the caller reuses its ``clone_url`` to (re)register the Project
|
||||
(#83/#84).
|
||||
"""
|
||||
if not self.enabled:
|
||||
msg = (
|
||||
"GitHub provisioning is not configured. Set "
|
||||
@@ -103,6 +116,13 @@ class GitHubProvisioningService:
|
||||
except httpx.HTTPError as exc:
|
||||
msg = f"GitHub repo creation failed for '{name}': {exc}"
|
||||
raise ProvisioningError(msg) from exc
|
||||
if (
|
||||
resp.status_code == _GITHUB_REPO_EXISTS_STATUS
|
||||
and "already exists" in (resp.text or "").lower()
|
||||
):
|
||||
# The repo is already on GitHub from a rolled-back prior attempt —
|
||||
# reuse it instead of orphaning the re-approval.
|
||||
return await self._fetch_existing_repo(name)
|
||||
if not resp.is_success:
|
||||
detail = resp.text[:200] if resp.text else "no body"
|
||||
msg = (
|
||||
@@ -117,6 +137,35 @@ class GitHubProvisioningService:
|
||||
html_url=str(body.get("html_url", "")),
|
||||
)
|
||||
|
||||
async def _fetch_existing_repo(self, name: str) -> ProvisionedRepo:
|
||||
"""GET ``org/name`` and rebuild a ProvisionedRepo (idempotent re-create)."""
|
||||
client = await self._http()
|
||||
try:
|
||||
resp = await client.get(
|
||||
f"{self._base_url}/repos/{self._org}/{name}",
|
||||
headers={
|
||||
"Authorization": f"Bearer {self._token}",
|
||||
"Accept": "application/vnd.github+json",
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
},
|
||||
timeout=self._timeout,
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
msg = f"GitHub repo fetch failed for '{name}': {exc}"
|
||||
raise ProvisioningError(msg) from exc
|
||||
if not resp.is_success:
|
||||
detail = resp.text[:200] if resp.text else "no body"
|
||||
msg = (
|
||||
f"GitHub repo fetch failed for '{name}' ({resp.status_code}): {detail}"
|
||||
)
|
||||
raise ProvisioningError(msg)
|
||||
body = resp.json()
|
||||
return ProvisionedRepo(
|
||||
full_name=str(body.get("full_name", f"{self._org}/{name}")),
|
||||
clone_url=str(body.get("clone_url", "")),
|
||||
html_url=str(body.get("html_url", "")),
|
||||
)
|
||||
|
||||
|
||||
def get_github_provisioning_service(
|
||||
client: httpx.AsyncClient | None = None,
|
||||
|
||||
@@ -8,6 +8,7 @@ Supports swimlanes, cross-cell views, and real-time updates.
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any, ClassVar
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
@@ -43,8 +44,25 @@ class KanbanService(BaseService):
|
||||
# CARD CREATION
|
||||
# =========================================================================
|
||||
|
||||
async def _load_subtask_counts(self, tasks: Sequence[TaskTable]) -> dict[UUID, int]:
|
||||
"""#198: batch-count direct children per parent in ONE grouped query, so a
|
||||
board of N cards doesn't fire N child-count queries (and so the count is
|
||||
real, not a hardcoded 0)."""
|
||||
parent_ids = [t.id for t in tasks if t.id is not None]
|
||||
if not parent_ids:
|
||||
return {}
|
||||
result = await self.session.execute(
|
||||
select(TaskTable.parent_task_id, func.count(TaskTable.id))
|
||||
.where(TaskTable.parent_task_id.in_(parent_ids))
|
||||
.group_by(TaskTable.parent_task_id)
|
||||
)
|
||||
return {row[0]: int(row[1]) for row in result.all() if row[0] is not None}
|
||||
|
||||
async def _task_to_card(
|
||||
self, task: TaskTable, swimlane_key: str | None = None
|
||||
self,
|
||||
task: TaskTable,
|
||||
swimlane_key: str | None = None,
|
||||
subtask_counts: dict[UUID, int] | None = None,
|
||||
) -> KanbanCard:
|
||||
"""Convert a task to a kanban card."""
|
||||
# Get assignee name if assigned
|
||||
@@ -62,8 +80,10 @@ class KanbanService(BaseService):
|
||||
if last_with_percentage:
|
||||
progress = last_with_percentage["percentage"]
|
||||
|
||||
# Count subtasks (would need a query in real implementation)
|
||||
subtask_count = 0
|
||||
# #198: real subtask count from the batch-loaded map (0 when no map / leaf).
|
||||
subtask_count = (
|
||||
subtask_counts.get(require_uuid(task.id), 0) if subtask_counts else 0
|
||||
)
|
||||
|
||||
return KanbanCard(
|
||||
id=require_uuid(task.id),
|
||||
@@ -143,8 +163,9 @@ class KanbanService(BaseService):
|
||||
|
||||
# Add cards to columns
|
||||
blocked_count = 0
|
||||
subtask_counts = await self._load_subtask_counts(tasks)
|
||||
for task in tasks:
|
||||
card = await self._task_to_card(task)
|
||||
card = await self._task_to_card(task, subtask_counts=subtask_counts)
|
||||
|
||||
# Find the right column for this task's status
|
||||
for col_id, _, col_status in column_config:
|
||||
@@ -208,6 +229,7 @@ class KanbanService(BaseService):
|
||||
lane_key: str,
|
||||
lane_tasks: list[TaskTable],
|
||||
column_config: list,
|
||||
subtask_counts: dict[UUID, int] | None = None,
|
||||
) -> tuple[list[KanbanColumn], int]:
|
||||
"""Build columns for a swimlane. Returns (columns, blocked_count)."""
|
||||
columns: list[KanbanColumn] = []
|
||||
@@ -215,7 +237,7 @@ class KanbanService(BaseService):
|
||||
|
||||
for col_id, col_title, col_status in column_config:
|
||||
cards = [
|
||||
await self._task_to_card(t, lane_key)
|
||||
await self._task_to_card(t, lane_key, subtask_counts=subtask_counts)
|
||||
for t in lane_tasks
|
||||
if t.status == col_status
|
||||
]
|
||||
@@ -246,6 +268,8 @@ class KanbanService(BaseService):
|
||||
agent_names = (
|
||||
await self._fetch_agent_names(tasks) if swimlane_by == "assignee" else {}
|
||||
)
|
||||
# #198: load subtask counts once for the whole board (not per lane).
|
||||
subtask_counts = await self._load_subtask_counts(tasks)
|
||||
|
||||
# Group tasks by swimlane key
|
||||
swimlane_groups: dict[str, list[TaskTable]] = {}
|
||||
@@ -259,7 +283,10 @@ class KanbanService(BaseService):
|
||||
|
||||
for lane_key in sorted(swimlane_groups.keys()):
|
||||
columns, blocked = await self._build_swimlane_columns(
|
||||
lane_key, swimlane_groups[lane_key], column_config
|
||||
lane_key,
|
||||
swimlane_groups[lane_key],
|
||||
column_config,
|
||||
subtask_counts=subtask_counts,
|
||||
)
|
||||
total_blocked += blocked
|
||||
|
||||
@@ -419,6 +446,12 @@ class KanbanService(BaseService):
|
||||
KanbanColumn(
|
||||
id="ux_ui", title="UX/UI", status=TaskStatus.IN_PROGRESS, cards=[]
|
||||
),
|
||||
KanbanColumn(
|
||||
id="coordination",
|
||||
title="Coordination",
|
||||
status=TaskStatus.IN_PROGRESS,
|
||||
cards=[],
|
||||
),
|
||||
KanbanColumn(
|
||||
id="done", title="Done", status=TaskStatus.COMPLETED, cards=[]
|
||||
),
|
||||
@@ -427,15 +460,21 @@ class KanbanService(BaseService):
|
||||
# Sort tasks into columns
|
||||
col_map = {col.id: col for col in columns}
|
||||
blocked_count = 0
|
||||
subtask_counts = await self._load_subtask_counts(tasks)
|
||||
|
||||
for task in tasks:
|
||||
card = await self._task_to_card(task)
|
||||
card = await self._task_to_card(task, subtask_counts=subtask_counts)
|
||||
if task.team == Team.BACKEND:
|
||||
col_map["backend"].cards.append(card)
|
||||
elif task.team == Team.FRONTEND:
|
||||
col_map["frontend"].cards.append(card)
|
||||
elif task.team == Team.UX_UI:
|
||||
col_map["ux_ui"].cards.append(card)
|
||||
else:
|
||||
# Non-cell teams (Main PM, Board, fullstack, system, ...) used
|
||||
# to be counted in total_cards but never columned — the card was
|
||||
# built and discarded (#196). Column them under Coordination.
|
||||
col_map["coordination"].cards.append(card)
|
||||
|
||||
if task.status == TaskStatus.BLOCKED:
|
||||
blocked_count += 1
|
||||
|
||||
+21
-1
@@ -34,6 +34,7 @@ from sqlalchemy import delete as sa_delete
|
||||
from sqlalchemy import select
|
||||
|
||||
from roboco.agents_config import get_agent_role
|
||||
from roboco.config import settings
|
||||
from roboco.db.tables import ModelAssignmentTable, ProviderConfigTable
|
||||
from roboco.models.base import AssignmentScope, ModelProvider
|
||||
from roboco.models.llm_catalog import (
|
||||
@@ -84,10 +85,12 @@ async def probe_ollama_tags(base_url: str) -> tuple[list[str], str | None]:
|
||||
except httpx.HTTPStatusError as exc:
|
||||
return [], f"Server at {base_url} returned HTTP {exc.response.status_code}"
|
||||
except Exception as exc:
|
||||
# Log the exception class only — ``str(exc)`` can carry connection
|
||||
# internals / stack traces that don't belong in a structured log.
|
||||
_log.error(
|
||||
"Unexpected error probing Ollama server",
|
||||
base_url=base_url,
|
||||
error=str(exc),
|
||||
error=exc.__class__.__name__,
|
||||
)
|
||||
return [], "An unexpected error occurred while probing the self-hosted server."
|
||||
|
||||
@@ -135,6 +138,23 @@ class ModelRoutingService(BaseService):
|
||||
route = await self._route_from_resolved(resolved, agent_slug)
|
||||
if route is not None:
|
||||
return route
|
||||
elif resolved is not None and not resolved.provider.enabled:
|
||||
# Configured but disabled — distinguishable from "no assignment"
|
||||
# so the bypass is surfaced, not silent. Default stays graceful (a
|
||||
# stalled spawn is worse than a routing miss); ROBOCO_ROUTING_STRICT
|
||||
# opts into fail-closed for operators who'd rather it stall.
|
||||
self.log.warning(
|
||||
"Configured provider is disabled; downgrading to legacy Anthropic path",
|
||||
agent_slug=agent_slug,
|
||||
role=role,
|
||||
provider_id=str(resolved.provider.id),
|
||||
)
|
||||
if settings.routing_strict:
|
||||
raise RuntimeError(
|
||||
f"routing_strict: agent {agent_slug!r} has a disabled configured "
|
||||
f"provider {resolved.provider.id}; refusing to silently downgrade "
|
||||
f"to the legacy Anthropic path"
|
||||
)
|
||||
return self._legacy_route(role)
|
||||
|
||||
async def _resolve_assignment(
|
||||
|
||||
@@ -5,6 +5,7 @@ Collects and aggregates metrics for reporting and dashboards.
|
||||
Tracks velocity, blockers, completion rates, and agent performance.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any, ClassVar
|
||||
from uuid import UUID
|
||||
@@ -172,6 +173,38 @@ class MetricsService(BaseService):
|
||||
# BLOCKER METRICS
|
||||
# =========================================================================
|
||||
|
||||
async def _blocked_since_map(
|
||||
self, blocked_tasks: Sequence[TaskTable]
|
||||
) -> dict[str, datetime]:
|
||||
"""Per-task ``blocked since`` timestamp from the ``task.blocked`` audit row.
|
||||
|
||||
The real blockage start is the audit transition (#67), indexed on
|
||||
(target_id, event_type, timestamp); the old ``updated_at`` heuristic
|
||||
over-counted when a blocked task was later touched for a non-blocking
|
||||
reason. Returns ``{str(task_id): blocked_at}``; callers fall back to
|
||||
``updated_at or created_at`` for tasks with no audit row.
|
||||
"""
|
||||
blocked_ids = [t.id for t in blocked_tasks]
|
||||
if not blocked_ids:
|
||||
return {}
|
||||
audit_result = await self.session.execute(
|
||||
select(
|
||||
AuditLogTable.target_id,
|
||||
func.max(AuditLogTable.timestamp).label("ts"),
|
||||
)
|
||||
.where(
|
||||
AuditLogTable.event_type == "task.blocked",
|
||||
AuditLogTable.target_type == "task",
|
||||
AuditLogTable.target_id.in_(blocked_ids),
|
||||
)
|
||||
.group_by(AuditLogTable.target_id)
|
||||
)
|
||||
return {
|
||||
str(row.target_id): row.ts
|
||||
for row in audit_result.all()
|
||||
if row.ts is not None
|
||||
}
|
||||
|
||||
async def get_blocker_metrics(self) -> BlockerMetrics:
|
||||
"""Get metrics about blocked tasks."""
|
||||
# Count active blockers
|
||||
@@ -188,6 +221,10 @@ class MetricsService(BaseService):
|
||||
)
|
||||
blocked_tasks = blocked_result.scalars().all()
|
||||
|
||||
# ``blocked since`` from the ``task.blocked`` audit row, falling back to
|
||||
# ``updated_at or created_at`` when no audit row exists (#67).
|
||||
blocked_at = await self._blocked_since_map(blocked_tasks)
|
||||
|
||||
# Calculate average blocked time
|
||||
now = datetime.now(UTC)
|
||||
blocked_hours = []
|
||||
@@ -195,8 +232,9 @@ class MetricsService(BaseService):
|
||||
longest_hours = 0.0
|
||||
|
||||
for task in blocked_tasks:
|
||||
# Assume task got blocked around last update or creation
|
||||
blocked_since = task.updated_at or task.created_at
|
||||
blocked_since = (
|
||||
blocked_at.get(str(task.id)) or task.updated_at or task.created_at
|
||||
)
|
||||
hours = (now - blocked_since).total_seconds() / 3600
|
||||
blocked_hours.append(hours)
|
||||
|
||||
|
||||
@@ -459,6 +459,51 @@ class NotificationService:
|
||||
)
|
||||
return to_agents_uuids
|
||||
|
||||
async def _duplicate_unacked_exists(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
*,
|
||||
from_agent_uuid: UUID,
|
||||
params: CreateNotificationParams,
|
||||
to_agents_uuids: list[UUID],
|
||||
) -> bool:
|
||||
"""True when an unacked same-purpose notification already exists.
|
||||
|
||||
Purpose-based dedup (CEO directive, 2026-06-10): same sender, type,
|
||||
task, overlapping recipients, while a prior one is still unacked —
|
||||
agents re-send the same signal (often reworded) and each copy inflates
|
||||
the recipient's unacked set, soft-blocking i_am_idle and driving respawn
|
||||
churn. Body text is NOT compared. Dedup applies only to ACTION-REQUIRED
|
||||
types; informational carries distinct content per send and acking is
|
||||
voluntary, so deduping them would silently drop broadcasts.
|
||||
"""
|
||||
related = params.related_task_id
|
||||
if not ACK_REQUIRED_BY_TYPE.get(params.notification_type, True):
|
||||
return False
|
||||
dup_q = (
|
||||
select(NotificationTable.id)
|
||||
.where(NotificationTable.from_agent == from_agent_uuid)
|
||||
.where(NotificationTable.type == params.notification_type)
|
||||
.where(NotificationTable.to_agents.overlap(to_agents_uuids))
|
||||
.where(~NotificationTable.acked_by.contains(to_agents_uuids))
|
||||
.where(
|
||||
NotificationTable.related_task_id == related
|
||||
if related is not None
|
||||
else NotificationTable.related_task_id.is_(None)
|
||||
)
|
||||
.limit(1)
|
||||
)
|
||||
if await db.scalar(dup_q) is not None:
|
||||
logger.info(
|
||||
"Suppressed duplicate notification (same purpose, unacked)",
|
||||
from_agent=str(from_agent_uuid),
|
||||
type=params.notification_type.value,
|
||||
related_task_id=str(related) if related is not None else None,
|
||||
to_agents=[str(a) for a in to_agents_uuids],
|
||||
)
|
||||
return True
|
||||
return False
|
||||
|
||||
async def _create_notification(self, params: CreateNotificationParams) -> None:
|
||||
"""Create a notification via the database and deliver it."""
|
||||
async with get_db_context() as db:
|
||||
@@ -503,44 +548,17 @@ class NotificationService:
|
||||
to_agents=[str(a) for a in to_agents_uuids],
|
||||
)
|
||||
return
|
||||
# Purpose-based dedup (CEO directive, 2026-06-10): do NOT create a
|
||||
# second notification for the SAME purpose — same sender, same type,
|
||||
# same task, overlapping recipients — while a prior one is still
|
||||
# unacknowledged. Agents loop and re-send the same signal (often
|
||||
# reworded); each copy inflates the recipient's unacked set, which
|
||||
# soft-blocks their i_am_idle and drives respawn churn. A different
|
||||
# type, a different task, a different sender, or a recipient who has
|
||||
# already acked all go through. Body text is NOT compared, so
|
||||
# rewording cannot defeat the guard.
|
||||
#
|
||||
# Dedup only applies to ACTION-REQUIRED types; informational types
|
||||
# carry distinct content per send and acking is voluntary, so
|
||||
# deduping them would silently drop broadcasts.
|
||||
related = params.related_task_id
|
||||
is_ack_required = ACK_REQUIRED_BY_TYPE.get(params.notification_type, True)
|
||||
if is_ack_required:
|
||||
dup_q = (
|
||||
select(NotificationTable.id)
|
||||
.where(NotificationTable.from_agent == from_agent_uuid)
|
||||
.where(NotificationTable.type == params.notification_type)
|
||||
.where(NotificationTable.to_agents.overlap(to_agents_uuids))
|
||||
.where(~NotificationTable.acked_by.contains(to_agents_uuids))
|
||||
.where(
|
||||
NotificationTable.related_task_id == related
|
||||
if related is not None
|
||||
else NotificationTable.related_task_id.is_(None)
|
||||
)
|
||||
.limit(1)
|
||||
)
|
||||
if await db.scalar(dup_q) is not None:
|
||||
logger.info(
|
||||
"Suppressed duplicate notification (same purpose, unacked)",
|
||||
from_agent=str(from_agent_uuid),
|
||||
type=params.notification_type.value,
|
||||
related_task_id=str(related) if related is not None else None,
|
||||
to_agents=[str(a) for a in to_agents_uuids],
|
||||
)
|
||||
return
|
||||
# Purpose-based dedup (CEO directive, 2026-06-10): suppress a second
|
||||
# notification for the SAME purpose while a prior one is unacked. See
|
||||
# ``_duplicate_unacked_exists`` for the rationale + the action-only
|
||||
# scope (informational types carry distinct content per send).
|
||||
if await self._duplicate_unacked_exists(
|
||||
db,
|
||||
from_agent_uuid=from_agent_uuid,
|
||||
params=params,
|
||||
to_agents_uuids=to_agents_uuids,
|
||||
):
|
||||
return
|
||||
notification = NotificationTable(
|
||||
type=params.notification_type,
|
||||
priority=params.priority,
|
||||
|
||||
@@ -446,11 +446,18 @@ class NotificationDeliveryService(BaseService):
|
||||
|
||||
await self.session.flush()
|
||||
|
||||
# Publish ACK event
|
||||
# Defer the ACK event to the session's after_commit (mirror ``deliver``):
|
||||
# the row state above is only flushed, not durable, so firing the bus
|
||||
# event now would publish an ACK for an acknowledgement that a rollback
|
||||
# can still drop. The event is dropped on rollback (no phantom) and fired
|
||||
# once the row is durable. Best-effort: a bus-init failure is logged but
|
||||
# never propagates — the ack row state is already flushed and the bus is
|
||||
# a secondary channel (the row is the durable store).
|
||||
try:
|
||||
bus = get_event_bus()
|
||||
if bus.is_connected():
|
||||
await bus.publish(
|
||||
defer_bus_publish(
|
||||
self.session,
|
||||
Event(
|
||||
type=EventType.NOTIFICATION_ACKED,
|
||||
data={
|
||||
@@ -458,10 +465,14 @@ class NotificationDeliveryService(BaseService):
|
||||
"agent_id": str(agent_id),
|
||||
"ack_type": ack_type,
|
||||
},
|
||||
)
|
||||
),
|
||||
)
|
||||
except Exception as e:
|
||||
self.log.warning("Failed to publish ACK event", error=str(e))
|
||||
self.log.warning(
|
||||
"Failed to defer ACK bus publish",
|
||||
notification_id=str(notification_id),
|
||||
error=e.__class__.__name__,
|
||||
)
|
||||
|
||||
self.log.info(
|
||||
"Notification acknowledged",
|
||||
|
||||
@@ -1069,13 +1069,13 @@ class OptimalService:
|
||||
if result is not None and not result.success:
|
||||
raise RuntimeError(f"Failed to record learning: {result.error}")
|
||||
|
||||
# Track in database
|
||||
import hashlib
|
||||
|
||||
content_hash = hashlib.md5(
|
||||
params.content.encode(), usedforsecurity=False
|
||||
).hexdigest()[:12]
|
||||
source = f"roboco://learnings/learn-{content_hash}"
|
||||
# Track in database. The tracking row's ``source`` MUST match the URI
|
||||
# the plugin embedded the chunks under (``roboco://learnings/{doc_id}``,
|
||||
# doc_id = ``lrn-{md5(content[:100])[:12]}``) so a later de-index /
|
||||
# lookup-by-source against the tracking row finds the chunk rows. The
|
||||
# plugin already returned that doc_id — reuse it instead of recomputing
|
||||
# a divergent ``learn-{md5(full_content)}`` that orphans the chunk rows.
|
||||
source = f"roboco://learnings/{doc_id}"
|
||||
await self._track_indexed_document(
|
||||
IndexType.LEARNINGS,
|
||||
source=source,
|
||||
|
||||
@@ -264,6 +264,18 @@ class VectorStore:
|
||||
for chunk in chunks
|
||||
if chunk.embedding is not None
|
||||
]
|
||||
# #181: an empty ``chunks`` list is a deliberate clear (matches the
|
||||
# prior delete-then-no-op-add behavior). But chunks passed with NO
|
||||
# usable embedding is an embedder failure — wiping the source's existing
|
||||
# rows on a failed embed would lose good index rows for nothing. No-op
|
||||
# there (distinct from the deliberate empty-list clear below).
|
||||
if chunks and not records:
|
||||
logger.warning(
|
||||
"replace_chunks: every chunk lacked an embedding (embedder "
|
||||
"failure?); skipping wipe to preserve existing rows",
|
||||
extra={"source": source, "chunk_count": len(chunks)},
|
||||
)
|
||||
return
|
||||
pool = self._require_pool()
|
||||
# One acquire, one transaction: the DELETE and INSERT share a single
|
||||
# connection and commit together (or roll back together on failure).
|
||||
|
||||
@@ -9,8 +9,12 @@ lifecycle unchanged.
|
||||
Partial-failure note: GitHub repo creation is an external side effect that
|
||||
cannot be rolled back with the DB transaction. If provisioning fails partway,
|
||||
the DB writes roll back (the route does not commit) but any repos already
|
||||
created on GitHub remain; re-approval will collide on the repo name. The CEO
|
||||
resolves such a rare case manually.
|
||||
created on GitHub remain. Re-approval is idempotent end-to-end: a committed
|
||||
Project/Product is reused by slug (no re-create), and a GitHub repo left
|
||||
orphaned by a rolled-back prior attempt is reused by name —
|
||||
``GitHubProvisioningService.create_repo`` treats GitHub's 422 "name already
|
||||
exists" as "fetch and return the existing repo" instead of erroring (#83/#84).
|
||||
A re-approval therefore never collides; the CEO need only re-approve.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -129,8 +129,8 @@ class PlaybookService(BaseService):
|
||||
resource_type="playbook",
|
||||
)
|
||||
playbook.status = PlaybookStatus.ARCHIVED.value
|
||||
playbook.approved_by = approver_id
|
||||
playbook.approved_at = datetime.now(UTC)
|
||||
playbook.archived_by = approver_id
|
||||
playbook.archived_at = datetime.now(UTC)
|
||||
await self.session.flush()
|
||||
self.log.info("Playbook archived", playbook_id=str(playbook_id))
|
||||
return playbook
|
||||
@@ -186,8 +186,8 @@ class PlaybookService(BaseService):
|
||||
resource_type="playbook",
|
||||
)
|
||||
playbook.status = PlaybookStatus.ARCHIVED.value
|
||||
playbook.approved_by = approver_id
|
||||
playbook.approved_at = datetime.now(UTC)
|
||||
playbook.archived_by = approver_id
|
||||
playbook.archived_at = datetime.now(UTC)
|
||||
await self.session.flush()
|
||||
self.log.info("Playbook rejected", playbook_id=str(playbook_id), reason=reason)
|
||||
return playbook
|
||||
|
||||
@@ -202,10 +202,11 @@ class ProjectService(BaseService):
|
||||
self.log.error("Failed to encrypt git token", error=str(e))
|
||||
raise
|
||||
|
||||
# Apply updates for non-None fields (excluding git_token which we handled)
|
||||
update_data = data.model_dump(
|
||||
exclude_unset=True, exclude_none=True, exclude={"git_token"}
|
||||
)
|
||||
# Apply updates for explicitly-set fields (excluding git_token which we
|
||||
# handled above). exclude_unset keeps UNSET fields out; we do NOT also
|
||||
# exclude_none, so a field the caller explicitly set to None clears the
|
||||
# stored value (distinct from unset = leave unchanged) — #197.
|
||||
update_data = data.model_dump(exclude_unset=True, exclude={"git_token"})
|
||||
for key, value in update_data.items():
|
||||
if hasattr(project, key):
|
||||
setattr(project, key, value)
|
||||
|
||||
+103
-26
@@ -21,7 +21,11 @@ from sqlalchemy import select
|
||||
|
||||
from roboco.db.tables import AgentTable, TaskTable
|
||||
from roboco.foundation.identity import CELL_TEAMS
|
||||
from roboco.foundation.policy.batch import is_batch_umbrella, main_pm_cannot_own_code
|
||||
from roboco.foundation.policy.batch import (
|
||||
is_batch_umbrella,
|
||||
main_pm_cannot_own_code,
|
||||
pm_cannot_own_code,
|
||||
)
|
||||
from roboco.foundation.policy.content.validators import coerce_str_list
|
||||
from roboco.foundation.policy.sequencing.models import DraftSurface, SequencePlan
|
||||
from roboco.models.base import (
|
||||
@@ -179,13 +183,20 @@ class PrompterService:
|
||||
team=board until approved (else the CEO's Approve & Start gate, which
|
||||
keys on team=board, never appears and the task strands). "Approve &
|
||||
Start" (assignee main-pm) and the post-approval state are team=main_pm.
|
||||
|
||||
Product/board routing is consulted BEFORE the multi-cell-map Main-PM
|
||||
force: a product draft that also carries a ≥2-cell the_work map is still
|
||||
a product root, and if it is on the board-review path it must stay
|
||||
team=board — forcing Main PM here would strand the task past the CEO's
|
||||
Approve & Start gate (#160).
|
||||
"""
|
||||
if team_override is not None:
|
||||
return team_override
|
||||
if len(_draft_cell_map(draft_data)) >= _MULTI_CELL_MIN:
|
||||
# Ad-hoc multi-cell map → coordination root, like a product root.
|
||||
return Team.MAIN_PM
|
||||
if resolved_product_id is None:
|
||||
# No product: ad-hoc multi-cell map → coordination root; else the
|
||||
# single-cell lead cell.
|
||||
if len(_draft_cell_map(draft_data)) >= _MULTI_CELL_MIN:
|
||||
return Team.MAIN_PM
|
||||
return self._lead_cell_team(draft_data, default=default_lead)
|
||||
if resolved_assigned_to is not None and await self._assignee_is_board(
|
||||
resolved_assigned_to
|
||||
@@ -242,6 +253,43 @@ class PrompterService:
|
||||
return UUID(str(draft_data["assigned_to"]))
|
||||
return None
|
||||
|
||||
def _coerce_pm_code_to_planning(
|
||||
self,
|
||||
*,
|
||||
team: Team,
|
||||
task_type: TaskType,
|
||||
resolved_assigned_to: UUID | None,
|
||||
title: str,
|
||||
) -> TaskType:
|
||||
"""Coerce code->planning when the owner is a coordination role (PM).
|
||||
|
||||
Two layers: team-based (main_pm) and assignee-based (a PM assignee on a
|
||||
cell team). A brand-new intake task is never in needs_revision so the
|
||||
issue-resolution carve-out does not apply.
|
||||
"""
|
||||
if main_pm_cannot_own_code(team=team, task_type=task_type):
|
||||
self.log.info(
|
||||
"Main-PM intake task coerced code->planning",
|
||||
team=str(getattr(team, "value", team)),
|
||||
title=title,
|
||||
)
|
||||
return TaskType.PLANNING
|
||||
if resolved_assigned_to is None:
|
||||
return task_type
|
||||
from roboco.foundation.identity import role_for_uuid_or_none
|
||||
|
||||
assignee_role = role_for_uuid_or_none(resolved_assigned_to)
|
||||
if assignee_role is not None and pm_cannot_own_code(
|
||||
role=assignee_role, task_type=task_type, is_issue_resolution=False
|
||||
):
|
||||
self.log.info(
|
||||
"PM-assignee intake task coerced code->planning",
|
||||
role=str(getattr(assignee_role, "value", assignee_role)),
|
||||
title=title,
|
||||
)
|
||||
return TaskType.PLANNING
|
||||
return task_type
|
||||
|
||||
async def create_task_from_draft(
|
||||
self,
|
||||
draft_data: dict[str, Any],
|
||||
@@ -255,8 +303,9 @@ class PrompterService:
|
||||
|
||||
Recomposes the description, validates exactly-one target, coerces enums,
|
||||
routes the owning team (product → Main PM, project → lead cell), and
|
||||
persists via ``TaskService.create``. Mutates ``draft_data['description']``
|
||||
in place. ``confirmed_by_human=True`` — the CEO confirmed it.
|
||||
persists via ``TaskService.create``. Operates on a copy — the caller's
|
||||
draft dict is never mutated (#59). ``confirmed_by_human=True`` — the
|
||||
CEO confirmed it.
|
||||
|
||||
``status`` defaults to ``BACKLOG``. The live-intake buttons pass
|
||||
``PENDING`` + an ``assigned_to`` (a board agent for "Board review &
|
||||
@@ -272,6 +321,10 @@ class PrompterService:
|
||||
draft through to the task so the analyzer's surface is persisted.
|
||||
"""
|
||||
place = placement or BatchPlacement()
|
||||
# Coerce + recompose on a copy so the caller's draft is never mutated
|
||||
# (#59) — the_work unit dicts are copied too, since coercion rewrites
|
||||
# each unit's ``items``.
|
||||
draft_data = _copy_draft(draft_data)
|
||||
self._validate_and_coerce_draft(draft_data)
|
||||
# Recompose the description from the (possibly edited) structured fields —
|
||||
# the task always carries a freshly-composed, consistent description.
|
||||
@@ -290,9 +343,16 @@ class PrompterService:
|
||||
]
|
||||
resolved_project_id = None
|
||||
resolved_product_id = None
|
||||
elif len(cell_map) == 1:
|
||||
elif (
|
||||
len(cell_map) == 1
|
||||
and resolved_project_id is None
|
||||
and resolved_product_id is None
|
||||
):
|
||||
# A lone the_work cell with no top-level target → single-project
|
||||
# task on that cell's project. A top-level project_id/product_id
|
||||
# wins over a redundant 1-cell map — the explicit target is
|
||||
# preserved, not silently dropped (#57).
|
||||
resolved_project_id = cell_map[0][1]
|
||||
resolved_product_id = None
|
||||
self._validate_draft_target(
|
||||
resolved_project_id,
|
||||
resolved_product_id,
|
||||
@@ -317,22 +377,15 @@ class PrompterService:
|
||||
default_lead=_lead,
|
||||
)
|
||||
|
||||
# A Main PM coordinates — it never owns a code task. A main_pm + code
|
||||
# draft is the structural mismatch behind the 2026-06-27 MegaTask
|
||||
# meltdown (the git/PR/review layer treated the root as code while the
|
||||
# ownership layer treated it as coordination → pr_fail loop). Intake
|
||||
# coerces code -> planning here so the combo can never persist; a
|
||||
# root-subtask / umbrella / single-task main_pm route is a coordination
|
||||
# root whose code ACs live on the delegated cell/dev leaves. The
|
||||
# TaskService.create backstop rejects main_pm + code for non-intake
|
||||
# create paths (the HTTP route).
|
||||
if main_pm_cannot_own_code(team=team, task_type=task_type):
|
||||
self.log.info(
|
||||
"Main-PM intake task coerced code->planning",
|
||||
team=str(getattr(team, "value", team)),
|
||||
title=_text(draft_data.get("title")) or "",
|
||||
)
|
||||
task_type = TaskType.PLANNING
|
||||
# A coordination role (PM) never owns a code task — coerce code->planning
|
||||
# (team-based main_pm path + assignee-based PM path). See
|
||||
# ``_coerce_pm_code_to_planning`` for the 2026-06-27 meltdown rationale.
|
||||
task_type = self._coerce_pm_code_to_planning(
|
||||
team=team,
|
||||
task_type=task_type,
|
||||
resolved_assigned_to=resolved_assigned_to,
|
||||
title=_text(draft_data.get("title")) or "",
|
||||
)
|
||||
|
||||
req = TaskCreateRequest(
|
||||
title=draft_data["title"],
|
||||
@@ -872,8 +925,15 @@ def _draft_cell_map(draft: dict[str, Any]) -> list[tuple[Team, UUID]]:
|
||||
continue
|
||||
try:
|
||||
pid = UUID(str(pid_raw))
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
except (ValueError, TypeError) as exc:
|
||||
# A present-but-malformed project_id is a hard error, not an LLM
|
||||
# cosmetic guess to skip: silently dropping it would collapse a
|
||||
# 2-cell map to 1-cell and mis-route the draft as a single-project
|
||||
# task (#58). Reject with a clean 400 so the human re-enters it.
|
||||
raise ValidationError(
|
||||
message=f"Invalid project_id in the_work cell '{team_raw}': {pid_raw}",
|
||||
field="project_id",
|
||||
) from exc
|
||||
seen_teams.add(team)
|
||||
out.append((team, pid))
|
||||
return out
|
||||
@@ -893,6 +953,23 @@ def _clean_list(value: Any) -> list[str]:
|
||||
return coerce_str_list(value)
|
||||
|
||||
|
||||
def _copy_draft(draft: dict[str, Any]) -> dict[str, Any]:
|
||||
"""A shallow copy of the draft whose ``the_work`` unit dicts are also copied.
|
||||
|
||||
Draft coercion (``_validate_and_coerce_draft``) rewrites the top-level list
|
||||
fields and each the_work unit's ``items``; without copying it would mutate
|
||||
the caller's draft in place (#59). The unit dicts are copied because their
|
||||
``items`` is reassigned; nested scalars are read-only.
|
||||
"""
|
||||
out = dict(draft)
|
||||
work = draft.get("the_work")
|
||||
if isinstance(work, list):
|
||||
out["the_work"] = [
|
||||
dict(unit) if isinstance(unit, dict) else unit for unit in work
|
||||
]
|
||||
return out
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
"""Trimmed string from a possibly-missing scalar field."""
|
||||
return str(value or "").strip()
|
||||
|
||||
@@ -12,6 +12,7 @@ real git / ``make quality`` / ``gh`` work on a writable clone.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -96,7 +97,25 @@ class ReleaseExecutor:
|
||||
detail="make quality failed — aborted before commit (fail-closed).",
|
||||
)
|
||||
|
||||
commit_sha = await self._ops.commit_and_push(version)
|
||||
try:
|
||||
commit_sha = await self._ops.commit_and_push(version)
|
||||
except RuntimeError as exc:
|
||||
# The ops layer raises RuntimeError on a failed add/commit/push
|
||||
# (gpgsign/pre-commit reject/no-op bump/non-fast-forward push). That
|
||||
# is the correct fail-closed abort at the ops layer; the EXECUTOR
|
||||
# turns it into a structured outcome so the CEO sees the cause
|
||||
# instead of a 500 bubbling out of ``approve``.
|
||||
logger.error("release commit/push failed", error=str(exc)[:300])
|
||||
return ReleaseResult(
|
||||
status="commit_failed",
|
||||
version=version,
|
||||
files_changed=files,
|
||||
commit_sha=None,
|
||||
release_url=None,
|
||||
detail=(
|
||||
f"release commit/push failed — not published (fail-closed): {exc}"
|
||||
)[:280],
|
||||
)
|
||||
|
||||
if not await self._ops.wait_for_ci(commit_sha):
|
||||
return ReleaseResult(
|
||||
@@ -108,7 +127,24 @@ class ReleaseExecutor:
|
||||
detail="release-commit CI was not green — not published (fail-closed).",
|
||||
)
|
||||
|
||||
release_url = await self._ops.publish_release(version, report.drafted_changelog)
|
||||
try:
|
||||
release_url = await self._ops.publish_release(
|
||||
version, report.drafted_changelog
|
||||
)
|
||||
except RuntimeError as exc:
|
||||
# ``gh release create`` failed (auth/quota/network). The commit is
|
||||
# already pushed and CI is green, so the release is half-landed —
|
||||
# surface it as a structured outcome (not a 500) so the CEO can
|
||||
# retry ``gh release create`` for the same version.
|
||||
logger.error("release publish failed", error=str(exc)[:300])
|
||||
return ReleaseResult(
|
||||
status="publish_failed",
|
||||
version=version,
|
||||
files_changed=files,
|
||||
commit_sha=commit_sha,
|
||||
release_url=None,
|
||||
detail=f"gh release create failed — not published (fail-closed): {exc}",
|
||||
)
|
||||
logger.info(
|
||||
"release published",
|
||||
version=version,
|
||||
@@ -155,7 +191,12 @@ async def _await_proc(
|
||||
try:
|
||||
out, _ = await asyncio.wait_for(proc.communicate(), timeout=timeout)
|
||||
except TimeoutError:
|
||||
proc.kill()
|
||||
# Reap the killed child so its PID/PGID and the pipe transport are
|
||||
# released — a kill without a wait leaves a zombie that wedges the
|
||||
# release loop's next clone (and leaks FDs over a long release session).
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
proc.kill() # already-exited between the timeout and the kill is fine
|
||||
await proc.wait()
|
||||
return _TIMEOUT_RC, f"subprocess timed out after {int(timeout)}s"
|
||||
return proc.returncode or 0, out.decode("utf-8", "replace")
|
||||
|
||||
|
||||
@@ -11,8 +11,10 @@ executor is fail-closed and the proposal stays open unless a publish succeeds.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
from uuid import uuid4
|
||||
|
||||
import redis.asyncio as redis
|
||||
|
||||
@@ -33,10 +35,38 @@ if TYPE_CHECKING:
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ReleaseLockUnavailable(Exception):
|
||||
"""Redis is unreachable so the release mutex can't be acquired.
|
||||
|
||||
Distinct from "the lock is held": a held lock is a concurrent approve (retry
|
||||
later); an unavailable Redis is an infrastructure failure (fix Redis, then
|
||||
retry). Both stay fail-closed — the execute never runs without the mutex.
|
||||
"""
|
||||
|
||||
|
||||
# Redis mutex guarding the ~40min release execute against concurrent
|
||||
# approves; TTL backstops a crash, lock is released on completion.
|
||||
# approves. The TTL only backstops a crash; a background heartbeat refreshes
|
||||
# it while the execute owns the lock, and a fencing token makes the release
|
||||
# compare-and-del so a late first-finally can't delete a usurper's lock.
|
||||
_RELEASE_LOCK_PREFIX = "roboco:release_proposal:"
|
||||
_RELEASE_LOCK_TTL_SECONDS = 3000 # 50 min > 40 min CI ceiling
|
||||
_RELEASE_LOCK_TTL_SECONDS = 3000 # 50 min > 40 min CI ceiling; crash backstop
|
||||
_RELEASE_LOCK_HEARTBEAT_SECONDS = 60.0
|
||||
# Only delete/extend the lock when its value still equals our fencing token.
|
||||
_RELEASE_LOCK_RELEASE_SCRIPT = """
|
||||
if redis.call("get", KEYS[1]) == ARGV[1] then
|
||||
return redis.call("del", KEYS[1])
|
||||
else
|
||||
return 0
|
||||
end
|
||||
"""
|
||||
_RELEASE_LOCK_HEARTBEAT_SCRIPT = """
|
||||
if redis.call("get", KEYS[1]) == ARGV[1] then
|
||||
return redis.call("expire", KEYS[1], ARGV[2])
|
||||
else
|
||||
return 0
|
||||
end
|
||||
"""
|
||||
|
||||
|
||||
class ReleaseProposalService(BaseService):
|
||||
@@ -57,12 +87,16 @@ class ReleaseProposalService(BaseService):
|
||||
when the release actually publishes — a gate/CI failure leaves it open so
|
||||
the CEO can retry after the cause is fixed.
|
||||
|
||||
F013: a Redis ``SET NX`` mutex keyed by the proposal id guards the
|
||||
~40min execute against concurrent approves (double-click / panel retry)
|
||||
that would race on the shared, ``rm -rf``'d writable release clone. A
|
||||
second approve while the lock is held returns ``already_in_progress``
|
||||
without running the executor. Fail-closed on Redis outage — a release is
|
||||
rare and CEO-gated, and the race it prevents corrupts the release.
|
||||
A Redis ``SET NX`` mutex keyed by the proposal id guards the execute
|
||||
against concurrent approves (double-click / panel retry) that would race
|
||||
on the shared, ``rm -rf``'d writable release clone. The lock value is a
|
||||
fencing token; a background heartbeat refreshes the TTL while the
|
||||
execute owns it, and the release is a compare-and-del — so a second
|
||||
approve can't acquire mid-execute (TTL never expires while it's alive)
|
||||
and a late first-finally can't delete a usurper's lock. A second approve
|
||||
while the lock is held returns ``already_in_progress`` without running
|
||||
the executor. Fail-closed on Redis outage — a release is rare and
|
||||
CEO-gated, and the race it prevents corrupts the release.
|
||||
"""
|
||||
task = await get_task_service(self.session).get(task_id)
|
||||
if task is None or task.source != RELEASE_MANAGER_SOURCE:
|
||||
@@ -72,9 +106,27 @@ class ReleaseProposalService(BaseService):
|
||||
return None
|
||||
|
||||
lock_key = f"{_RELEASE_LOCK_PREFIX}{task_id}"
|
||||
lock = await self._acquire_release_lock(lock_key)
|
||||
if lock is None:
|
||||
report = report_from_dict(report_dict)
|
||||
report = report_from_dict(report_dict)
|
||||
try:
|
||||
lock_token = await self._acquire_release_lock(lock_key)
|
||||
except ReleaseLockUnavailable as exc:
|
||||
# Fail-closed: the execute never runs without the mutex. But a Redis
|
||||
# outage is NOT a concurrent approve — surface the real cause so the
|
||||
# CEO fixes Redis instead of waiting on a phantom second approve.
|
||||
logger.error("release lock unavailable (redis down): %s", exc)
|
||||
return ReleaseResult(
|
||||
status="redis_unavailable",
|
||||
version=report.proposed_version,
|
||||
files_changed=[],
|
||||
commit_sha=None,
|
||||
release_url=None,
|
||||
detail=(
|
||||
"Redis is unavailable so the release mutex can't be acquired"
|
||||
" (fail-closed: the execute did not run). Restore Redis and"
|
||||
" retry — this is not a concurrent-approve conflict."
|
||||
),
|
||||
)
|
||||
if lock_token is None:
|
||||
return ReleaseResult(
|
||||
status="already_in_progress",
|
||||
version=report.proposed_version,
|
||||
@@ -87,45 +139,93 @@ class ReleaseProposalService(BaseService):
|
||||
),
|
||||
)
|
||||
|
||||
heartbeat_task: asyncio.Task[None] | None = None
|
||||
try:
|
||||
report = report_from_dict(report_dict)
|
||||
executor = await get_release_executor(self.session)
|
||||
heartbeat_task = asyncio.create_task(
|
||||
self._heartbeat_loop(lock_key, lock_token)
|
||||
)
|
||||
result = await executor.execute(report)
|
||||
if result.status == "published":
|
||||
task.status = TaskStatus.COMPLETED
|
||||
await self.session.flush()
|
||||
return result
|
||||
finally:
|
||||
await self._release_release_lock(lock_key)
|
||||
if heartbeat_task is not None:
|
||||
heartbeat_task.cancel()
|
||||
await asyncio.gather(heartbeat_task, return_exceptions=True)
|
||||
await self._release_release_lock(lock_key, lock_token)
|
||||
|
||||
async def _acquire_release_lock(self, lock_key: str) -> bool | None:
|
||||
"""``SET NX EX`` the release mutex. Returns True if acquired, None if held
|
||||
or Redis is unavailable (fail-closed → treat as held)."""
|
||||
async def _acquire_release_lock(self, lock_key: str) -> str | None:
|
||||
"""``SET NX EX`` the release mutex with a fencing-token value.
|
||||
|
||||
Returns the token if acquired, None if held (a concurrent approve owns
|
||||
it). Raises :class:`ReleaseLockUnavailable` if Redis itself is
|
||||
unreachable so the caller can distinguish an infra failure from a
|
||||
concurrent-approve conflict (both stay fail-closed).
|
||||
"""
|
||||
token = uuid4().hex
|
||||
try:
|
||||
conn = redis.from_url(settings.redis_url)
|
||||
try:
|
||||
acquired = await conn.set(
|
||||
lock_key, "release", nx=True, ex=_RELEASE_LOCK_TTL_SECONDS
|
||||
lock_key, token, nx=True, ex=_RELEASE_LOCK_TTL_SECONDS
|
||||
)
|
||||
# redis-py returns True on SET NX success, None on conflict.
|
||||
return True if acquired else None
|
||||
return token if acquired else None
|
||||
finally:
|
||||
await conn.aclose()
|
||||
except Exception as exc:
|
||||
logger.warning("release lock acquire failed (redis): %s", exc)
|
||||
return None
|
||||
raise ReleaseLockUnavailable(str(exc)) from exc
|
||||
|
||||
async def _release_release_lock(self, lock_key: str) -> None:
|
||||
"""Best-effort ``DEL`` the release mutex (the TTL is the backstop)."""
|
||||
async def _release_release_lock(self, lock_key: str, token: str) -> None:
|
||||
"""Compare-and-del the release mutex (only if we still own it)."""
|
||||
try:
|
||||
conn = redis.from_url(settings.redis_url)
|
||||
try:
|
||||
await conn.delete(lock_key)
|
||||
await conn.eval(_RELEASE_LOCK_RELEASE_SCRIPT, 1, lock_key, token)
|
||||
finally:
|
||||
await conn.aclose()
|
||||
except Exception as exc:
|
||||
logger.warning("release lock release failed (redis): %s", exc)
|
||||
|
||||
async def _heartbeat_release_lock(self, lock_key: str, token: str) -> bool:
|
||||
"""Compare-and-expire the release mutex. True if we still own it."""
|
||||
conn = redis.from_url(settings.redis_url)
|
||||
try:
|
||||
res = await conn.eval(
|
||||
_RELEASE_LOCK_HEARTBEAT_SCRIPT,
|
||||
1,
|
||||
lock_key,
|
||||
token,
|
||||
_RELEASE_LOCK_TTL_SECONDS,
|
||||
)
|
||||
return bool(res)
|
||||
finally:
|
||||
await conn.aclose()
|
||||
|
||||
async def _heartbeat_loop(self, lock_key: str, token: str) -> None:
|
||||
"""Refresh the lock TTL while the execute owns it.
|
||||
|
||||
Refreshes before the first sleep so a fast execute still extends the
|
||||
TTL. A refresh error logs and continues (never crashes the execute); if
|
||||
the lock is no longer ours (returned 0) we stop — the TTL backstop and
|
||||
the fencing token still hold the line.
|
||||
"""
|
||||
while True:
|
||||
try:
|
||||
if not await self._heartbeat_release_lock(lock_key, token):
|
||||
logger.critical(
|
||||
"release lock no longer owned during execute — "
|
||||
"TTL backstop active; a concurrent approve was refused "
|
||||
"by the fencing token"
|
||||
)
|
||||
return
|
||||
except Exception as exc:
|
||||
logger.warning("release lock heartbeat failed (redis): %s", exc)
|
||||
await asyncio.sleep(_RELEASE_LOCK_HEARTBEAT_SECONDS)
|
||||
|
||||
async def reject(self, task_id: UUID, required_changes: str) -> TaskTable | None:
|
||||
"""Record the CEO's required changes; keep the proposal held for revision."""
|
||||
task = await get_task_service(self.session).get(task_id)
|
||||
|
||||
@@ -441,6 +441,15 @@ def _canonical_bump_files(root: Path, version: str) -> list[str]:
|
||||
# First release has no prior ``chore(release):`` commit, so derivation
|
||||
# returns [] — fall back to the version-reference scan: files embedding the
|
||||
# version are exactly the set a first release must bump. Read-only.
|
||||
#
|
||||
# This makes ``_version_ref_gaps`` empty on first release BY DESIGN: the
|
||||
# bump plan IS every version-bearing file, so the completeness invariant
|
||||
# (every version-bearing file is in the plan) holds vacuously — there is no
|
||||
# file that "holds the version but won't be bumped." The CEO still sees the
|
||||
# set via ``version_bump_plan`` in the report. Re-introducing the old
|
||||
# ``return []`` would flag EVERY version-bearing file as a gap on first
|
||||
# release (false alarms). Subsequent releases derive the plan from the last
|
||||
# release commit, so a NEWLY version-bearing file correctly shows as a gap.
|
||||
return _tracked_files_with_version(root, version)
|
||||
|
||||
|
||||
|
||||
@@ -67,6 +67,9 @@ def _fingerprint(signal_name: str) -> str:
|
||||
return hashlib.sha256(signal_name.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
_NOTIFY_DEDUPE_KEY_PREFIX = "self_heal:notified:"
|
||||
|
||||
|
||||
class SelfHealEngine(BaseService):
|
||||
"""Detect a regression in RoboCo's own repo; surface it (and later open a fix)."""
|
||||
|
||||
@@ -105,24 +108,105 @@ class SelfHealEngine(BaseService):
|
||||
new regression and STOPS. It never starts, approves, merges, or deploys.
|
||||
Writes (any opened task) are flushed here; the caller (the orchestrator
|
||||
loop) owns the commit.
|
||||
|
||||
#43: the notify loop dedupes per fingerprint — a regression that stays
|
||||
red across cycles pings the CEO once per episode, not every tick (the
|
||||
notification layer's purpose-dedup only holds while unacked; without
|
||||
this guard a post-ack persistent red state re-fires each cycle). The
|
||||
CEO alert links the open self-heal fix task when one exists so the panel
|
||||
can route the CEO to the fix. The dedupe check fails open: a Redis
|
||||
outage still lets the notify through (better a duplicate ping than a
|
||||
swallowed regression).
|
||||
"""
|
||||
if not settings.self_heal_enabled:
|
||||
return []
|
||||
observations = await self.assess()
|
||||
if not observations:
|
||||
return []
|
||||
fp_to_task = await self._open_self_heal_task_ids_by_fp()
|
||||
notifier = NotificationService()
|
||||
for obs in observations:
|
||||
if await self._already_notified(obs.fingerprint):
|
||||
continue
|
||||
body = f"[self-heal] {obs.summary}\n\n{obs.detail}"
|
||||
if obs.raw_ref:
|
||||
body += f"\n\nEvidence: {obs.raw_ref}"
|
||||
await notifier.send_ack_notification(
|
||||
from_agent="system", to_agent="ceo", body=body
|
||||
from_agent="system",
|
||||
to_agent="ceo",
|
||||
body=body,
|
||||
task_id=fp_to_task.get(obs.fingerprint),
|
||||
)
|
||||
await self._mark_notified(obs.fingerprint)
|
||||
if settings.self_heal_originate_enabled:
|
||||
await self._originate(observations)
|
||||
return observations
|
||||
|
||||
async def _open_self_heal_task_ids_by_fp(self) -> dict[str, UUID]:
|
||||
"""Map each open self-heal task's fingerprint to its task id.
|
||||
|
||||
Best-effort: a DB/query error returns ``{}`` so the notify loop still
|
||||
runs (task_id simply stays None — the alert floats free, which is the
|
||||
pre-fix behaviour, never a crash). Used both to link the CEO alert to
|
||||
the fix task and as a durable "already notified" corroboration.
|
||||
"""
|
||||
try:
|
||||
task_svc = get_task_service(self.session)
|
||||
open_tasks = await task_svc.list_open_self_heal_tasks()
|
||||
except Exception:
|
||||
self.log.exception("self-heal open-task lookup failed; notify floats")
|
||||
return {}
|
||||
mapping: dict[str, UUID] = {}
|
||||
for existing in open_tasks:
|
||||
fp = extract_self_heal_fingerprint(existing)
|
||||
if fp and existing.id is not None:
|
||||
mapping[fp] = cast("UUID", existing.id)
|
||||
return mapping
|
||||
|
||||
async def _already_notified(self, fingerprint: str) -> bool:
|
||||
"""True if this fingerprint was already CEO-notified this episode.
|
||||
|
||||
Fail-open: a Redis error returns False (notify anyway) — a dedupe
|
||||
outage must never swallow a regression alert.
|
||||
"""
|
||||
try:
|
||||
import redis.asyncio # local: redis is an agent-runtime dep
|
||||
|
||||
r = redis.asyncio.from_url(settings.redis_url)
|
||||
try:
|
||||
return bool(await r.get(self._dedupe_key(fingerprint)))
|
||||
finally:
|
||||
await r.aclose()
|
||||
except Exception:
|
||||
self.log.exception("self-heal notify-dedupe check failed; failing open")
|
||||
return False
|
||||
|
||||
async def _mark_notified(self, fingerprint: str) -> None:
|
||||
"""Record that this fingerprint was CEO-notified, with a TTL window.
|
||||
|
||||
Best-effort: a Redis error is logged and swallowed — the notify already
|
||||
fired, so a missed mark can at worst cause one duplicate on the next
|
||||
cycle, never a lost alert.
|
||||
"""
|
||||
try:
|
||||
import redis.asyncio # local: redis is an agent-runtime dep
|
||||
|
||||
r = redis.asyncio.from_url(settings.redis_url)
|
||||
try:
|
||||
await r.set(
|
||||
self._dedupe_key(fingerprint),
|
||||
"1",
|
||||
ex=settings.self_heal_notify_dedupe_seconds,
|
||||
)
|
||||
finally:
|
||||
await r.aclose()
|
||||
except Exception:
|
||||
self.log.exception("self-heal notify-dedupe mark dropped")
|
||||
|
||||
@staticmethod
|
||||
def _dedupe_key(fingerprint: str) -> str:
|
||||
return f"{_NOTIFY_DEDUPE_KEY_PREFIX}{fingerprint}"
|
||||
|
||||
async def _originate(self, observations: list[RegressionObservation]) -> int:
|
||||
"""Open a PENDING fix task per NEW regression, then STOP. Returns count.
|
||||
|
||||
|
||||
@@ -230,6 +230,32 @@ def _surfaced_siblings(siblings: list) -> list:
|
||||
]
|
||||
|
||||
|
||||
def _same_assignee_lane_edges(siblings: list) -> list[tuple[object, object]]:
|
||||
"""Undeclared-surface fallback: chain each (project, assignee) lane by
|
||||
``(priority, sequence)`` so same-assignee same-repo siblings share a working
|
||||
tree without an out-of-order merge conflict. Same-assignee scoped so
|
||||
cross-dev parallel work is untouched; the edge survives reassignment. Same
|
||||
stable sort → re-runs only add edges, never flip a pair into a reverse.
|
||||
"""
|
||||
lanes: dict[tuple[str, object], list] = defaultdict(list)
|
||||
for s in siblings:
|
||||
proj = getattr(s, "project_id", None)
|
||||
owner = getattr(s, "assigned_to", None)
|
||||
if proj is not None and owner is not None:
|
||||
lanes[(str(proj), owner)].append(s)
|
||||
fallback: list[tuple[object, object]] = []
|
||||
for members in lanes.values():
|
||||
members.sort(
|
||||
key=lambda s: (
|
||||
int(getattr(s, "priority", 2)),
|
||||
int(getattr(s, "sequence", 0)),
|
||||
)
|
||||
)
|
||||
for prev, cur in pairwise(members):
|
||||
fallback.append((prev.id, cur.id))
|
||||
return fallback
|
||||
|
||||
|
||||
def dev_task_collision_edges(siblings: list) -> list[tuple[object, object]]:
|
||||
"""Wire the dev-task collision DAG for a parent's surfaced siblings.
|
||||
|
||||
@@ -285,28 +311,10 @@ def dev_task_collision_edges(siblings: list) -> list[tuple[object, object]]:
|
||||
if edges:
|
||||
return edges
|
||||
|
||||
# Undeclared-surface fallback: same-assignee same-repo siblings share a
|
||||
# working tree, so chain each (project, assignee) lane by (priority,
|
||||
# sequence) to avoid an out-of-order merge conflict. Same-assignee scoped so
|
||||
# cross-dev parallel work is untouched; the edge survives reassignment.
|
||||
# Only fires with zero collision edges; same stable sort -> re-runs only add.
|
||||
lanes: dict[tuple[str, object], list] = defaultdict(list)
|
||||
for s in siblings:
|
||||
proj = getattr(s, "project_id", None)
|
||||
owner = getattr(s, "assigned_to", None)
|
||||
if proj is not None and owner is not None:
|
||||
lanes[(str(proj), owner)].append(s)
|
||||
fallback: list[tuple[object, object]] = []
|
||||
for members in lanes.values():
|
||||
members.sort(
|
||||
key=lambda s: (
|
||||
int(getattr(s, "priority", 2)),
|
||||
int(getattr(s, "sequence", 0)),
|
||||
)
|
||||
)
|
||||
for prev, cur in pairwise(members):
|
||||
fallback.append((prev.id, cur.id))
|
||||
return fallback
|
||||
# Undeclared-surface fallback (zero collision edges): chain same-assignee
|
||||
# same-repo lanes so they don't merge out-of-order. See
|
||||
# ``_same_assignee_lane_edges`` for the rationale + re-run idempotency.
|
||||
return _same_assignee_lane_edges(siblings)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -60,6 +60,7 @@ FEATURE_FLAGS: tuple[tuple[str, str], ...] = (
|
||||
("dep_update_enabled", "Dependency-update bot"),
|
||||
("release_manager_enabled", "Gated release manager"),
|
||||
("org_memory_enabled", "Organizational memory loop"),
|
||||
("routing_strict", "Strict model routing (fail-closed on a disabled provider)"),
|
||||
)
|
||||
_FEATURE_FLAG_KEYS = tuple(key for key, _ in FEATURE_FLAGS)
|
||||
|
||||
|
||||
+264
-58
@@ -41,6 +41,7 @@ from roboco.foundation.policy.batch import (
|
||||
is_branchless_coordination,
|
||||
is_valid_batch_shape,
|
||||
main_pm_cannot_own_code,
|
||||
pm_cannot_own_code,
|
||||
)
|
||||
from roboco.foundation.policy.content import markers
|
||||
from roboco.foundation.policy.content.validators import ContentValidationError
|
||||
@@ -50,6 +51,7 @@ from roboco.models.base import (
|
||||
BlockerResolverType,
|
||||
Complexity,
|
||||
JournalEntryType,
|
||||
NotificationPriority,
|
||||
TaskNature,
|
||||
TaskStatus,
|
||||
TaskType,
|
||||
@@ -101,6 +103,31 @@ _ROLE_CLAIM_STATUSES: dict[str, set[TaskStatus]] = {
|
||||
},
|
||||
}
|
||||
|
||||
# Statuses a task may be escalated FROM into BLOCKED. The strict transition
|
||||
# validator only allows IN_PROGRESS→BLOCKED, but a chain escalation legitimately
|
||||
# blocks a task from any active work state (and re-escalates an already-blocked
|
||||
# task to a new target). BACKLOG is excluded — a never-activated task has no
|
||||
# business being escalated to BLOCKED (no spec edge, nonsensical). Terminal
|
||||
# statuses are refused separately by ``_is_terminal_task``. This is the explicit
|
||||
# escalation exemption to the validator: an enumerated source set, not an
|
||||
# arbitrary source→BLOCKED write.
|
||||
_ESCALATABLE_TO_BLOCKED: frozenset[TaskStatus] = frozenset(
|
||||
{
|
||||
TaskStatus.PENDING,
|
||||
TaskStatus.CLAIMED,
|
||||
TaskStatus.IN_PROGRESS,
|
||||
TaskStatus.VERIFYING,
|
||||
TaskStatus.AWAITING_QA,
|
||||
TaskStatus.NEEDS_REVISION,
|
||||
TaskStatus.AWAITING_DOCUMENTATION,
|
||||
TaskStatus.AWAITING_PR_REVIEW,
|
||||
TaskStatus.AWAITING_PM_REVIEW,
|
||||
TaskStatus.AWAITING_CEO_APPROVAL,
|
||||
TaskStatus.PAUSED,
|
||||
TaskStatus.BLOCKED,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# Board / advisory roles review and advise; they never own or execute a
|
||||
# descendant code task. Handing one to them (e.g. via the main_pm→product_owner
|
||||
@@ -540,6 +567,14 @@ class TaskService(BaseService):
|
||||
|
||||
service_name: ClassVar[str] = "task"
|
||||
_background_tasks: ClassVar[set[asyncio.Task[None]]] = set()
|
||||
# #216: consecutive FS/permission failures during terminal worktree
|
||||
# cleanup. A systemic FS issue (stuck mount, perms) would otherwise leak
|
||||
# per-task worktrees indefinitely with only a warning log; after this many
|
||||
# consecutive OSError failures a CEO alert fires once (suppressed until a
|
||||
# success resets the streak). Class-level so it spans all cleanup calls.
|
||||
_worktree_cleanup_fail_streak: ClassVar[int] = 0
|
||||
_worktree_cleanup_escalated: ClassVar[bool] = False
|
||||
_WORKTREE_CLEANUP_ESCALATE_AFTER: ClassVar[int] = 3
|
||||
|
||||
# =========================================================================
|
||||
# STATUS TRANSITION HELPER
|
||||
@@ -698,7 +733,9 @@ class TaskService(BaseService):
|
||||
# Rework counter: a bounce INTO needs_revision (not a re-entry) is one
|
||||
# rework cycle. Incremented at this single chokepoint — every transition
|
||||
# path funnels its audit through here exactly once — so the rework rate
|
||||
# is an O(1) column read. Synchronous (part of this unit of work).
|
||||
# is an O(1) column read. Synchronous (part of this unit of work). The
|
||||
# pre-block RESTORE path undoes this bump when it restores to a
|
||||
# snapshotted needs_revision (same cycle resuming, not a new rejection).
|
||||
if (
|
||||
to_status == TaskStatus.NEEDS_REVISION.value
|
||||
and from_status != TaskStatus.NEEDS_REVISION.value
|
||||
@@ -719,7 +756,7 @@ class TaskService(BaseService):
|
||||
except (ValueError, AttributeError):
|
||||
agent_uuid = None
|
||||
|
||||
details = {
|
||||
details: dict[str, Any] = {
|
||||
"from_status": from_status,
|
||||
"to_status": to_status,
|
||||
"agent_role": agent_role,
|
||||
@@ -861,6 +898,39 @@ class TaskService(BaseService):
|
||||
"(same batch_id, top-level)."
|
||||
)
|
||||
|
||||
def _enforce_no_pm_code_on_create(self, req: TaskCreateRequest) -> None:
|
||||
"""Refuse a create that would hand a code task to a PM (coordination role).
|
||||
|
||||
Two layers: the team-based Main-PM impossibility backstop (a Main PM
|
||||
coordinates, never owns code — the 2026-06-27 MegaTask meltdown shape),
|
||||
and the assignee-based PM/code guard that closes the create-with-cell-
|
||||
PM-assignee hole the team check misses. A brand-new task is never in
|
||||
``needs_revision``, so the issue-resolution carve-out does not apply.
|
||||
"""
|
||||
if main_pm_cannot_own_code(team=req.team, task_type=req.task_type):
|
||||
raise ValidationError(
|
||||
"MAIN_PM_NO_CODE: A Main PM task coordinates — it does not"
|
||||
" execute code. Re-draft as `planning` with coordination-level"
|
||||
" acceptance criteria, or target a cell so a developer owns the"
|
||||
" code.",
|
||||
field="task_type",
|
||||
)
|
||||
if not req.assigned_to:
|
||||
return
|
||||
from roboco.foundation.identity import role_for_uuid_or_none
|
||||
|
||||
assignee_role = role_for_uuid_or_none(req.assigned_to)
|
||||
if assignee_role is not None and pm_cannot_own_code(
|
||||
role=assignee_role, task_type=req.task_type, is_issue_resolution=False
|
||||
):
|
||||
raise ValidationError(
|
||||
"PM_NO_CODE: the assignee is a PM — PMs coordinate, they"
|
||||
" do not execute code. Re-draft as `planning` and delegate"
|
||||
" the code to a developer, or assign the code task to a"
|
||||
" developer.",
|
||||
field="task_type",
|
||||
)
|
||||
|
||||
async def create(self, req: TaskCreateRequest) -> TaskTable:
|
||||
"""
|
||||
Create a new task.
|
||||
@@ -874,22 +944,9 @@ class TaskService(BaseService):
|
||||
if req.parent_task_id:
|
||||
await self._validate_parent_depth(req.parent_task_id)
|
||||
|
||||
# Impossibility backstop: a Main PM coordinates — it never owns a code
|
||||
# task. ``main_pm`` + ``code`` on the same task is the structural
|
||||
# mismatch behind the 2026-06-27 MegaTask meltdown (a root-subtask the
|
||||
# git/PR/review layer treated as code while ownership treated it as
|
||||
# coordination — never reconciled, pr_fail looped). Intake
|
||||
# (create_task_from_draft) coerces code→planning, so this fires only on
|
||||
# a non-intake create (the HTTP route / a direct internal create) that
|
||||
# tries to persist the forbidden combo.
|
||||
if main_pm_cannot_own_code(team=req.team, task_type=req.task_type):
|
||||
raise ValidationError(
|
||||
"MAIN_PM_NO_CODE: A Main PM task coordinates — it does not"
|
||||
" execute code. Re-draft as `planning` with coordination-level"
|
||||
" acceptance criteria, or target a cell so a developer owns the"
|
||||
" code.",
|
||||
field="task_type",
|
||||
)
|
||||
# PM/code guards: a coordination role never owns a freshly created code
|
||||
# task (the Main-PM impossibility + the cell-PM-assignee hole).
|
||||
self._enforce_no_pm_code_on_create(req)
|
||||
|
||||
# Stable per-criterion ids (1:1 with acceptance_criteria) so children can
|
||||
# reference specific parent criteria; generated here when not supplied.
|
||||
@@ -1186,24 +1243,35 @@ class TaskService(BaseService):
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def list_open_ci_watch_tasks(
|
||||
self, git_url: str | None = None
|
||||
self, git_url: str | None = None, workflow: str | None = None
|
||||
) -> list[TaskTable]:
|
||||
"""Non-terminal ci_watch fix tasks — the dedupe + open-cap basis.
|
||||
|
||||
Optionally scoped to one repo by ``git_url``: a monorepo registers
|
||||
several cell-projects on ONE git_url, so CI-watch dedupe must key on the
|
||||
repo, not the project slug — otherwise a red monorepo would open one fix
|
||||
task per cell-project. While an open task exists for a repo the loop must
|
||||
not originate a second; the rolling open-task cap counts these.
|
||||
task per cell-project. Optionally further scoped by ``workflow``: the
|
||||
dedupe key is ``(git_url, effective workflow)`` so a multi-workflow
|
||||
monorepo with two RED workflows gets a fix task per workflow, not one
|
||||
collapsed task that silently leaves the second workflow un-remediated
|
||||
(#44). The effective workflow is ``COALESCE(ci_watch_workflow, default)``
|
||||
so a NULL-workflow project row matches the default workflow.
|
||||
"""
|
||||
stmt = select(TaskTable).where(
|
||||
TaskTable.source == CI_WATCH_SOURCE,
|
||||
TaskTable.status.notin_([TaskStatus.COMPLETED, TaskStatus.CANCELLED]),
|
||||
)
|
||||
if git_url is not None:
|
||||
stmt = stmt.join(
|
||||
ProjectTable, TaskTable.project_id == ProjectTable.id
|
||||
).where(ProjectTable.git_url == git_url)
|
||||
if git_url is not None or workflow is not None:
|
||||
stmt = stmt.join(ProjectTable, TaskTable.project_id == ProjectTable.id)
|
||||
if git_url is not None:
|
||||
stmt = stmt.where(ProjectTable.git_url == git_url)
|
||||
if workflow is not None:
|
||||
from roboco.config import settings
|
||||
|
||||
effective = func.coalesce(
|
||||
ProjectTable.ci_watch_workflow, settings.ci_watch_default_workflow
|
||||
)
|
||||
stmt = stmt.where(effective == workflow)
|
||||
result = await self.session.execute(stmt)
|
||||
return list(result.scalars().all())
|
||||
|
||||
@@ -2064,6 +2132,7 @@ class TaskService(BaseService):
|
||||
*,
|
||||
actor_id: str | UUID | None = None,
|
||||
actor_role: str | None = None,
|
||||
force: bool = False,
|
||||
) -> TaskTable | None:
|
||||
"""Privileged override: set a task's status directly, always audited.
|
||||
|
||||
@@ -2072,6 +2141,11 @@ class TaskService(BaseService):
|
||||
task whose work already merged out-of-band). The change is recorded in
|
||||
the audit log like any other transition — no status change may skip it.
|
||||
|
||||
#13: ``force`` marks the override as an explicit, acknowledged bypass
|
||||
(the route requires it for terminal/final hatch states). It is stamped
|
||||
into the audit row's ``details`` so a forced override is distinguishable
|
||||
from an in-band transition in the audit journey.
|
||||
|
||||
Taking a task OUT of ``blocked`` here (operator PATCH, or the
|
||||
orchestrator's auto-recover/auto-resume) restores the pre-block owner
|
||||
exactly as ``unblock(restore=True)`` does. Without this, a code task
|
||||
@@ -2104,12 +2178,40 @@ class TaskService(BaseService):
|
||||
agent_role=actor_role,
|
||||
audit_agent_id=actor_id,
|
||||
)
|
||||
if force:
|
||||
# #13: a distinct audit row marks the override as an explicit,
|
||||
# acknowledged bypass past the lifecycle gate — distinguishable
|
||||
# from the in-band transition row above in the audit journey.
|
||||
from roboco.db.tables import AuditLogTable
|
||||
|
||||
agent_uuid_force: UUID | None = None
|
||||
if actor_id is not None:
|
||||
try:
|
||||
agent_uuid_force = UUID(str(actor_id))
|
||||
except (ValueError, AttributeError):
|
||||
agent_uuid_force = None
|
||||
self.session.add(
|
||||
AuditLogTable(
|
||||
event_type="task.admin_override",
|
||||
agent_id=agent_uuid_force,
|
||||
target_type="task",
|
||||
target_id=task.id,
|
||||
severity="warning",
|
||||
details={
|
||||
"from_status": from_status,
|
||||
"to_status": new_status.value,
|
||||
"agent_role": actor_role,
|
||||
"forced": True,
|
||||
},
|
||||
)
|
||||
)
|
||||
self.log.info(
|
||||
"Task status set via admin override",
|
||||
task_id=str(task_id),
|
||||
from_status=from_status,
|
||||
to_status=new_status.value,
|
||||
actor=str(actor_id) if actor_id else None,
|
||||
forced=force,
|
||||
)
|
||||
return task
|
||||
|
||||
@@ -4939,6 +5041,24 @@ class TaskService(BaseService):
|
||||
await self._unblock_dependents(task_id)
|
||||
return task
|
||||
|
||||
async def _escalation_diverts_to_pool(
|
||||
self, task: TaskTable, target_agent_id: UUID
|
||||
) -> bool:
|
||||
"""True when an escalation target cannot own the task — divert to the pool.
|
||||
|
||||
Two shapes: a board/advisory role assigned work it has no verb for
|
||||
(``_board_cannot_own``), and a Main-PM target handed a main_pm+code task
|
||||
(the 2026-06-27 meltdown shape — a coordinator with no code verb cannot
|
||||
fix the code). Either diverts to a pool release for a role-matched reclaim.
|
||||
"""
|
||||
if _board_cannot_own(task) and await self._is_board_advisory_agent(
|
||||
target_agent_id
|
||||
):
|
||||
return True
|
||||
return main_pm_cannot_own_code(
|
||||
team=task.team, task_type=task.task_type
|
||||
) and await self._is_main_pm_agent(target_agent_id)
|
||||
|
||||
async def apply_escalation(
|
||||
self,
|
||||
*,
|
||||
@@ -4987,27 +5107,27 @@ class TaskService(BaseService):
|
||||
target=target_slug,
|
||||
)
|
||||
return False
|
||||
if _board_cannot_own(task) and await self._is_board_advisory_agent(
|
||||
target_agent_id
|
||||
):
|
||||
await self._release_code_task_to_pool(
|
||||
task=task,
|
||||
escalator_slug=escalator_slug,
|
||||
blocked_target_slug=target_slug,
|
||||
reason=reason,
|
||||
current = (
|
||||
task.status
|
||||
if isinstance(task.status, TaskStatus)
|
||||
else TaskStatus(str(task.status))
|
||||
)
|
||||
if current not in _ESCALATABLE_TO_BLOCKED:
|
||||
# No arbitrary source→BLOCKED write: a status outside the escalation
|
||||
# exemption (e.g. BACKLOG, never activated) has no spec edge into
|
||||
# BLOCKED. Refuse rather than bypass the validator silently.
|
||||
self.log.warning(
|
||||
"Refusing escalation from non-escalatable status",
|
||||
task_id=str(task.id),
|
||||
status=str(task.status),
|
||||
escalator=escalator_slug,
|
||||
target=target_slug,
|
||||
)
|
||||
return True
|
||||
# Impossibility backstop: a Main-PM target must never receive (back) a
|
||||
# main_pm + code task — a coordinator with no code verb cannot fix the
|
||||
# code, so escalating it to Main PM perpetuates the mismatch (the
|
||||
# 2026-06-27 meltdown shape). Scoped to the team+type combo (NOT a broad
|
||||
# code+main-pm-target rule) so a legacy main_pm+code task can still be
|
||||
# escalated to a cell dev — the correct remediation. The combo is
|
||||
# uncreatable going forward (create backstop + intake coercion), so this
|
||||
# is a backstop for legacy / direct-ORM-write tasks.
|
||||
if main_pm_cannot_own_code(
|
||||
team=task.team, task_type=task.task_type
|
||||
) and await self._is_main_pm_agent(target_agent_id):
|
||||
return False
|
||||
if await self._escalation_diverts_to_pool(task, target_agent_id):
|
||||
# A board/advisory or Main-PM-coordination target cannot own this
|
||||
# task — divert to a pool release so a role-matched agent reclaims
|
||||
# it (see ``_escalation_diverts_to_pool`` for the two shapes).
|
||||
await self._release_code_task_to_pool(
|
||||
task=task,
|
||||
escalator_slug=escalator_slug,
|
||||
@@ -5198,13 +5318,14 @@ class TaskService(BaseService):
|
||||
task.completed_at = datetime.now(UTC)
|
||||
# Validate transition with CEO role requirement
|
||||
self._validate_and_set_status(task, TaskStatus.COMPLETED, "ceo")
|
||||
await self.session.flush()
|
||||
# Mirror ``complete()``: close the work session (the PR is merged) and
|
||||
# run the full completion-hook fan-out (commits + dev_notes indexing),
|
||||
# not a lone learnings task. Without this the work session stayed ACTIVE
|
||||
# forever after a CEO approval and code/decision RAG indexing was skipped.
|
||||
await self._close_work_session_for_task(task, reason="ceo approved")
|
||||
await self._remove_task_worktree_on_terminal(task)
|
||||
|
||||
# Extract learnings (fire-and-forget)
|
||||
bg_task = asyncio.create_task(self._extract_completion_learnings(task, None))
|
||||
self._background_tasks.add(bg_task)
|
||||
bg_task.add_done_callback(self._background_tasks.discard)
|
||||
await self.session.flush()
|
||||
await self._trigger_completion_hooks(task, None)
|
||||
|
||||
# Unblock any tasks waiting on this one
|
||||
await self._unblock_dependents(task_id)
|
||||
@@ -5334,6 +5455,7 @@ class TaskService(BaseService):
|
||||
released = False
|
||||
for child in await self.get_subtasks(cast("UUID", umbrella.id)):
|
||||
if child.batch_id is not None and child.status == TaskStatus.BACKLOG:
|
||||
pre_status = child.status.value
|
||||
child.status = TaskStatus.PENDING
|
||||
child.team = cast("Any", Team.MAIN_PM)
|
||||
# a board-routed root-subtask is created in BACKLOG with
|
||||
@@ -5349,6 +5471,20 @@ class TaskService(BaseService):
|
||||
task_id=str(child.id),
|
||||
)
|
||||
child.task_type = cast("Any", TaskType.PLANNING)
|
||||
# No status change may bypass the audit log — record the
|
||||
# backlog→pending activation so the child's lifecycle
|
||||
# reconstruction (the metric source of truth) keeps its start.
|
||||
# audit_agent_id is None: approve_and_start receives no approver
|
||||
# id, and the audit_log.agent_id FK is nullable — attributing a
|
||||
# hardcoded ceo uuid would FK-violate when the ceo agent row is
|
||||
# absent. agent_role="ceo" still labels the actor.
|
||||
self._emit_status_transition_audit(
|
||||
child,
|
||||
from_status=pre_status,
|
||||
to_status=TaskStatus.PENDING.value,
|
||||
agent_role="ceo",
|
||||
audit_agent_id=None,
|
||||
)
|
||||
released = True
|
||||
if released:
|
||||
await self.session.flush()
|
||||
@@ -5619,12 +5755,63 @@ class TaskService(BaseService):
|
||||
if not project_slug:
|
||||
return
|
||||
await self._remove_task_worktree_best_effort(task, project_slug)
|
||||
except OSError as e:
|
||||
# FS/permission failure (stuck mount, perms) — systemic, not
|
||||
# task-specific. Track the streak and escalate a CEO alert once
|
||||
# after N consecutive failures so worktrees don't leak silently.
|
||||
TaskService._worktree_cleanup_fail_streak += 1
|
||||
self.log.warning(
|
||||
"Terminal worktree cleanup skipped (FS/permission)",
|
||||
task_id=str(task.id),
|
||||
streak=TaskService._worktree_cleanup_fail_streak,
|
||||
error=str(e),
|
||||
)
|
||||
if (
|
||||
TaskService._worktree_cleanup_fail_streak
|
||||
>= TaskService._WORKTREE_CLEANUP_ESCALATE_AFTER
|
||||
and not TaskService._worktree_cleanup_escalated
|
||||
):
|
||||
TaskService._worktree_cleanup_escalated = True
|
||||
self.log.error(
|
||||
"Recurring worktree-cleanup FS failure escalating to CEO",
|
||||
streak=TaskService._worktree_cleanup_fail_streak,
|
||||
task_id=str(task.id),
|
||||
)
|
||||
await self._escalate_worktree_cleanup_failure(task, e)
|
||||
except Exception as e:
|
||||
# Non-FS failure (git, DB) — task-specific, best-effort, no streak.
|
||||
TaskService._worktree_cleanup_fail_streak = 0
|
||||
self.log.warning(
|
||||
"Terminal worktree cleanup skipped",
|
||||
task_id=str(task.id),
|
||||
error=str(e),
|
||||
)
|
||||
else:
|
||||
TaskService._worktree_cleanup_fail_streak = 0
|
||||
TaskService._worktree_cleanup_escalated = False
|
||||
|
||||
async def _escalate_worktree_cleanup_failure(
|
||||
self, task: TaskTable, exc: BaseException
|
||||
) -> None:
|
||||
"""Best-effort CEO alert on recurring worktree-cleanup FS failure."""
|
||||
try:
|
||||
from roboco.services.notification import NotificationService
|
||||
|
||||
await NotificationService().send_ack_notification(
|
||||
from_agent="system",
|
||||
to_agent="ceo",
|
||||
body=(
|
||||
f"Recurring worktree-cleanup FS failure "
|
||||
f"(last error: {str(exc)[:160]}). "
|
||||
f"Task {task.id} terminal worktree could not be removed — "
|
||||
"per-task worktrees may be leaking on disk. Investigate the "
|
||||
"workspace mount / permissions."
|
||||
),
|
||||
priority=NotificationPriority.HIGH,
|
||||
task_id=str(task.id),
|
||||
)
|
||||
except Exception as e:
|
||||
self.log.warning("Worktree-cleanup escalation notify failed", error=str(e))
|
||||
|
||||
async def _close_work_session_for_task(self, task: TaskTable, reason: str) -> None:
|
||||
"""Close the task's work session on successful completion.
|
||||
@@ -5668,9 +5855,13 @@ class TaskService(BaseService):
|
||||
# Cancel all descendants first (children, grandchildren, etc.)
|
||||
# Skip tasks already in terminal states (completed or cancelled).
|
||||
# Route every descendant through _validate_and_set_status so role
|
||||
# restrictions (e.g., only CEO can cancel awaiting_ceo_approval) still
|
||||
# apply to cascaded cancels — skip descendants that fail validation
|
||||
# rather than bypassing the rules.
|
||||
# restrictions still apply to cascaded cancels. A non-terminal
|
||||
# descendant the caller's role can't cancel REFUSES the whole
|
||||
# cancel — never silently skip it and leave an orphaned subtree
|
||||
# under a cancelled parent. (The spec gates every cancel edge to
|
||||
# {cell_pm, main_pm, ceo} uniformly, so a PM/CEO cancel won't hit
|
||||
# this; the narrow catch + refusal keeps a future per-edge role
|
||||
# gate from silently orphaning.) Non-validation errors propagate.
|
||||
descendants = await self.get_all_descendants(task_id)
|
||||
cancelled_count = 0
|
||||
for descendant in descendants:
|
||||
@@ -5680,15 +5871,23 @@ class TaskService(BaseService):
|
||||
self._validate_and_set_status(
|
||||
descendant, TaskStatus.CANCELLED, agent_role
|
||||
)
|
||||
except Exception as e:
|
||||
except TaskLifecycleError as e:
|
||||
self.log.warning(
|
||||
"Skipping cascade-cancel of descendant; role not permitted",
|
||||
"Refusing cascade-cancel: caller role cannot cancel descendant",
|
||||
descendant_id=str(descendant.id),
|
||||
descendant_status=descendant.status.value,
|
||||
agent_role=agent_role,
|
||||
error=str(e),
|
||||
)
|
||||
continue
|
||||
raise TaskLifecycleError(
|
||||
current_status=descendant.status.value,
|
||||
target_status=TaskStatus.CANCELLED.value,
|
||||
message=(
|
||||
f"Cannot cancel descendant {descendant.id} in state "
|
||||
f"{descendant.status.value} with role {agent_role!r}; "
|
||||
"refusing parent cancel to avoid an orphaned subtree."
|
||||
),
|
||||
) from e
|
||||
cancelled_count += 1
|
||||
await self._abandon_work_session_for_task(
|
||||
descendant, reason="parent task cancelled"
|
||||
@@ -8247,7 +8446,9 @@ class TaskService(BaseService):
|
||||
await self.session.flush()
|
||||
# This restore path sets the status directly (bypassing the strict
|
||||
# transition validator), so emit the audit explicitly — no status
|
||||
# change may skip the audit log.
|
||||
# change may skip the audit log. A restore to a snapshotted
|
||||
# needs_revision is the same rework cycle resuming, not a fresh
|
||||
# rejection, so undo the bump the chokepoint applied above.
|
||||
self._emit_status_transition_audit(
|
||||
task,
|
||||
from_status=pre_status,
|
||||
@@ -8255,6 +8456,11 @@ class TaskService(BaseService):
|
||||
agent_role=None,
|
||||
audit_agent_id=restored_owner,
|
||||
)
|
||||
if (
|
||||
restored_status == TaskStatus.NEEDS_REVISION
|
||||
and pre_status != TaskStatus.NEEDS_REVISION.value
|
||||
):
|
||||
task.revision_count = max((task.revision_count or 1) - 1, 0)
|
||||
return task
|
||||
|
||||
async def cell_pm_complete(
|
||||
|
||||
@@ -227,22 +227,34 @@ class TranscriptionService:
|
||||
while self._running:
|
||||
try:
|
||||
await asyncio.sleep(self.config.flush_interval_seconds)
|
||||
|
||||
async for buffer in self.get_ready_buffers():
|
||||
# Notify callbacks
|
||||
for callback in self._segment_callbacks:
|
||||
try:
|
||||
callback(buffer)
|
||||
except Exception as e:
|
||||
self.log.error(
|
||||
"Callback error",
|
||||
error=str(e),
|
||||
)
|
||||
|
||||
await self._flush_ready_buffers()
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
self.log.error("Periodic flush error", error=str(e))
|
||||
self.log.error("Periodic flush error", error=e.__class__.__name__)
|
||||
|
||||
async def _flush_ready_buffers(self) -> None:
|
||||
"""One flush pass: notify callbacks for each ready buffer, then remove it.
|
||||
|
||||
``get_ready_buffers`` only peeks — without flushing, a ready buffer was
|
||||
re-yielded on every tick and ``_buffers`` grew unbounded (#96). Sync
|
||||
callbacks are offloaded to a worker thread so a slow callback can't
|
||||
block the flush task / event loop (#97). A failing callback is logged
|
||||
and skipped; a sibling callback still runs and the buffer is still
|
||||
flushed.
|
||||
"""
|
||||
async for buffer in self.get_ready_buffers():
|
||||
for callback in self._segment_callbacks:
|
||||
try:
|
||||
await asyncio.to_thread(callback, buffer)
|
||||
except Exception as e:
|
||||
self.log.error(
|
||||
"Callback error",
|
||||
error=e.__class__.__name__,
|
||||
)
|
||||
# Remove the buffer now that its segment has been handed off —
|
||||
# without this the same ready buffer accumulates forever.
|
||||
await self.flush_buffer(buffer.agent_id, buffer.session_id)
|
||||
|
||||
async def _flush_all(self) -> None:
|
||||
"""Flush all buffers (called on shutdown)."""
|
||||
|
||||
@@ -77,8 +77,13 @@ class UsageService(BaseService):
|
||||
async def get_summary(self, period: str = "24h") -> dict[str, Any]:
|
||||
"""Return aggregated token and cost totals for the given period.
|
||||
|
||||
Queries daily_usage_rollups for whole-day periods; falls back to
|
||||
agent_spawn_sessions for sub-day precision.
|
||||
Sums the RAW ``agent_spawn_sessions`` rows that started within the
|
||||
period and have closed (``ended_at IS NOT NULL``) — sub-day precise, no
|
||||
rollup dependency. ``daily_usage_rollups`` is the day-grain snapshot
|
||||
written by the background sweeper (``_rollup_daily_usage``) and read by
|
||||
``get_today_summary``; the two can diverge for "today" until the sweeper
|
||||
closes + aggregates still-open sessions (#66 — the prior docstring
|
||||
falsely claimed this read the rollups).
|
||||
|
||||
Returns dict with: tokens_input, tokens_output, total_tokens,
|
||||
total_cost_usd, trend_pct.
|
||||
|
||||
@@ -555,6 +555,126 @@ class WorkspaceService:
|
||||
await asyncio.to_thread(_ensure_agent_owned, worktree)
|
||||
await asyncio.to_thread(_ensure_agent_owned, clone_root)
|
||||
|
||||
async def _fetch_branch_ref(
|
||||
self, clone_root: Path, branch: str, project_slug: str
|
||||
) -> None:
|
||||
"""Token-aware ``git fetch origin <branch>`` into clone_root. Best-effort.
|
||||
|
||||
A clone's ``.git/config`` carries the token-scrubbed remote URL, so a
|
||||
private-repo fetch needs the token re-injected via ``http.extraheader``
|
||||
(mirrors ``fetch_branch_for_inspection``). Never raises — a failed
|
||||
fetch falls through to ``ensure_worktree``'s ``-b`` from ``origin/HEAD``.
|
||||
"""
|
||||
from roboco.services.project import get_project_service
|
||||
from roboco.utils.crypto import EncryptionError
|
||||
|
||||
project_service = get_project_service(self.session)
|
||||
project = await project_service.get_by_slug(project_slug)
|
||||
git_token: str | None = None
|
||||
if project is not None:
|
||||
try:
|
||||
git_token = await project_service.get_decrypted_token_by_slug(
|
||||
project_slug
|
||||
)
|
||||
except EncryptionError:
|
||||
git_token = None
|
||||
|
||||
prefix: list[str] = []
|
||||
if git_token:
|
||||
import base64
|
||||
|
||||
basic = base64.b64encode(f"x-access-token:{git_token}".encode()).decode()
|
||||
prefix = ["-c", f"http.extraheader=Authorization: Basic {basic}"]
|
||||
|
||||
def _do_fetch() -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
[
|
||||
"git",
|
||||
"-C",
|
||||
str(clone_root),
|
||||
*prefix,
|
||||
"fetch",
|
||||
"--no-tags",
|
||||
"origin",
|
||||
branch,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=settings.workspace_refresh_fetch_timeout_seconds,
|
||||
check=False,
|
||||
)
|
||||
|
||||
result = await asyncio.to_thread(_do_fetch)
|
||||
if result.returncode != 0:
|
||||
logger.warning(
|
||||
"ensure_worktree_self_heal: branch fetch returned non-zero",
|
||||
branch=branch,
|
||||
clone_root=str(clone_root),
|
||||
stderr=result.stderr.strip(),
|
||||
)
|
||||
|
||||
async def ensure_worktree_self_heal(
|
||||
self,
|
||||
clone_root: Path,
|
||||
worktree: Path,
|
||||
branch: str,
|
||||
project_slug: str,
|
||||
) -> None:
|
||||
"""Re-attach a per-task worktree, self-healing a vanished clone + ref.
|
||||
|
||||
F123 spawn-time chokepoint. A resumed agent's clone can vanish (disk
|
||||
loss, a redeploy that wiped ``/data/workspaces``, manual cleanup) while
|
||||
the task's DB row keeps ``branch_name`` and the reaper-style claim
|
||||
release preserves ownership — so the next dispatch is a RESUME, not a
|
||||
fresh claim, and ``create_branch`` never re-runs to re-clone.
|
||||
``ensure_worktree_for_resume`` then runs ``git -C <clone_root>`` against
|
||||
a missing directory and fatal-loops (claim released -> re-dispatched
|
||||
into the same missing clone -> repeat, every tick).
|
||||
|
||||
The orchestrator calls ``ensure_workspace`` first when the clone is
|
||||
unhealthy (re-clone from default). This then recovers the task branch
|
||||
ref so the worktree re-attaches with the pushed work intact:
|
||||
|
||||
1. A present, registered worktree is a no-op (just venv + ownership).
|
||||
2. If the local ``refs/heads/{branch}`` ref is absent (a re-clone has
|
||||
none), fetch ``origin <branch>`` (token-aware) and create the local
|
||||
ref from ``refs/remotes/origin/{branch}`` when origin has it —
|
||||
recovering the pushed commits. (``create_branch`` always pushes at
|
||||
claim time, so a claimed task's branch is on origin.)
|
||||
3. ``ensure_worktree`` reuses the recovered ref, or — if origin doesn't
|
||||
have it (a never-pushed branch) — re-creates it from ``origin/HEAD``;
|
||||
no pushed work is lost because none existed.
|
||||
|
||||
A transient fetch failure falls through to the ``origin/HEAD`` ``-b``
|
||||
rather than fatal-looping; a diverged branch re-syncs on the agent's
|
||||
first ``sync_branch``.
|
||||
"""
|
||||
if worktree.exists() and (worktree / ".git").is_file():
|
||||
self._link_shared_venv(worktree, clone_root)
|
||||
await asyncio.to_thread(_ensure_agent_owned, worktree)
|
||||
await asyncio.to_thread(_ensure_agent_owned, clone_root)
|
||||
return
|
||||
local = self._worktree_git(
|
||||
clone_root,
|
||||
["rev-parse", "--verify", "--quiet", f"refs/heads/{branch}"],
|
||||
check=False,
|
||||
)
|
||||
if local.returncode != 0:
|
||||
await self._fetch_branch_ref(clone_root, branch, project_slug)
|
||||
remote = self._worktree_git(
|
||||
clone_root,
|
||||
["rev-parse", "--verify", "--quiet", f"refs/remotes/origin/{branch}"],
|
||||
check=False,
|
||||
)
|
||||
if remote.returncode == 0:
|
||||
self._worktree_git(
|
||||
clone_root,
|
||||
["branch", branch, f"refs/remotes/origin/{branch}"],
|
||||
check=False,
|
||||
)
|
||||
# Reuse refs/heads/{branch} if recovered; else -b from origin/HEAD.
|
||||
await self.ensure_worktree(clone_root, worktree, branch, "origin/HEAD")
|
||||
|
||||
async def remove_worktree(self, clone_root: Path, worktree: Path) -> None:
|
||||
"""Remove a per-task worktree (cancel / terminal / reaper evict).
|
||||
|
||||
|
||||
@@ -114,6 +114,12 @@ def _format_field(name: str, field_info: Any) -> str:
|
||||
def _annot_str(annot: Any) -> str:
|
||||
if annot is None or annot is type(None):
|
||||
return "None"
|
||||
# Drop Annotated[T, ...] metadata (BeforeValidator, etc.) — render the bare
|
||||
# type so the prompt table never leaks ``BeforeValidator(func=...)`` repr
|
||||
# (with a memory address) into agent-facing prompt text (#199).
|
||||
metadata = getattr(annot, "__metadata__", None)
|
||||
if metadata is not None:
|
||||
annot = annot.__origin__
|
||||
origin = get_origin(annot)
|
||||
if origin is None:
|
||||
return getattr(annot, "__name__", str(annot))
|
||||
|
||||
@@ -29,6 +29,7 @@ from __future__ import annotations
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import warnings
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
@@ -97,7 +98,21 @@ def _postgres_reachable() -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _warn_if_pg_unavailable(available: bool, host: str, port: int) -> None:
|
||||
"""#90: warn loudly when the test DB is unreachable so a green run of all-skips
|
||||
is not mistaken for a pass. The per-test ``pytest.skip`` still applies; this
|
||||
only adds a visible import-time warning."""
|
||||
if not available:
|
||||
warnings.warn(
|
||||
f"Postgres unreachable at {host}:{port} — every DB test will be "
|
||||
"SKIPPED. Set ROBOCO_TEST_DB_HOST/PORT/USER/PASSWORD or start "
|
||||
"Postgres (local: ROBOCO_TEST_DB_PORT=55432 ROBOCO_TEST_DB_USER=renzof).",
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
|
||||
_PG_AVAILABLE = _postgres_reachable()
|
||||
_warn_if_pg_unavailable(_PG_AVAILABLE, _TEST_DB_HOST, _TEST_DB_PORT)
|
||||
|
||||
# Sanity-check imported tables registered themselves on Base.metadata. Tied to
|
||||
# `roboco_tables` so static analysis treats the import as load-bearing.
|
||||
|
||||
@@ -4,10 +4,12 @@ from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
|
||||
from roboco import agents_config
|
||||
from roboco.agents_config import CHANNEL_ACCESS
|
||||
from roboco.foundation import identity
|
||||
from roboco.foundation.policy import communications
|
||||
from roboco.models.base import NotificationPriority, NotificationType
|
||||
from roboco.seeds import initial_data as seeds_initial_data
|
||||
|
||||
|
||||
def test_priority_enum_matches_notification_priority() -> None:
|
||||
@@ -141,3 +143,21 @@ def test_parse_priority_explicit_priority_wins_over_legacy_flag() -> None:
|
||||
communications.parse_priority("normal", legacy_urgent_flag=True)
|
||||
is communications.Priority.NORMAL
|
||||
)
|
||||
|
||||
|
||||
def test_team_scoped_roles_is_single_sourced() -> None:
|
||||
"""#212: ``TEAM_SCOPED_ROLES`` is defined once in the foundation module and
|
||||
every consumer (agents_config, seeds/initial_data) references THAT object —
|
||||
no duplicate definitions that could drift."""
|
||||
expected = frozenset(
|
||||
{
|
||||
identity.Role.DEVELOPER,
|
||||
identity.Role.QA,
|
||||
identity.Role.DOCUMENTER,
|
||||
identity.Role.CELL_PM,
|
||||
}
|
||||
)
|
||||
assert expected == communications.TEAM_SCOPED_ROLES
|
||||
# Same object identity — not a re-built copy that could drift.
|
||||
assert agents_config._TEAM_SCOPED_ROLES is communications.TEAM_SCOPED_ROLES
|
||||
assert seeds_initial_data._TEAM_SCOPED_ROLES is communications.TEAM_SCOPED_ROLES
|
||||
|
||||
@@ -18,10 +18,9 @@ from roboco.services.gateway import content_actions
|
||||
|
||||
# Cell-member roles that are subject to a channel's team_scope. Cross-cell
|
||||
# roles (MAIN_PM, AUDITOR, CEO, board) are NOT filtered — they participate
|
||||
# regardless of which team a cell channel scopes to.
|
||||
_TEAM_SCOPED_ROLES: frozenset[Role] = frozenset(
|
||||
{Role.DEVELOPER, Role.QA, Role.DOCUMENTER, Role.CELL_PM}
|
||||
)
|
||||
# regardless of which team a cell channel scopes to. Single-sourced in the
|
||||
# foundation module under test.
|
||||
_TEAM_SCOPED_ROLES: frozenset[Role] = communications.TEAM_SCOPED_ROLES
|
||||
|
||||
|
||||
def _expected_slugs(role_set: frozenset[Role], team_scope: Team | None) -> set[str]:
|
||||
|
||||
@@ -241,3 +241,40 @@ def test_role_for_slug_or_none_unknown_returns_none() -> None:
|
||||
def test_role_for_slug_or_none_known_returns_role() -> None:
|
||||
assert identity.role_for_slug_or_none("be-pm") == identity.Role.CELL_PM
|
||||
assert identity.role_for_slug_or_none("ceo") == identity.Role.CEO
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# #49: human-only-role + spawnable-slug predicates (layered skip guards)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_is_human_only_role_for_each_human_role() -> None:
|
||||
for role in (identity.Role.CEO, identity.Role.PROMPTER, identity.Role.SECRETARY):
|
||||
assert identity.is_human_only_role(role) is True
|
||||
|
||||
|
||||
def test_is_human_only_role_false_for_agent_roles_and_none() -> None:
|
||||
assert identity.is_human_only_role(identity.Role.DEVELOPER) is False
|
||||
assert identity.is_human_only_role(identity.Role.QA) is False
|
||||
assert identity.is_human_only_role(identity.Role.CELL_PM) is False
|
||||
assert identity.is_human_only_role(None) is False
|
||||
|
||||
|
||||
def test_is_spawnable_agent_slug_true_for_known_non_human() -> None:
|
||||
assert identity.is_spawnable_agent_slug("be-dev-1") is True
|
||||
assert identity.is_spawnable_agent_slug("be-qa") is True
|
||||
assert identity.is_spawnable_agent_slug("be-pm") is True
|
||||
|
||||
|
||||
def test_is_spawnable_agent_slug_false_for_live_human_slugs() -> None:
|
||||
# ceo / intake-1 (prompter) / secretary-1 are the seeded human-only slugs.
|
||||
for slug in ("ceo", "intake-1", "secretary-1"):
|
||||
assert identity.is_spawnable_agent_slug(slug) is False
|
||||
|
||||
|
||||
def test_is_spawnable_agent_slug_false_for_stale_slug() -> None:
|
||||
"""#49: a slug that resolves to no known agent must not be treated as
|
||||
spawnable — a stale/ex-human slug must not slip past a skip guard that
|
||||
only checks ``role in (CEO, PROMPTER, SECRETARY)`` (None is not in it)."""
|
||||
assert identity.is_spawnable_agent_slug("old-renamed-secretary") is False
|
||||
assert identity.is_spawnable_agent_slug("does-not-exist-in-agents") is False
|
||||
|
||||
@@ -247,6 +247,10 @@ def test_status_transitions_includes_ceo_paths() -> None:
|
||||
) in sources
|
||||
assert (spec.Status.AWAITING_CEO_APPROVAL, spec.Status.COMPLETED) in sources
|
||||
assert (spec.Status.AWAITING_CEO_APPROVAL, spec.Status.NEEDS_REVISION) in sources
|
||||
# #100: a branchless coordination root rejected by the CEO routes to PENDING
|
||||
# (Main PM re-plans) — the edge is in the spec so the audited privileged
|
||||
# override that applies it can't be wedged by future admin-override tightening.
|
||||
assert (spec.Status.AWAITING_CEO_APPROVAL, spec.Status.PENDING) in sources
|
||||
# A blocked task the PM cannot resolve can also be surfaced to the CEO.
|
||||
assert (spec.Status.BLOCKED, spec.Status.AWAITING_CEO_APPROVAL) in sources
|
||||
|
||||
@@ -717,7 +721,12 @@ def test_valid_next_verbs_developer_in_progress_includes_open_pr_and_i_am_done()
|
||||
|
||||
|
||||
def test_valid_next_verbs_pm_pending_includes_i_will_plan() -> None:
|
||||
verbs = spec.valid_next_verbs(spec.Role.CELL_PM, _stub_task(status="pending"))
|
||||
# A PM i_will_plan's a PLANNING task (coordination), not a code task — the
|
||||
# PM/code claim carve-out (Fix 2) removes i_will_plan from a pending code
|
||||
# task's verb set, so the legitimate path is exercised with task_type=planning.
|
||||
verbs = spec.valid_next_verbs(
|
||||
spec.Role.CELL_PM, _stub_task(status="pending", task_type="planning")
|
||||
)
|
||||
assert "i_will_plan" in verbs
|
||||
|
||||
|
||||
@@ -891,3 +900,65 @@ def test_unmigrated_is_pinned() -> None:
|
||||
)
|
||||
== spec.UNMIGRATED
|
||||
)
|
||||
|
||||
|
||||
# --- PM/code claim invariant (Fix 2 + bug-1): the claim gate does NOT block a
|
||||
# PM claiming a code task. A PM's only claim verb is i_will_plan, and planning a
|
||||
# code-typed PARENT (to decompose + delegate the code) is legitimate (bug-1:
|
||||
# scoping pm_cannot_execute_code to i_will_plan deadlocked the slice). Execution
|
||||
# is blocked at the intent level — i_will_work_on is _DEV_ROLES only. The
|
||||
# create/delegate guards (pm_cannot_own_code) block a PM from being ASSIGNED a
|
||||
# fresh code task; the needs_revision carve-out (a PM resolving review issues
|
||||
# directly / recovering a rejected coordination task) is naturally allowed
|
||||
# because PMs claim NEEDS_REVISION. These pin that the claim gate does not
|
||||
# regress bug-1.
|
||||
|
||||
|
||||
def _claim_task(*, status: str, task_type: str) -> Any:
|
||||
return SimpleNamespace(status=status, task_type=task_type)
|
||||
|
||||
|
||||
def test_claim_allows_cell_pm_claiming_code_from_pending() -> None:
|
||||
"""bug-1: a cell PM i_will_plan-ing a code-typed parent (PENDING) to plan +
|
||||
delegate the code MUST be allowed — rejecting it deadlocks the slice."""
|
||||
t = _claim_task(status="pending", task_type="code")
|
||||
d = spec.can_invoke_action(spec.Role.CELL_PM, "claim", t)
|
||||
assert d.allowed, d.message
|
||||
|
||||
|
||||
def test_claim_allows_cell_pm_claiming_code_from_needs_revision() -> None:
|
||||
"""Carve-out: a PM may take a code task in needs_revision to resolve the
|
||||
review/QA issues directly / recover a rejected coordination task."""
|
||||
t = _claim_task(status="needs_revision", task_type="code")
|
||||
d = spec.can_invoke_action(spec.Role.CELL_PM, "claim", t)
|
||||
assert d.allowed, d.message
|
||||
|
||||
|
||||
def test_claim_allows_main_pm_claiming_code_from_needs_revision() -> None:
|
||||
"""The same carve-out holds for the Main PM (coordination-recovery path)."""
|
||||
t = _claim_task(status="needs_revision", task_type="code")
|
||||
d = spec.can_invoke_action(spec.Role.MAIN_PM, "claim", t)
|
||||
assert d.allowed, d.message
|
||||
|
||||
|
||||
def test_claim_allows_main_pm_claiming_code_from_pending() -> None:
|
||||
"""bug-1 parity: a Main PM planning a code-typed parent (PENDING) is allowed
|
||||
for the same reason as the cell PM — execution is blocked at i_will_work_on,
|
||||
not at the claim gate."""
|
||||
t = _claim_task(status="pending", task_type="code")
|
||||
d = spec.can_invoke_action(spec.Role.MAIN_PM, "claim", t)
|
||||
assert d.allowed, d.message
|
||||
|
||||
|
||||
def test_claim_allows_pm_claiming_planning_from_pending() -> None:
|
||||
"""A PM claiming a planning task is the legitimate coordination path."""
|
||||
t = _claim_task(status="pending", task_type="planning")
|
||||
assert spec.can_invoke_action(spec.Role.CELL_PM, "claim", t).allowed
|
||||
assert spec.can_invoke_action(spec.Role.MAIN_PM, "claim", t).allowed
|
||||
|
||||
|
||||
def test_claim_allows_developer_claiming_code_from_pending() -> None:
|
||||
"""A developer claiming fresh code is unaffected (the PM invariant is
|
||||
enforced at create/delegate + i_will_work_on, not the claim gate)."""
|
||||
t = _claim_task(status="pending", task_type="code")
|
||||
assert spec.can_invoke_action(spec.Role.DEVELOPER, "claim", t).allowed
|
||||
|
||||
@@ -76,6 +76,24 @@ def test_submit_root_is_main_pm_only_and_opens_a_pr() -> None:
|
||||
assert iv.pre_side_effects == ("create_root_pr",)
|
||||
|
||||
|
||||
def test_submit_root_rejects_a_code_typed_root() -> None:
|
||||
"""#148: submit_root's prose asserts 'a Main-PM root is planning-typed,
|
||||
never code' — the spec must back that claim, not only the upstream creation
|
||||
guard (``main_pm_cannot_own_code``). A code-typed root is rejected at the
|
||||
spec gate as invalid_state (defense in depth: such a root is already
|
||||
unreachable via creation, but the spec mustn't accept it either)."""
|
||||
t = _task("in_progress", team="main_pm", task_type="code")
|
||||
d = spec.can_invoke_intent(Role.MAIN_PM, "submit_root", t)
|
||||
assert not d.allowed
|
||||
assert d.rejection_kind == "invalid_state"
|
||||
|
||||
|
||||
def test_submit_root_allows_a_planning_typed_root() -> None:
|
||||
"""The legitimate Main-PM root shape — planning-typed, in_progress — passes."""
|
||||
t = _task("in_progress", team="main_pm", task_type="planning")
|
||||
assert spec.can_invoke_intent(Role.MAIN_PM, "submit_root", t).allowed
|
||||
|
||||
|
||||
def test_gate_cannot_skip_straight_to_terminal_or_ceo() -> None:
|
||||
targets = spec.STATUS_GRAPH[Status.AWAITING_PR_REVIEW]
|
||||
assert Status.COMPLETED not in targets
|
||||
|
||||
@@ -70,7 +70,9 @@ async def _get_or_create_agent(
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def _seed_project(db: AsyncSession, slug: str, git_url: str) -> ProjectTable:
|
||||
async def _seed_project(
|
||||
db: AsyncSession, slug: str, git_url: str, *, workflow: str | None = None
|
||||
) -> ProjectTable:
|
||||
project = ProjectTable(
|
||||
id=uuid4(),
|
||||
name=slug,
|
||||
@@ -79,6 +81,7 @@ async def _seed_project(db: AsyncSession, slug: str, git_url: str) -> ProjectTab
|
||||
assigned_cell=Team.BACKEND,
|
||||
created_by=SYSTEM_UUID,
|
||||
ci_watch_enabled=True,
|
||||
ci_watch_workflow=workflow,
|
||||
)
|
||||
db.add(project)
|
||||
await db.flush()
|
||||
@@ -157,3 +160,52 @@ async def test_disabled_is_noop(
|
||||
proj = await _seed_project(db_session, "red-e", "https://github.com/x/e.git")
|
||||
src = _FakeSource([_breach("red-e")])
|
||||
assert await get_ci_watch_engine(db_session, source=src).run_cycle([proj]) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# #44: dedupe by (git_url, workflow) — a multi-workflow monorepo with two red
|
||||
# workflows gets a fix task per workflow, not one collapsed task per repo
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_two_red_workflows_of_one_monorepo_both_open(
|
||||
db_session: AsyncSession,
|
||||
) -> None:
|
||||
"""#44: same git_url, DIFFERENT workflows, both red → two fix tasks."""
|
||||
git = "https://github.com/x/mono.git"
|
||||
p_a = await _seed_project(db_session, "mono-wf-a", git, workflow="wf-a.yml")
|
||||
p_b = await _seed_project(db_session, "mono-wf-b", git, workflow="wf-b.yml")
|
||||
src = _FakeSource([_breach("mono-wf-a"), _breach("mono-wf-b")])
|
||||
projects = [p_a, p_b]
|
||||
created = await get_ci_watch_engine(db_session, source=src).run_cycle(projects)
|
||||
assert len(created) == len(projects)
|
||||
assert {c.project_id for c in created} == {p_a.id, p_b.id}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_same_git_url_same_workflow_still_deduped(
|
||||
db_session: AsyncSession,
|
||||
) -> None:
|
||||
"""Regression guard: two cell-projects on one repo, SAME workflow → one task
|
||||
(the (git_url, workflow) key still collapses a same-workflow monorepo)."""
|
||||
git = "https://github.com/x/mono2.git"
|
||||
p1 = await _seed_project(db_session, "mono2-a", git, workflow="wf-a.yml")
|
||||
p2 = await _seed_project(db_session, "mono2-b", git, workflow="wf-a.yml")
|
||||
src = _FakeSource([_breach("mono2-a"), _breach("mono2-b")])
|
||||
created = await get_ci_watch_engine(db_session, source=src).run_cycle([p1, p2])
|
||||
assert len(created) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_default_workflow_null_rows_deduped(
|
||||
db_session: AsyncSession,
|
||||
) -> None:
|
||||
"""Two NULL-workflow project rows on one repo both use the default workflow
|
||||
→ deduped to one task (coalesce treats NULL as the default workflow)."""
|
||||
git = "https://github.com/x/mono3.git"
|
||||
p1 = await _seed_project(db_session, "mono3-a", git) # ci_watch_workflow=None
|
||||
p2 = await _seed_project(db_session, "mono3-b", git)
|
||||
src = _FakeSource([_breach("mono3-a"), _breach("mono3-b")])
|
||||
created = await get_ci_watch_engine(db_session, source=src).run_cycle([p1, p2])
|
||||
assert len(created) == 1
|
||||
|
||||
@@ -124,6 +124,59 @@ async def test_corrupt_file_falls_back_to_last_ok(
|
||||
assert health.last_ok_sha == "ok1"
|
||||
|
||||
|
||||
async def test_health_recovers_after_in_place_repair(
|
||||
db_session: AsyncSession, tmp_path: Path
|
||||
) -> None:
|
||||
"""#132: ``health`` re-reads the LIVE file status — a cached ``degraded``
|
||||
row must not hide an in-place repair at the same (stale) head key."""
|
||||
project = await _seed_project(
|
||||
db_session, head_commit="h", workspace_path=str(tmp_path)
|
||||
)
|
||||
conv = tmp_path / ".roboco"
|
||||
conv.mkdir()
|
||||
(conv / "conventions.yml").write_text("modules: [unterminated\n")
|
||||
svc = get_conventions_service(db_session)
|
||||
await svc.get_map(project) # caches the degraded state
|
||||
assert (await svc.health(project)).status == "degraded"
|
||||
|
||||
# Repair the file in place at the SAME head key.
|
||||
(conv / "conventions.yml").write_text(
|
||||
"modules:\n - path: lib/fixed\n purpose: fixed\n"
|
||||
)
|
||||
assert (await svc.health(project)).status == "ok"
|
||||
|
||||
|
||||
async def test_get_map_recovers_after_in_place_repair(
|
||||
db_session: AsyncSession, tmp_path: Path
|
||||
) -> None:
|
||||
"""#132: ``get_map`` does not serve a cached ``degraded`` (last-good) map
|
||||
once the file is repaired in place — it re-derives the fixed map."""
|
||||
project = await _seed_project(
|
||||
db_session, head_commit="ok1", workspace_path=str(tmp_path)
|
||||
)
|
||||
conv = tmp_path / ".roboco"
|
||||
conv.mkdir()
|
||||
(conv / "conventions.yml").write_text(
|
||||
"modules:\n - path: lib/special\n purpose: special\n"
|
||||
)
|
||||
svc = get_conventions_service(db_session)
|
||||
await svc.get_map(project) # caches ok1 (lib/special)
|
||||
|
||||
project.head_commit = "bad1"
|
||||
await db_session.flush()
|
||||
(conv / "conventions.yml").write_text("modules: [unterminated\n")
|
||||
degraded = await svc.get_map(project) # falls back to last_good (lib/special)
|
||||
assert any(m.path == "lib/special" for m in degraded.modules)
|
||||
|
||||
# Repair in place at the same bad1 head with a NEW module.
|
||||
(conv / "conventions.yml").write_text(
|
||||
"modules:\n - path: lib/fixed\n purpose: fixed\n"
|
||||
)
|
||||
recovered = await svc.get_map(project)
|
||||
assert any(m.path == "lib/fixed" for m in recovered.modules)
|
||||
assert not any(m.path == "lib/special" for m in recovered.modules)
|
||||
|
||||
|
||||
async def test_baseline_constraints_include_block_rules(
|
||||
db_session: AsyncSession, tmp_path: Path
|
||||
) -> None:
|
||||
|
||||
@@ -634,3 +634,34 @@ async def test_get_dashboard_service_factory(
|
||||
) -> None:
|
||||
svc = get_dashboard_service(db_session)
|
||||
assert isinstance(svc, DashboardService)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_auditor_flags_are_in_memory_and_lost_on_storage_reset(
|
||||
db_session: AsyncSession,
|
||||
) -> None:
|
||||
"""Auditor flags/reports live in a process-singleton in-memory store, NOT
|
||||
the DB (#68 — ``DashboardStorage`` docstring: 'In production, these would be
|
||||
database tables'). A reset (the test/prod-restart analog) drops them. This
|
||||
test pins the documented limitation so a silent persistence regression is
|
||||
caught and the stub is not mistaken for a durable store. Persisting to real
|
||||
tables is a migration + service refactor (a feature), deliberately not
|
||||
half-implemented here.
|
||||
"""
|
||||
reset_storage()
|
||||
svc = DashboardService(db_session)
|
||||
flag = svc.create_flag(
|
||||
CreateFlagParams(
|
||||
severity="high",
|
||||
category="quality",
|
||||
title="stale-claim",
|
||||
description="be-dev-1 heartbeat stale",
|
||||
related_agent_id=uuid4(),
|
||||
)
|
||||
)
|
||||
assert flag.id in get_storage().flags
|
||||
|
||||
# A process restart / test reset drops the in-memory store.
|
||||
reset_storage()
|
||||
assert flag.id not in get_storage().flags
|
||||
assert get_storage().flags == {}
|
||||
|
||||
@@ -254,7 +254,7 @@ async def test_write_doc_creates_new_subfolder_empty(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_doc_updates_existing(docs_setup: dict, tmp_path: Path) -> None:
|
||||
"""When _find_similar_doc returns a path, update path is taken."""
|
||||
"""When _find_similar_doc returns a path AND the filename matches, update."""
|
||||
svc = docs_setup["svc"]
|
||||
existing_path = "backend/api/existing.md"
|
||||
with (
|
||||
@@ -266,7 +266,7 @@ async def test_write_doc_updates_existing(docs_setup: dict, tmp_path: Path) -> N
|
||||
agent_id="be-doc",
|
||||
req=WriteDocInput(
|
||||
task_id=docs_setup["task_id"],
|
||||
filename="example.md",
|
||||
filename="existing.md",
|
||||
doc_type="api",
|
||||
title="New Title",
|
||||
content="# Updated",
|
||||
@@ -311,7 +311,7 @@ async def test_write_doc_update_preserves_existing_metadata(
|
||||
agent_id="be-doc",
|
||||
req=WriteDocInput(
|
||||
task_id=docs_setup["task_id"],
|
||||
filename="example.md",
|
||||
filename="existing.md",
|
||||
doc_type="api",
|
||||
title="New Title",
|
||||
content="# Updated",
|
||||
@@ -323,6 +323,155 @@ async def test_write_doc_update_preserves_existing_metadata(
|
||||
assert doc_ref.updated_by == "be-doc"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_doc_no_collapse_on_different_filename(
|
||||
docs_setup: dict, tmp_path: Path
|
||||
) -> None:
|
||||
"""#35: a similar doc with a DIFFERENT filename must not be overwritten —
|
||||
the agent named a new file, so create it instead of collapsing onto the
|
||||
similar doc's path."""
|
||||
svc = docs_setup["svc"]
|
||||
existing_path = "backend/api/existing.md"
|
||||
with (
|
||||
patch("roboco.services.docs.DOCS_BASE_PATH", tmp_path),
|
||||
patch.object(svc, "_find_similar_doc", AsyncMock(return_value=existing_path)),
|
||||
patch.object(svc, "_index_doc_in_rag", AsyncMock(return_value=None)),
|
||||
):
|
||||
rel_path, _doc_ref, is_update = await svc.write_doc(
|
||||
agent_id="be-doc",
|
||||
req=WriteDocInput(
|
||||
task_id=docs_setup["task_id"],
|
||||
filename="other.md",
|
||||
doc_type="api",
|
||||
title="New Title",
|
||||
content="# New",
|
||||
),
|
||||
)
|
||||
assert is_update is False
|
||||
# A new file is created at the requested filename, NOT the similar doc path.
|
||||
assert rel_path.endswith("other.md")
|
||||
assert rel_path != existing_path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_doc_update_path_containment_checked(
|
||||
docs_setup: dict, tmp_path: Path
|
||||
) -> None:
|
||||
"""#33: the RAG-returned update path is containment-checked — a ``source``
|
||||
that escapes the docs dir (``../../etc/evil.md``) is refused, not written."""
|
||||
svc = docs_setup["svc"]
|
||||
escaping = "../../etc/evil.md"
|
||||
with (
|
||||
patch("roboco.services.docs.DOCS_BASE_PATH", tmp_path),
|
||||
patch.object(svc, "_find_similar_doc", AsyncMock(return_value=escaping)),
|
||||
patch.object(svc, "_index_doc_in_rag", AsyncMock(return_value=None)),
|
||||
pytest.raises(ValidationError),
|
||||
):
|
||||
await svc.write_doc(
|
||||
agent_id="be-doc",
|
||||
req=WriteDocInput(
|
||||
task_id=docs_setup["task_id"],
|
||||
filename="evil.md",
|
||||
doc_type="api",
|
||||
title="Title",
|
||||
content="# x",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_doc_commit_status_skipped_when_no_branch(
|
||||
docs_setup: dict, tmp_path: Path
|
||||
) -> None:
|
||||
"""#34: when there is no task branch to commit onto, the doc still saves to
|
||||
/app/docs and the doc_ref carries ``commit_status='skipped'`` (not a silent
|
||||
None) so the agent knows the repo commit did not happen."""
|
||||
svc = docs_setup["svc"]
|
||||
with (
|
||||
patch("roboco.services.docs.DOCS_BASE_PATH", tmp_path),
|
||||
patch.object(svc, "_find_similar_doc", AsyncMock(return_value=None)),
|
||||
patch.object(svc, "_index_doc_in_rag", AsyncMock(return_value=None)),
|
||||
):
|
||||
_rel, doc_ref, _is_update = await svc.write_doc(
|
||||
agent_id="be-doc",
|
||||
req=WriteDocInput(
|
||||
task_id=docs_setup["task_id"],
|
||||
filename="example.md",
|
||||
doc_type="api",
|
||||
title="Title",
|
||||
content="# Hello",
|
||||
),
|
||||
)
|
||||
# The fixture task has no branch_name → commit is skipped, not silent.
|
||||
assert doc_ref.commit_status == "skipped"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_commit_doc_to_repo_returns_failed_on_git_error(
|
||||
docs_setup: dict,
|
||||
) -> None:
|
||||
"""#34: a git hiccup surfaces as ``failed`` (fail-loud), not a swallowed
|
||||
None — the agent can tell the cell PM the doc did not reach the repo."""
|
||||
svc = docs_setup["svc"]
|
||||
# Give the task a branch + project so the commit path is entered.
|
||||
result = await svc.session.execute(
|
||||
select(TaskTable).where(TaskTable.id == docs_setup["task_id"])
|
||||
)
|
||||
task = result.scalar_one()
|
||||
task.branch_name = "feature/docs"
|
||||
await svc.session.flush()
|
||||
|
||||
fake_git = MagicMock()
|
||||
fake_git.get_workspace = AsyncMock(side_effect=RuntimeError("git boom"))
|
||||
with patch("roboco.services.git.get_git_service", return_value=fake_git):
|
||||
status = await svc._commit_doc_to_repo(
|
||||
"be-doc",
|
||||
WriteDocInput(
|
||||
task_id=docs_setup["task_id"],
|
||||
filename="example.md",
|
||||
doc_type="api",
|
||||
title="Title",
|
||||
content="# Hello",
|
||||
),
|
||||
"api",
|
||||
)
|
||||
assert status == "failed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_commit_doc_to_repo_returns_committed_on_success(
|
||||
docs_setup: dict, tmp_path: Path
|
||||
) -> None:
|
||||
"""#34: a successful repo commit reports ``committed``."""
|
||||
svc = docs_setup["svc"]
|
||||
result = await svc.session.execute(
|
||||
select(TaskTable).where(TaskTable.id == docs_setup["task_id"])
|
||||
)
|
||||
task = result.scalar_one()
|
||||
task.branch_name = "feature/docs"
|
||||
await svc.session.flush()
|
||||
|
||||
fake_git = MagicMock()
|
||||
fake_git.get_workspace = AsyncMock(return_value=tmp_path)
|
||||
fake_git.commit = AsyncMock(return_value={"oid": "abc"})
|
||||
with (
|
||||
patch("roboco.services.git.get_git_service", return_value=fake_git),
|
||||
patch.object(svc, "_write_file", AsyncMock(return_value=None)),
|
||||
):
|
||||
status = await svc._commit_doc_to_repo(
|
||||
str(docs_setup["agent_id"]),
|
||||
WriteDocInput(
|
||||
task_id=docs_setup["task_id"],
|
||||
filename="example.md",
|
||||
doc_type="api",
|
||||
title="Title",
|
||||
content="# Hello",
|
||||
),
|
||||
"api",
|
||||
)
|
||||
assert status == "committed"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _find_similar_doc
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user