mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
df87fcf059
* [sweep] lifecycle: 6 confirmed gaps fixed (cancel-ceo-gate, claim_pr_review gate, needs_team_match, valid_next_verbs narrowing, pr_reviewer unclaim, complete side_effect ordering)
* [chore] logical-gaps: route-layer force gate + privileged-field gate + pre-task audit attribution
tasks.py (5 gaps):
- _HATCH_OVERRIDE_STATES expanded to 7: a privileged PATCH INTO a gate
state (completed/cancelled/awaiting_{qa,documentation,pr_review,
pm_review,ceo_approval}) now requires explicit force — the panel hatch
is no longer a quiet click that drops a task into/out of a human gate.
- _RESURRECT_SOURCE_STATES: a privileged PATCH OUT of a terminal status
(completed/cancelled) resurrects finished work and likewise requires
force, audited as an override.
- _PRIVILEGED_UPDATE_FIELDS gate: a bare task owner (UPDATE_OWN, no
ASSIGN) cannot self-reassign / re-team / re-parent / re-depend /
re-block / rewrite-plan / re-project its task — those structural fields
are PM-gated; the REST surface must not bypass the verb-layer's
reassign/delegate/triage gate. A 403 names the touched fields + the
verb to use instead.
- pre-task create denial: a role that cannot create tasks is now logged
via log_task_creation_denial (distinct task_creation target_type +
attempted payload) instead of a 'N/A' task_id that coerced to NULL and
left the role-escalation attempt unattributable.
audit.py:
- split log_task_action_denial (5-param, under PLR0913) from
log_task_creation_denial (4-param) — the create path has no task_id;
the non-UUID sentinel (N/A) is preserved in details[target_id_raw]
rather than dropped to a NULL target_id indistinguishable from any
other NULL-target denial.
tests:
- test_tasks_routes.py: parametrized admin-override gate (force
required for gate + terminal states, force succeeds).
- test_tasks_route_privileged_fields.py: dev owner 403 on
assigned_to/team/parent_task_id, 200 on dev-facing description.
- test_audit.py: pre-task attribution via log_task_creation_denial +
non-UUID sentinel preservation.
* [chore] logical-gaps: kanban board column coverage + status-class fixes (6 gaps)
models/kanban.py:
- DEV_COLUMNS: cover all 15 lifecycle statuses (was 7; dropped BACKLOG,
PAUSED, VERIFYING, NEEDS_REVISION, AWAITING_PR_REVIEW, AWAITING_PM_REVIEW,
AWAITING_CEO_APPROVAL, CANCELLED). A dev whose task bounced to
needs_revision or sits in a gate used to see their own task vanish.
- PM_COLUMNS: add the gate/revision/paused/cancelled/backlog columns so the
cell PM sees the QA->docs->PR-review->PM-review->CEO chain on its board.
- QA_COLUMNS: drop the 'In Review'->VERIFYING mapping. VERIFYING is the dev's
self-verification (task still with the dev, not with QA); it misrepresented
dev mid-verification as active QA work.
services/kanban.py:
- _build_flat_board: add an 'Other' fallback column for any task whose status
matches no configured column, so total_cards == sum(card_count) and no card
is built-then-silently-dropped (the vanished-card leak).
- get_qa_board: drop VERIFYING from qa_statuses (consistent with the column
change).
- get_documenter_board: scope to task_type=documentation so a dev IN_PROGRESS
code task sharing the cell team no longer appears under 'Gathering'.
- get_main_pm_board_flat: widen the status filter to include PENDING/CLAIMED/
COMPLETED and route those to the incoming/distributed/done columns, which
were structurally always empty under the in-flight-only filter.
tests/integration/test_kanban_service.py: parametrized coverage of every
dropped dev status, PM gate/revision states, QA excludes VERIFYING,
documenter excludes dev code tasks, flat Main PM incoming/distributed/done
populated, and the 'Other' fallback invariant.
* [chore] logical-gaps: lifecycle-enforcement validators + status-class fixes (5 gaps)
enforcement/task_lifecycle.py:
- drop the spurious VERIFYING->awaiting_documentation legacy edge. The
canonical exit is submit_qa -> awaiting_qa -> (qa_pass) ->
awaiting_documentation; the direct edge bypassed the entire QA review hop
(ungated — no role gate existed for it).
- is_waiting_state: add awaiting_pr_review. The PR-review gate parks the PM on
the reviewer; it is a waiting state. The hard-coded set was never updated
when AWAITING_PR_REVIEW was added to the enum, so the gate status was
miscategorized as active.
foundation/_validate_lifecycle.py:
- _check_status_enum_coverage: replace the tautology (STATUS_GRAPH keys every
Status by construction) with a real bidirectional check — every non-terminal
Status is the source of a transition (catches orphan states), and every
source/target referenced is a real Status member (catches stray-string
targets).
- _check_terminal_exits: split the {COMPLETED, CANCELLED} reachability into a
COMPLETED-path requirement + a cancel-exit requirement. The cancel fan-out
made the old check structurally trivial — a status whose sole exit was cancel
passed with no real forward completion path.
- _check_status_enum_parity (new, registered): cross-check spec.Status against
models.base.TaskStatus at import so the ORM column type and the lifecycle
map cannot drift (TaskType had this guard; Status did not).
tests: verifying->awaiting_documentation rejected, self-fail preserved,
awaiting_pr_review is waiting, mutually-disjoint classification invariant,
status enum parity, stray-string-target / orphan-source / cancel-only-exit
validator rejections.
* [chore] logical-gaps: stream-bus poison-pill ACK + dead-letter, periodic reclaim, cancelled-handler marker cleanup (3 gaps)
stream_bus.py:
- _handle_message isolates Event.from_json in its own try/except; an
undecodable payload (unknown EventType, bad UUID/timestamp, malformed
JSON) is dead-lettered then ACKed instead of falling through to the
broad except that only logged — a poison pill stayed pending forever
and re-failed on every reclaim. (gap: stream-bus-malformed-event-poison-pill)
- _reclaim_loop spawned alongside _listen_loop in start_listening (cancelled
in disconnect). XREADGROUP '>' delivers only NEW messages, so a runtime
handler failure left its message pending and unretried until a restart;
the loop re-runs recover_pending every 60s so the idempotency-guarded
replay actually fires. (gap: stream-bus-no-runtime-reclaim-loop)
- _run_handler_guarded marker cleanup catches BaseException so a handler
cancelled mid-flight (asyncio.CancelledError is BaseException-derived
since 3.8) clears its SET-NX marker; otherwise the guard suppressed the
very redelivery that would complete the work. (gap: stream-bus-cancelled-
handler-keeps-idempotency-marker)
TDD: 4 red->green tests in tests/unit/events/test_bus.py.
* [chore] logical-gaps: verb_runner trailing-None side-effect guard + actor_agent_id threading (3 gaps)
_verb_runner.py:
- run_intent skips the side_effects loop when a TRAILING composed action
returned None (its source-status check failed under a concurrent
transition). Previously the loop ran unconditionally on the None task
and _do_push_branch(None)/_do_pr_merge(None) crashed with a
NoneType AttributeError, turning the clean INVALID_STATE the
entry/intermediate guards give into a 500/respawn loop. The trailing
None now flows to the caller's `if task is None` handler. Latent today
(no shipped intent has both a None-capable compose and trailing
side_effects) but the runner is generic. (gap: runner-side-effects-fire-
on-trailing-none-task)
- _do_push_branch / _do_create_pr / _do_create_root_pr forward
actor_agent_id=agent.id into git_service (push_branch / create_pr),
matching _do_pr_merge. Without it, a verb on a task whose assigned_to
was cleared before the side effect falls through to created_by and
pushes from / opens a PR against the wrong workspace.
(gap: side-effect-handlers-drop-actor-agent-id)
- _do_escalate_to_ceo forwards actor_agent_id=agent.id so the
awaiting_ceo_approval audit row attributes to the specific PM/Board
agent. (gap: do-escalate-to-ceo-drops-actor-agent-id)
task.py: escalate_to_ceo gains actor_agent_id param, passed as
audit_agent_id to _validate_and_set_status and recorded as
escalated_by_agent_id in the event payload + log. escalate_to_ceo_for_agent
forwards agent.agent_id.
_impl.py: the main_pm complete->escalate path forwards
actor_agent_id=main_pm_agent_id.
TDD: 5 red->green tests (synthetic trailing-None intent, actor forwarding
for push_branch/create_pr/create_root_pr/escalate_to_ceo) + real-DB audit
test asserting the awaiting_ceo_approval row carries the actor UUID.
Updated 3 board escalate_to_ceo tests to assert the forwarded actor.
* [B-REL] release executor: idempotent half-landed retry + commit-scoped CI + decoupled workflow
Three confirmed gaps in the release fail-closed pipeline (#87/#318/#402):
#87 publish_failed retry duplicates changelog: execute() only short-circuits
on an existing tag. A publish_failed outcome (commit pushed + CI green, no
tag) left no tag, so a retry re-ran apply_version_bumps + write_changelog_entry
(re-inserting the entry above the already-present heading -> duplicate) and
commit_and_push (a second chore(release) commit). Add ReleaseOps
.release_commit_sha(version) detecting a prior release commit on the branch
(clone already at the target version); when present, skip the bump/changelog/
gate/commit pipeline and rejoin the shared CI -> publish tail on the existing
commit. No second commit, no duplicate entry.
#318 wait_for_ci polls branch-latest, not the release commit: a later push to
master during the ~40min wait made the latest run's head_sha != the release
sha forever, exhausting _CI_MAX_POLLS -> false ci_failed on a release whose
own CI was green. Thread head_sha through get_latest_ci_conclusion /
_fetch_latest_ci_run (GitHub actions/runs?head_sha=) so the gate polls the
release commit's own run; a concurrent push can no longer mask it.
#402 release CI gate reuses self_heal_ci_workflow: that setting documents an
empty-string mode for single-workflow repos which, inherited here, degraded
the fail-closed gate to the all-workflows mode git.py itself flags as
unreliable. Add release_ci_workflow (default ci.yml) and _resolve_release_
ci_workflow(); the release gate always resolves a NAMED workflow, never None.
Refactor: bundle the CI-fetch per-project inputs into a _CiRunQuery dataclass
so _fetch_latest_ci_run stays under the arg-count gate; unify the half-landed
path into execute's shared tail (drops a separate _publish_existing, one
return path). TDD red->green; ruff/mypy clean.
* [chore] logical-gaps: a2a service hierarchy gate (typed, unconditional) + persist skill on message row (3 gaps)
create_a2a_notification gated A2A hierarchy only when both ends resolved
(`if from_agent and target_agent:`), so an unattributed (from_agent falsy)
or unresolvable-target request slipped past the hierarchy matrix and
dispatched with from_agent='unknown' / to_agent='' — and a denial came back
as a bare ValueError indistinguishable from the missing-task_id ValueError.
Require both ends present, then validate via the shared typed
validate_a2a_access path (A2AAccessDeniedError + route_hint) so the legacy
notification surface enforces the same who-may-talk-to-whom invariant as the
conversation path.
send() accepts skill= and the gateway callers (qa/doc/pr_gate) pass it
expecting the receiver to learn which capability the message is about, but
send_chat_message never read it from options — silently dropped. Persist a
nullable skill column (migration 054) on a2a_messages, wire it through
send_chat_message + _msg_to_model + the A2AChatMessage model, and fix the
send() docstring (it claimed 'recorded in message metadata').
TDD: 4 red→green (skill recorded on message + surfaces in inbox; permission
denied raises typed A2AAccessDeniedError with route_hint; self-A2A raises
typed; missing from_agent raises instead of silent dispatch). 103 a2a
integration tests green; ruff/mypy clean; migration 054 verified
upgrade/downgrade on throwaway PG.
* [chore] logical-gaps: release-proposal already_published closes proposal + heartbeat-lock-loss cancels execute (2 gaps)
approve() closed the proposal only on status=='published'. A retry that finds
the tag already shipped returns 'already_published' (is_already_published),
so if a prior publish's route commit failed / HTTP 504'd, the proposal stayed
non-terminal forever — every retry returned already_published and never
closed it; only a manual cancel unstuck it. Close on both published and
already_published: the release shipped either way.
_heartbeat_loop returned silently when the lock was no longer owned (a >TTL
Redis outage let the mutex expire mid-execute), leaving executor.execute
running UNGUARDED — a concurrent approve (once Redis returns) could then
acquire the lock and _prepare_release_clone rm -rf the in-flight shared
release clone while the first execute was still mid-run_gate, re-opening the
very rm -rf-clone race the mutex+heartbeat exist to prevent. Run execute as a
task; on lock-loss the heartbeat sets a flag and cancels it, and approve()
turns the CancelledError into a structured 'lock_lost' result (an external
cancellation of approve itself still propagates — distinguished by the flag).
TDD: 2 red→green (already_published → COMPLETED not wedged; heartbeat lock-loss
→ lock_lost + execute cancelled, proposal not completed). 8 concurrency tests
green; ruff/mypy clean.
* [chore] logical-gaps: release approve async dispatch (202) — kill the 40min synchronous HTTP 504
The approve route ran the whole fail-closed execute inline: clone(600s) +
gate(1800s) + CI poll(2400s) + publish(300s) ≈ up to 85min worst case. nginx
(the single :3000 entry point, ~60s read timeout) 504'd long before it
finished, so the CEO's approve always appeared to fail even when the release
succeeded server-side — the structured ReleaseResult was unreachable over the
wire. dispatch_approve spawns the execute in a background task with a fresh
session (built from the request session's engine) and the route returns 202
'accepted' immediately; _INFLIGHT_APPROVES tracks the dispatched task for
observability (self-cleans via done-callback; the Redis mutex still refuses a
double-execute on a second click). The panel already polls GET /proposal every
30s, so it observes the final status (COMPLETED on published/already_published,
else the proposal stays open for retry); the card's approve toast now treats
'accepted' as an info 'dispatched, running in the background' instead of the
old 'Release halted' warning.
TDD: 2 route tests red→green (approve returns 202 'accepted' + the proposal
transitions to COMPLETED / stays PENDING once the background faked execute
completes; the dispatched task is awaited while the executor patch is live).
83 release tests green; ruff/mypy clean; panel typecheck+lint+format+test
green.
* [chore] mcp-servers: normalize exception bodies to Envelope + lift task_id/correlation_id on circuit_open (#232 #359 #57)
flow_server/do_server: the non-404 JSON path returned exception-handler bodies
raw (dict `error` from roboco/generic/http exception handlers, or a 422
`detail` list) — neither is the Envelope wire format the agent is prompted to
trust (string error kind + message + remediate + missing), so on any
service/validation failure the agent got no remediate and flailed until the
breaker tripped. _normalize_exception_envelope lifts the body into a real
Envelope (code -> counted string kind via _classify_dict_error_code, NOT_FOUND
-> not_found, message lifted, remediate synthesized, missing=[]; 422 -> incomplete_input with the validation detail preserved). The synthesized
envelope still flows through the breaker so a 500/422 storm trips it.
_record_and_check_circuit: the circuit_open substitution dropped task_id /
correlation_id from the top level (the SDK's envelope omits them); lift them
from the original rejection so the agent's envelope contract and ops audit-join
of the trip event still work, not just nested in inner.
intake_server._post_event: capture the relay response body under `detail` on
non-success so the grok intake agent gets the real reason (e.g. 'session not in
MegaTask scope' on a 422) instead of an opaque http_422 token with no
remediation.
TDD red->green; ruff + mypy clean; 157 mcp/SDK-breaker tests pass.
* [chore] a2a-routes: authenticate send_message responder + gate cancel task (PM-only) (#116 #423)
send_message took the responder identity from a client-supplied
metadata.from_agent, so any caller could spoof anyone (e.g.
from_agent='ceo') in the task's notes and in the spawn/notification
routed back to the original requester. Stamp the authenticated caller's
slug as the responder instead (CurrentAgentContext).
cancel_task was ungated: no auth dependency and no role check, so any
agent (or any caller) could cancel a task the lifecycle rule reserves to
PM roles (Any -> cancelled: PM roles only) — and the cascade-cancel of
all non-terminal descendants ran with a hardcoded cell_pm role and no
recorded actor. Add require_any_authenticated_agent + a PM-or-above gate,
and thread the authenticated role (into the cascade role gate) and slug
(into the cancellation note) into A2AService.cancel_task.
Tests: send_message ignores a spoofed from_agent and records the
authenticated slug; cancel rejects a developer (403) and a missing auth
header; a PM cancel threads role + slug into the service; the pre-existing
cancel success/already-terminal/not-found tests now run under a PM context
(the success test's body was missing the A2A 'name' field and false-passed
on a 422 — now genuine).
* [chore] work-session-routes: ownership check on mutating routes + stamp merge_pr merged_by from auth (#158 #271)
Every mutating work-session route keyed off session_id alone after the
role gate, so any developer could commit into / abandon / complete a
peer's active session (breaking the single-active-WorkSession invariant
and stranding that task) and any PM could merge any cell's PR — the REST
surface bypassed the verb layer's active-claimant gate entirely. Add a
shared _assert_ownership guard: dev ops require session.agent_id to be
the caller; PM merge_pr requires a cell PM to own the session's task cell
(main PM / CEO / board coordinate every cell), 404 for a missing session.
merge_pr took merged_by from the request body, so any PM could record a
PR merge under another agent's id, corrupting the merge audit trail the
completion/CEO-approval chain and metrics rely on. Drop the body param
and stamp the authenticated caller's agent_id as merged_by (the
MergePRRequest schema is gone with it).
Tests: a second dev's token hitting a peer's /commits and /abandon -> 403
(session left active); a foreign-cell PM -> 403, same-cell PM -> 200; a
spoofed body merged_by is ignored and the persisted row records the PM.
* [chore] ci-watch/dep-update dedupe: normalize git_url + treat empty-string workflow as default (#148 #1267)
The per-repo open-task dedupe filtered ProjectTable.git_url == git_url
(exact), while the orchestrator collapses its poll set by repo_key
(lower / strip trailing '/' / drop '.git'). Two projects whose git_url
differs only by those accidentals (a monorepo's cell-projects, or a
re-registered canonical project) defeated the one-open-task-per-repo
invariant and opened duplicate fix / dep-update tasks. Extract
roboco.utils.converters.repo_key as the single source and match the
dedupe query on its SQL mirror (regexp_replace(rtrim(lower(...)))).
The ci_watch (git_url, workflow) dedupe used func.coalesce(ci_watch_workflow,
default), but SQL COALESCE only substitutes for NULL — a project saved with
ci_watch_workflow='' (reachable via panel/API) yielded coalesce('', default)
= '' != default, so the DB diverged from the engine/orchestrator (which
collapse '' to the default via Python truthiness) and opened a duplicate
fix task every red cycle. Wrap with func.nullif(..., '') so an empty string
collapses to the default too.
Tests: a ''-workflow + NULL-workflow project on one repo dedupe to one task;
git_url accidentals (.git suffix / trailing slash) dedupe across both
ci_watch and dep_update. The orchestrator _repo_key now delegates to repo_key.
* [chore] admin_set_status: attribute the blocked-restore to the admin actor + emit override row (#2176)
admin_set_status taking a BLOCKED task to pending/in_progress with a
pre-block snapshot returned early via _apply_pre_block_restore, which
emitted its audit row with agent_role=None and audit_agent_id=restored_owner
(the pre-block dev) — the admin actor_id/actor_role were dropped entirely.
Because this branch runs with force=false (pending/in_progress aren't hatch
destinations), the distinguishing task.admin_override row (written only on
the non-restore path, gated by force) was never written, so an operator
could silently re-own a blocked task with no trace of who triggered it.
Thread actor_id/actor_role into _apply_pre_block_restore (admin_set_status
passes them with admin_override=True) so the transition audit row attributes
the re-owning to the admin, and emit a task.admin_override row (forced=False,
restore=True) on this branch independent of the force flag. The in-band
unblock(restore=True) path passes no actor and keeps the legacy attribution
(restored owner) with no override row.
Test: admin PATCH status=pending on a BLOCKED task with a snapshot attributes
every audit row to the admin (not the restored dev) and emits the override
row.
* [chore] converters: typed InvalidIdentifierError from require_uuid + log the orchestrator drop (#25)
require_uuid raised a bare ValueError('UUID value cannot be None'), so a
malformed/None identifier propagated as an opaque error callers either let
500 or broad-catch-and-silently-swallow — the orchestrator reaper call site
wrapped it in a bare except-Exception return with NO log, dropping a bad
task_id_str invisibly. Introduce InvalidIdentifierError(ValueError) and
raise it from require_uuid for both None and unparseable input; it stays a
ValueError subclass so existing except-ValueError / except-Exception callers
are unaffected, but typed so a caller can handle a bad identifier distinctly.
The reaper now catches the typed error, logs at warning, and no-ops — the
drop is visible instead of swallowed.
Tests: None and an unparseable string both raise InvalidIdentifierError; it
subclasses ValueError (back-comat).
* [sweep] notification_delivery: list_system_notifications over-fetch-then-slice for pending_ack_only
The SQL limit was applied before the post-fetch 'not fully acked' Python
filter. A window of newer fully-acked ack-required rows filled the limit
and masked older unacked notifications the operator still needs to act on
(the pending-ACK queue silently under-reported; a CEO-approval notification
could be hidden by newer already-acked noise). pending_ack_only now drops
the SQL limit, filters in Python, then slices to limit; the non-pending
branch keeps the SQL limit unchanged.
* [sweep] proactive: drop vestigial code-patterns surface from context package
Code indexing was removed, so _find_code_patterns always returned [] yet
build_context_package still called it, ContextPackage.code_patterns stayed
a live field, _build_summary advertised 'Found N code patterns', and
_count_items counted it — a permanently-empty slot the system claimed to
populate. The dead method, its call, the summary line, and the count
reference are removed. The code_patterns field itself is retained
(always-empty, serialized in to_dict and the optimal route response) for
API/schema back-compat, marked deprecated in its docstring.
* [sweep] migration 052: integration-test the task_cell_projects unique constraint
The UNIQUE(task_id, team) 'one project per cell per task' invariant was
only exercised through SimpleNamespace stubs that never touch a DB
session, so the real Postgres constraint was unverified. If it were
mis-declared or dropped, two same-team rows could coexist and
_resolve_subtask_project would non-deterministically return one, cutting
a subtask's branch/PR against the wrong repo. Adds an integration test
that inserts two same-(task_id, team) rows and asserts IntegrityError on
uq_task_cell_projects_task_team, plus a positive different-teams case.
* [sweep] pr_gate: classify MegaTask root-subtask as root so its root->master PR gets COMMENT (#608)
_post_gate_review_to_pr identified a root->master PR by absence of a
parent_task_id. A MegaTask root-subtask opens its own root->master PR into
the project's master (submit_root, parent='master') but carries
parent_task_id=umbrella, so is_root was False and the gate posted APPROVE
(pr_pass) / REQUEST_CHANGES (pr_fail) instead of COMMENT. The APPROVE could
satisfy a single-approval master branch-protection rule and let a non-CEO
merge via the GitHub UI before the CEO, against the documented invariant
that only the CEO acts on master. is_root now also covers
is_batch_root_subtask (batch_id set + parented); a non-batch cell-PM
coordination root keeps batch_id=None so it stays a cell->root PR
(APPROVE/REQUEST_CHANGES). Extends the _task test helper with a batch_id
kwarg.
* [sweep] enforcement: complete the status-class partition + coverage invariant (#247)
is_waiting_state already covered awaiting_pr_review (the primary fix), but
the doc's coverage invariant was missing: backlog and pending fell through
ALL three predicates (terminal/active/waiting), so a future enum addition
could silently land in no category. is_waiting_state now also covers
pending (waiting for a claim) and backlog (waiting on PM activation), so
is_terminal_state / is_active_state / is_waiting_state partition the whole
Status enum. Adds test_status_classification_covers_every_enum_member
asserting every Status member is classified by exactly one predicate, so
an enum addition that drifts the partition fails the build.
* [chore] test-suite: unblock the quality gate (mypy + 2 behavior fixes)
12 mypy errors across 5 test files: drop banned type:ignore comments
(lifecycle_spec monkeypatch uses cast(Any, ...); the ignores were unused),
wrap SQLAlchemy-typed ids with cast(UUID, ...) for AgentContext / WorkSession
args (AgentTable.id is Mapped[sqla UUID], not uuid.UUID), annotate **kw: Any,
and cast(Any, svc) for a method-assignment mock.
test_cancel_descendants_cascades_for_authorized_pm: the child was parked in
awaiting_ceo_approval, which the spec gates to CEO-only cancel
(lifecycle.py:378-389) — a cell_pm cascade correctly refuses it (the #103
refuse path). Use a PM-cancelable in_progress child so the positive-cascade
assertion holds; the refuse case is already covered by its sibling test.
test_a2a_message_auth: /message/send now resolves the authenticated
responder slug via get_agent_context (a DB lookup, #116). This is a DB-free
unit test of the token gate + route body, so stub get_agent_context in the
fixture — the gate (require_any_authenticated_agent) still runs real and
401s on a missing/forged token before that dependency resolves.
* [chore] complexity: split 5 C-rank blocks to <=B for the xenon gate
No behavior change; each C-rank function factored into a helper so the
complexity gate (xenon --max-absolute B) holds.
- lifecycle.can_invoke_action: extract the team-match check into
_check_team_match.
- a2a.cancel_task: extract _status_value_of + _apply_cancel_note.
- task._apply_pre_block_restore: extract _restore_block_ownership (status/
owner restore + snapshot clear) and _emit_admin_override_audit (#2176).
- release_proposal.approve: extract _finalize_release_lock (heartbeat/
execute cancel + mutex release) out of the finally.
- kanban.get_main_pm_board_flat: dict-dispatch the column routing instead
of a 7-branch if/elif ladder (status wins over team; in-flight + no cell
team falls through to Coordination, #196).
* [chore] lifecycle artifacts: regenerate to match the spec (foundation-check)
The rendered artifacts (docs/rag/lifecycle, panel/lib/lifecycle.json, the
_generated role-prompt fragments) had drifted from the spec — the prior
sweep commits (cancel-CEO gate, claim_pr_review preconditions, pr_reviewer
unclaim, complete merge-first ordering) changed spec data without
regenerating, and the foundation-check render+diff stage never ran because
mypy failed earlier in the gate. make foundation-check now passes.
* [fix] chat: wire live message delivery end-to-end (MESSAGE_SENT)
send_message persisted messages but never broadcast them, there was no
MESSAGE_SENT event type or bridge forwarder, and the panel session view
had no websocket subscription — the live chat path was dead end-to-end.
- add EventType.MESSAGE_SENT and publish it best-effort on every persisted
send (a bus outage logs, never rolls back the durable row)
- bridge _handle_message_event forwards to /ws/sessions/{id} and
/ws/channels/{id}; subscribe it in register_websocket_bridge_handlers
- panel useSessionStream subscribes the session view; the page invalidates
the transcript + session-detail queries on each message.new so the held
(staleTime Infinity) views refresh live without the manual Refresh
* [fix] chat: return session task_links in one read; drop panel N+1
GET /sessions/{id} ran a bare select and session_to_response omitted
task_links, so it always returned them empty — the panel worked around it
with a triple-fetch (get session, get-tasks-for-session which re-fetched
the same endpoint, then a task GET per link), and the links never showed.
- add get_session_with_links(_or_raise) that eager-loads task_links -> task
- add session_to_response_with_links; GET /sessions/{id} uses both
- panel useSession now relies on the single populated response; remove the
dead getTasksForSession + per-task fetch and the unused tasksApi import
* [fix] chat: validate reply_to against the effective session; guard closed-session composer
Posting to a closed session transparently redirects the message to the
group's active session (intended for agents holding stale refs), but
reply_to was validated against the requested session, not the one the
message lands in — letting a cross-session reply slip through — and the
panel silently posted there too, so the message vanished from the view.
- validate reply_to against session.id (the effective, possibly-redirected
session), not req.session_id
- panel: render a "session is closed" notice instead of the composer for a
non-active session; if a send still lands elsewhere (stale status), toast
that it went to the active session rather than letting it appear to vanish
* [fix] chat: close session/group/message read IDOR; fix doubled 404s
get_session and the messages-list took an agent id but never used it, and
get_group took none at all — any authenticated agent could read any private
channel's group, session, and message transcripts. Three NotFoundError sites
also passed a full sentence as resource_type, yielding "... not found not found".
- add require_group_read_access / require_session_read_access (channel
member / silent observer / privileged, mirroring list_group_sessions_for_agent)
and get_session_with_links_for_agent; enforce on GET /sessions/{id},
GET /sessions/{id}/tasks, GET /messages, GET /groups/{id} (-> 403 on deny)
- fix the three doubled-404 sites to the NotFoundError(resource_type, resource_id) form
Also folds two gate fixes for the prior chat commits: cast session.id to UUID
for the reply_to validation, and ruff import/format touch-ups.
Note: POST /messages intentionally still skips the channel write-ACL on the
HTTP (human-CEO/panel) path — the CEO is not in writers for 8/11 channels, so
enforcing it there would block the panel; the gateway/agent path enforces it.
* [fix] secretary: harden live chat — stuck spinner, mid-reply clobber, reload
The Secretary live chat had three live-behaviour bugs: a dropped SSE
connection left a permanent "thinking…" spinner (openStream set no
transport-error handler, so the no-data error Event was swallowed by the
JSON-parse guard and streaming never reset); sending mid-reply wiped the
accumulation buffer and pushed a user message without guarding the in-flight
turn, abandoning/duplicating the reply; and the chat lived only in React
state, so a reload wiped it.
- route the dual-purpose `error` listener: server-sent JSON → handleEvent,
transport error (no data) → reset streaming, surface a notice, close stream
- guard send while streaming (streamingRef); disable the composer Send/Enter
while a reply is in flight
- persist sessionId + messages to localStorage (TTL'd) and, on mount, restore
+ re-attach the stream once the backend confirms the session is still alive
(mirrors the intake/prompter durability)
* [chore] groups: extract group-read helper to keep module rank A
The get_group IDOR access-check added try/except branches that tipped the
module to xenon rank B. Extract the service-error→HTTP mapping into a small
helper so get_group stays lean and the module is rank A again (behaviour
unchanged; covered by the groups route tests).
* [fix] chat: correct panel session-task mutation endpoints
linkTask/unlinkTask posted to /add-task and /remove-task (with a body), but
the backend exposes POST /sessions/{id}/tasks and DELETE
/sessions/{id}/tasks/{task_id} (path param) — so every call 404'd. updateTaskLink
targeted /update-task, a route that does not exist at all. Point linkTask and
unlinkTask at the real routes and drop the phantom updateTaskLink. All three were
unused, so no behaviour changes today — this removes a latent 404 trap.
* [docs] chat: document live message delivery (MESSAGE_SENT / message.new)
Document the live transcript-update path the chat-subsystem fixes wired:
- docs/api/websockets.md: add the message.new event-types row (carried on
/ws/sessions + /ws/channels from EventType.MESSAGE_SENT) and note the
forwarder sets type:"message.new"
- docs/panel/communications-and-journals.md: the session transcript updates
live; a closed session is read-only (composer disabled)
- CLAUDE.md: name message.new on the per-resource streams and make
MESSAGE_SENT the worked example of the add-a-live-event recipe
The internal roboco_map slices (gitignored) were updated in place to match.
* [docs] reconcile published docs with code since v0.13.0
Drift caught by the doc-reconciliation pass (all verified against HEAD):
- CLAUDE.md + rag: pr_reviewer gained the unclaim verb (16b71be8)
- rag permissions/task-states/task-tools: awaiting_ceo_approval -> cancelled is
CEO-only, not PM+CEO (16b71be8 cancel-ceo-gate; lifecycle.py:373-382)
- deploy/env-reference: ROBOCO_APP_VERSION default 0.9.0 -> 0.14.0 (config.py:31);
add ROBOCO_RELEASE_CI_WORKFLOW row (2759edf7, config.py:454)
- deploy/data-and-migrations: 44->54 revisions, head 054_a2a_message_skill
- optional/autonomous-maintenance: CI-watch dedupe is per (repo, workflow) (d34bc1a7)
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
574 lines
47 KiB
Markdown
574 lines
47 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Licensing
|
||
|
||
RoboCo is licensed under **AGPL-3.0** (see `LICENSE`). Copyright (c) 2026 Renzo Franceschini. Do NOT reintroduce an MIT or other license reference anywhere (README, headers, package metadata) — the project is AGPL.
|
||
|
||
Contributions require a signed **Contributor License Agreement** (`CLA.md`), automated via the CLA Assistant workflow (`.github/workflows/cla.yml`). The CLA preserves the option to dual-license / offer a commercial edition later; keep copyright assignment language intact. See `CONTRIBUTING.md`.
|
||
|
||
## Project Overview
|
||
|
||
**RoboCo** is an AI Agentic Company - a virtual organization of 25 AI agents + 1 human CEO, designed to operate as a complete software development workforce. The system implements a structured organizational hierarchy with formal communication protocols, task management, and quality controls.
|
||
|
||
### Core Architecture
|
||
|
||
```
|
||
CEO (Renzo - Human)
|
||
|
|
||
+-- Intake (on-demand interviewer: chats only with the CEO to draft a task)
|
||
+-- Secretary (on-demand chief-of-staff: reads company state, runs gated CEO directives)
|
||
+-- PR Reviewer (read-only: the main reviewer — inbound external/fork + internal PRs, and the root→master in-path gate)
|
||
|
|
||
+-- Board (3 agents)
|
||
+-- Product Owner
|
||
+-- Head of Marketing
|
||
+-- Auditor (silent observer, reports to CEO)
|
||
|
|
||
+-- Main PM (coordinates all cells)
|
||
|
|
||
+-- Backend Cell (6 agents: 2 Devs, 1 QA, 1 PM, 1 Documenter, 1 PR Reviewer)
|
||
+-- Frontend Cell (6 agents: 2 Devs, 1 QA, 1 PM, 1 Documenter, 1 PR Reviewer)
|
||
+-- UX/UI Cell (6 agents: 2 Devs, 1 QA, 1 PM, 1 Documenter, 1 PR Reviewer)
|
||
```
|
||
|
||
### Hardware Infrastructure
|
||
|
||
- **Olares One (Powerhouse)**: Intel Ultra 9 + RTX 5090, runs Claude Code instances and AI inference - NOT YET ARRIVED
|
||
- **UGREEN NAS (Warehouse)**: 36TB RAID6, 128GB RAM, hosts PostgreSQL, Redis
|
||
- **Pi Cluster (Operations)**: Monitoring, notifications, smart home
|
||
|
||
## Development Standards
|
||
|
||
### Python (Backend)
|
||
```bash
|
||
# Package manager
|
||
uv
|
||
|
||
# Before any commit
|
||
uv run ruff format .
|
||
uv run ruff check .
|
||
uv run mypy roboco/
|
||
uv run pytest
|
||
|
||
# Coverage target: 80%
|
||
```
|
||
|
||
### TypeScript (Frontend)
|
||
```bash
|
||
# Package manager
|
||
pnpm
|
||
|
||
# Before any commit
|
||
pnpm format
|
||
pnpm lint
|
||
pnpm typecheck
|
||
pnpm test
|
||
|
||
# Coverage target: 80%
|
||
```
|
||
|
||
## Technology Stack
|
||
|
||
| Layer | Technology |
|
||
|-------|------------|
|
||
| API Framework | FastAPI |
|
||
| Database | PostgreSQL + asyncpg |
|
||
| Vector Store | PostgreSQL + pgvector (in-house engine) |
|
||
| RAG Engine | in-house (asyncpg + pgvector, hybrid retrieval) |
|
||
| Cache/Queue | Redis |
|
||
| Container Runtime | Docker + Docker Compose |
|
||
| Cloud LLM | Claude API (claude-opus-4-6) + xAI Grok (official `grok` CLI, SuperGrok subscription) |
|
||
| Local LLM | Ollama (glm-5.2:cloud for RAG/hybrid retrieval) |
|
||
| Embeddings | qwen3-embedding:0.6b (1024 dim) |
|
||
| Frontend | Next.js 16 + TypeScript + Tailwind + Radix UI (in `panel/`) |
|
||
| Edge / Proxy | nginx (single entry point on port 3000) |
|
||
|
||
## Multi-Agent Workspace Structure
|
||
|
||
Each agent gets their own git clone of a project, enabling parallel development without conflicts:
|
||
|
||
```
|
||
{ROBOCO_WORKSPACES_ROOT}/ # Default: /data/workspaces
|
||
+-- {project-slug}/
|
||
+-- {team}/
|
||
+-- {agent-slug}/
|
||
+-- [git repository]
|
||
```
|
||
|
||
**Example:**
|
||
```
|
||
/data/workspaces/
|
||
+-- roboco/
|
||
+-- backend/
|
||
| +-- be-dev-1/ # be-dev-1's workspace
|
||
| +-- be-dev-2/ # be-dev-2's workspace
|
||
+-- frontend/
|
||
+-- fe-dev-1/
|
||
+-- fe-dev-2/
|
||
```
|
||
|
||
Note: the Next.js control panel now lives at `roboco/panel/` inside this repo (no longer a separate `roboco-panel` project or workspace).
|
||
|
||
**Key Configuration (roboco/config.py):**
|
||
- `ROBOCO_WORKSPACES_ROOT`: Root directory for workspaces (default: `/data/workspaces`)
|
||
- `ROBOCO_WORKSPACE_AUTO_CLONE`: Auto-clone repos on first access (default: `true`)
|
||
- `ROBOCO_WORKSPACE_CLONE_TIMEOUT`: Clone timeout in seconds (default: `300`)
|
||
|
||
On a Python workspace, `WorkspaceService` runs `uv sync --extra dev` (not plain `uv sync`) so the clone's `.venv` carries the full gate toolchain (ruff/mypy/xenon/pytest) — the lint/type/complexity tools live in the `dev` **extra**, which plain `uv sync` skips. Without it an agent's `make quality` fails on `ruff: command not found` and the agent can't gate its own work.
|
||
|
||
Because the clone is shared across a dev's tasks, a **fresh claim** git-resets the workspace to a clean tree (`git reset --hard`) before checking out the new task's branch — discarding abandoned uncommitted cruft from a finished task while preserving all commits and the gitignored `.venv`. A resume short-circuits before this, so committed work is never reset.
|
||
|
||
## Git Workflow
|
||
|
||
### Branch Naming Convention
|
||
|
||
Branch names follow the pattern: `{type}/{team}/{task-hierarchy}`
|
||
|
||
**Types:** `feature`, `bug`, `chore`, `docs`, `hotfix`
|
||
|
||
**Task Hierarchy:** Uses `--` separator (not `/`) to avoid git ref conflicts.
|
||
|
||
**Examples:**
|
||
- Root task: `feature/backend/ABC12345`
|
||
- Subtask: `feature/backend/ABC12345--DEF67890`
|
||
- Sub-subtask: `feature/backend/ABC12345--DEF67890--GHI11111`
|
||
|
||
### Commit Format
|
||
|
||
Commits are automatically prefixed with the task ID:
|
||
|
||
```
|
||
[{task-id[:8]}] {message}
|
||
```
|
||
|
||
**Example:**
|
||
```
|
||
[ABC12345] Add user authentication endpoint
|
||
```
|
||
|
||
### Work Sessions
|
||
|
||
When a developer claims a task, a **WorkSession** is created that tracks:
|
||
- Branch name and base/target branches
|
||
- All commits made during the session
|
||
- Files modified
|
||
- PR number/URL when created
|
||
- Merge status and who merged
|
||
|
||
A task has at most **one active WorkSession**: re-claiming a task (pool release, reaper unclaim, escalation redirect) supersedes any prior agent's stale active session, enforced both at the service layer and by a DB partial-unique index (migration 047). Without it, duplicate active sessions made the one-row active lookup raise and crashed the claim/plan flow into a respawn loop.
|
||
|
||
A developer's clone is shared across all their tasks, so push and PR-head operate on the task's **recorded branch by name**, independent of the clone's current checkout — fixing the `BRANCH_MISMATCH` / "No commits between" failures when the clone was parked on a later task's branch. A missing local task-branch ref is first recovered from `origin/<branch>` before the push-by-name.
|
||
|
||
### Git Credentials
|
||
|
||
Git authentication is managed **per-project** through encrypted GitHub PATs:
|
||
|
||
- **Each project stores its own git token** - no global fallback
|
||
- **Tokens are encrypted at rest** using Fernet symmetric encryption
|
||
- **API never exposes tokens** - only returns `has_git_token: boolean`
|
||
- **Self-service via UI** - users set/update tokens in project settings
|
||
|
||
**Project fields:**
|
||
| Field | Description |
|
||
|-------|-------------|
|
||
| `git_token_encrypted` | Fernet-encrypted GitHub PAT (DB column) |
|
||
| `has_git_token` | Boolean indicator for API responses |
|
||
|
||
**Token flow:**
|
||
1. User creates project in UI, enters GitHub PAT
|
||
2. Token encrypted and stored in `projects.git_token_encrypted`
|
||
3. WorkspaceService decrypts token when cloning repos
|
||
4. GitService decrypts token for PR operations (gh CLI)
|
||
|
||
**HTTPS URLs require tokens** - attempting to clone without a token will raise `WorkspaceError`.
|
||
|
||
## Task Lifecycle
|
||
|
||
### Task States
|
||
|
||
The complete task lifecycle is defined in `roboco/foundation/policy/lifecycle.py` (`roboco/enforcement/task_lifecycle.py` is a backwards-compat shim over it):
|
||
|
||
```
|
||
backlog -> pending -> claimed -> in_progress -> [blocked|paused] -> verifying
|
||
| |
|
||
v v
|
||
awaiting_qa <------------------+ awaiting_documentation
|
||
| (needs_revision) | |
|
||
v | v
|
||
awaiting_documentation --------+ awaiting_pm_review
|
||
| |
|
||
v v
|
||
awaiting_pm_review awaiting_ceo_approval
|
||
| |
|
||
v v
|
||
completed completed
|
||
```
|
||
|
||
**In-path PR-review gate** (`awaiting_pr_review`): each assembled PR is reviewed before the PM merges. The cell PM's `submit_up` opens the cell→root PR and the Main PM's `submit_root` opens the root→master PR; both enter `awaiting_pr_review`, where a reviewer `pr_pass`es it on to `awaiting_pm_review` or `pr_fail`s it back to `needs_revision` — the merge-level reject the PM otherwise lacks. Leaf dev tasks and branchless coordination roots skip the gate.
|
||
|
||
**States:**
|
||
| State | Description |
|
||
|-------|-------------|
|
||
| `backlog` | PM setup phase - dependencies or session setup needed |
|
||
| `pending` | Ready for work - orchestrator can spawn agents |
|
||
| `claimed` | Agent has locked the task |
|
||
| `in_progress` | Active development |
|
||
| `blocked` | External dependency blocking progress |
|
||
| `paused` | Temporarily stopped (can resume) |
|
||
| `verifying` | Self-verification by developer |
|
||
| `needs_revision` | QA or CEO requested changes |
|
||
| `awaiting_qa` | Submitted for QA review — PR must already exist |
|
||
| `awaiting_documentation` | Documentation phase — PR already open from pre-QA; doc writes docs |
|
||
| `awaiting_pr_review` | In-path PR-review gate: a reviewer checks the assembled cell→root / root→master PR before the PM merges (assembled, PR-bearing tasks only) |
|
||
| `awaiting_pm_review` | Docs complete, PM reviews + merges |
|
||
| `awaiting_ceo_approval` | Major tasks escalated for CEO final approval |
|
||
| `completed` | Terminal state - work done and merged |
|
||
| `cancelled` | Terminal state - work cancelled |
|
||
|
||
### Role-Based Transitions
|
||
|
||
All status transitions are validated through the enforcement layer. Key restrictions:
|
||
|
||
| Transition | Allowed Roles |
|
||
|------------|---------------|
|
||
| `backlog` → `pending` (activate) | PM roles only |
|
||
| `pending` → `claimed` (claim) | Role must match task type (QA for awaiting_qa, etc.) |
|
||
| `claimed` → `pending` (unclaim) | Assignee or PM |
|
||
| `awaiting_qa` → `awaiting_documentation` (pass) | QA only |
|
||
| `awaiting_qa` → `needs_revision` (fail) | QA only |
|
||
| `awaiting_documentation` → `awaiting_pm_review` | Documenter or Developer (parallel completion) |
|
||
| `in_progress` → `awaiting_pr_review` (submit_up / submit_root) | PM roles (opens the assembled cell→root / root→master PR) |
|
||
| `awaiting_pr_review` → `awaiting_pm_review` (pr_pass) | PR reviewer only |
|
||
| `awaiting_pr_review` → `needs_revision` (pr_fail) | PR reviewer only |
|
||
| `awaiting_pm_review` → `completed` | PM roles only |
|
||
| `awaiting_pm_review` → `awaiting_ceo_approval` | PM roles only |
|
||
| `awaiting_ceo_approval` → `completed/needs_revision/cancelled` | CEO only |
|
||
| Any → `cancelled` | PM roles only |
|
||
|
||
**Unclaim Operation**: Agents can release claimed tasks back to the pool using `unclaim()`. This transitions `claimed` → `pending` and optionally reassigns to another agent.
|
||
|
||
**Board never owns a coordination root**: a Board role (Product Owner / Head of Marketing) is never assigned a Main-PM coordination root (delivery root or MegaTask root-subtask) via escalation or reassignment — Board roles have no `unblock` verb, so such a hand-off would deadlock. The transition is diverted to the pool for a role-matched Main-PM reclaim.
|
||
|
||
### Git Integration Requirements
|
||
|
||
All tasks follow git workflow. PR is created BEFORE QA review (not after) so QA can review the real PR diff on GitHub and downstream PM/CEO approval chain off a PR that already exists:
|
||
|
||
1. **claimed -> in_progress**: `branch_name` is auto-set on claim (hierarchical branches)
|
||
2. **verifying -> awaiting_qa** (submit-qa): Requires `self_verified`, `commits`, `pr_number` (PR open), and at least one `progress_updates` entry
|
||
3. **awaiting_qa -> awaiting_documentation** (pass-qa): Requires `pr_number` and substantive QA notes
|
||
4. **awaiting_documentation -> awaiting_pm_review**: Requires `docs_complete=True` (PR already exists from step 2 above)
|
||
5. **awaiting_pm_review -> awaiting_ceo_approval**: Must have `pr_number` set and all subtasks in a terminal state
|
||
|
||
### CEO Approval Workflow
|
||
|
||
Major tasks are escalated to CEO for final approval:
|
||
1. PM reviews and approves, escalates to `awaiting_ceo_approval`
|
||
2. CEO can:
|
||
- **Approve**: Merges PR, task -> `completed`
|
||
- **Request changes**: Task -> `needs_revision`
|
||
- **Cancel**: Task -> `cancelled`
|
||
|
||
## Data Models
|
||
|
||
### Core Models (roboco/models/)
|
||
|
||
| Model | Purpose |
|
||
|-------|---------|
|
||
| `Task` | Atomic unit of work with acceptance criteria |
|
||
| `Project` | Git repository configuration and CI/CD commands |
|
||
| `WorkSession` | Links agent work to task, tracks branch/commits/PR |
|
||
| `Agent` | AI agent with role, team, capabilities |
|
||
| `Session` | Communication session with messages |
|
||
| `Channel` | Team communication channel |
|
||
| `Message` | Extracted message from agent streams |
|
||
| `Notification` | Formal notification requiring acknowledgment |
|
||
| `Journal` | Agent personal log for reflections/learnings |
|
||
|
||
### Task Model Key Fields
|
||
|
||
```python
|
||
# Git configuration (all tasks follow git workflow)
|
||
task_type: TaskType # code, documentation, research, planning, design, administrative
|
||
project_id: UUID # Project this task works on (required)
|
||
branch_name: str # Branch for this task (auto-created on claim)
|
||
work_session_id: UUID # Active work session
|
||
|
||
# PR tracking (parallel execution in awaiting_documentation)
|
||
pr_number: int # GitHub/GitLab PR number
|
||
pr_url: str # Full URL to PR
|
||
docs_complete: bool # Documenter has finished
|
||
pr_created: bool # Developer has created PR
|
||
|
||
# Commits linked to task
|
||
commits: list[CommitRef] # All commits made for this task
|
||
```
|
||
|
||
## Communication Model
|
||
|
||
**Communication** = constant stream (always flowing, logged, observed) **Notifications** = formal signals (require acknowledgment, sent by PMs/Board only)
|
||
|
||
### Channel Structure
|
||
- Cell channels: `#backend-cell`, `#frontend-cell`, `#uxui-cell`
|
||
- Cross-cell: `#dev-all`, `#qa-all`, `#pm-all`, `#doc-all`
|
||
- Management: `#main-pm-board`, `#board-private`
|
||
- Special: `#announcements` (read-only except Board/Main PM), `#all-hands`
|
||
|
||
The Auditor has silent read access to ALL channels.
|
||
|
||
Agent learnings (`note` scope='learning') broadcast as knowledge-share notifications only to other **agents** — the human / human-driven roles (CEO, prompter, secretary) are excluded, since agent knowledge-sharing is noise in a human's inbox.
|
||
|
||
## Key Principles
|
||
|
||
1. **Everything is a task** - All work is tracked and documented
|
||
2. **No work without a task** - Create task record first
|
||
3. **No task without acceptance criteria** - How do we know it's done?
|
||
4. **No closure without documentation** - Future agents need context
|
||
5. **Communication is constant** - Stream reasoning, log everything
|
||
6. **State is sacred** - If interrupted, state must be recoverable
|
||
7. **The Auditor sees all** - Quality monitored silently
|
||
8. **Commits linked to tasks** - Every commit references its task ID
|
||
9. **CEO approves major changes** - Escalation path for important work
|
||
|
||
## Agent Gateway
|
||
|
||
Agents do not call the API or per-domain MCP tools directly. They go through two thin MCP servers (`roboco-flow`, `roboco-do`) backed by the server-side **Choreographer** in `roboco/services/gateway/`. The Choreographer composes the existing services (TaskService, JournalService, GitService, etc.) into intent-verb sequences. Tracing, claim-locking, evidence assembly, and remediation hints are all centralized there.
|
||
|
||
Each agent gets a **spawn manifest** at `/app/tool-manifest.json` listing the verbs its role is allowed to call. The orchestrator builds the manifest from `roboco/services/gateway/role_config.py` and mounts it read-only into the agent container.
|
||
|
||
### Verb surface (canonical source: `lifecycle.intents_for_role`; every role also gets `i_am_idle`)
|
||
|
||
| Role | Flow verbs (beyond `i_am_idle`) |
|
||
|---------------|--------------------------------------------------------------------------------------------------|
|
||
| developer | `give_me_work`, `i_will_work_on`, `open_pr`, `i_am_done`, `i_am_blocked`, `resume`, `sync_branch`, `unclaim` |
|
||
| qa | `give_me_work`, `claim_review`, `pass_review`, `fail_review`, `i_am_blocked`, `resume`, `unclaim` |
|
||
| documenter | `give_me_work`, `claim_doc_task`, `i_documented`, `i_am_blocked`, `resume`, `unclaim` |
|
||
| cell_pm | `give_me_work`, `i_will_plan`, `delegate`, `complete`, `submit_up`, `triage`, `unblock`, `escalate_up`, `reassign`, `resume`, `unclaim` |
|
||
| main_pm | `give_me_work`, `i_will_plan`, `delegate`, `complete`, `submit_root`, `triage`, `triage_all`, `unblock`, `escalate_up`, `escalate_to_ceo`, `resume`, `unclaim` |
|
||
| pr_reviewer | `give_me_work`, `claim_pr_review`, `post_pr_review` (inbound external/fork PRs), `claim_gate_review`, `pr_pass`, `pr_fail` (in-path assembled-PR gate), `unclaim` |
|
||
| product_owner | `triage`, `escalate_to_ceo` |
|
||
| head_marketing| `triage`, `escalate_to_ceo` |
|
||
| auditor | `triage` (read-only — no `say`/`dm`) |
|
||
| prompter | (none beyond `i_am_idle` — not a delivery-lifecycle role; intake interviewer, human-only) |
|
||
| secretary | (none beyond `i_am_idle` — human-only chief-of-staff; reads company state + runs gated CEO directives) |
|
||
|
||
Content tools (do_server) — most roles: `commit`, `note`, `say`, `dm`, `evidence`. Delivery roles (developer / qa / documenter / cell_pm / main_pm) also get `draft_playbook` (draft a curated playbook for the KB). Auditor is restricted to `note` (scope=reflect) + `evidence`, plus the playbook-curation verbs `approve_playbook` / `reject_playbook` / `archive_playbook` (a bounded, deliberate expansion — KB curation, not agent comms, so its no-`say`/no-`dm` restriction holds). The `pr_reviewer` posts its change-request on the PR itself (no agent comms). The `prompter` (intake) and `secretary` are restricted to `note` + `evidence` — human-only, no `say`/`dm`/`notify`. The `note`/journal write returns as soon as the entry is persisted; RAG indexing (Ollama embedding) runs fire-and-forget, so the tool no longer times out under concurrent load.
|
||
|
||
### MCP servers running per agent container
|
||
|
||
| Server | Purpose |
|
||
|----------------------|----------------------------------------------------------------------|
|
||
| `roboco-flow` | Intent verbs (give_me_work, i_am_done, claim_review, complete, ...) |
|
||
| `roboco-do` | Content tools (commit, note, say, dm, evidence) |
|
||
| `roboco-git-readonly`| Read-only git: status, log, diff, branches |
|
||
| `roboco-optimal` | RAG: `roboco_ask_mentor`, `roboco_kb_search` |
|
||
| `roboco-docs` | Project docs file management (selected roles) |
|
||
|
||
Every verb returns a standardized **Envelope**:
|
||
- ok: `{status, task_id, next, evidence?, context_briefing}`
|
||
- error: `{error, message, remediate, missing}`
|
||
|
||
The `next` field tells the agent what to call next; the `remediate` field on errors tells them exactly how to fix and retry. Agents should not guess state — trust the response. The verb runner re-checks the task after each composed atomic action and, on a concurrent mid-verb state change, fails fast with a clean `INVALID_STATE` (re-fetch + re-issue) rather than crashing on a `None` dereference.
|
||
|
||
## Agent Providers
|
||
|
||
Agent backends are pluggable. `roboco/llm/providers/` defines an `AgentProvider` lifecycle ABC (`base.py`) and a `ProviderRegistry` keyed by `ModelProvider` (`registry.py`), with `ClaudeCodeProvider` (default) and `GrokCliProvider`. The orchestrator resolves a provider at spawn from the agent's `ModelProvider`; when no dedicated provider is registered it falls back to the built-in Claude Code spawn. `ModelProvider` (`roboco/models/base.py`) is `ANTHROPIC` (default), `GROK`, `LOCAL`, `OLLAMA_CLOUD`, `OPENAI` (reserved). The seam is additive: only `GROK` routes through `GrokCliProvider`; Anthropic / Ollama Cloud / self-hosted spawns are unchanged, and every provider gets the same MCP gateway + tool-manifest wiring by construction.
|
||
|
||
**Grok runtime.** `GROK` agents run xAI's official `grok` CLI (model `grok-build`) authenticated by a **SuperGrok subscription**, not a metered API key — so a Grok workforce can't stall mid-task on out-of-credits. The host `~/.grok/auth.json` is mounted **read-only** into each agent (`GrokCliProvider._append_grok_auth_mount`; `ROBOCO_HOST_GROK_DIR` is the host mount source, set up once with `grok login`). It reaches parity with the Claude path by construction: same MCP gateway + manifest, per-role tool-removal and git-operation deny rules, a prompt-injection guard on the task prompt, headless tool auto-approval, and per-agent token/cost capture from the grok session store. It covers both one-shot delivery roles and the interactive Intake (Prompter) and Secretary chats (per-turn `grok -p` with session resume).
|
||
|
||
**Token auto-refresh.** The grok access token has a fixed ~6h server-set TTL and the CLI cannot refresh it headlessly — on an expired token it hangs forever at an interactive login prompt. The orchestrator mints a fresh token from the offline-access refresh token (xAI's OIDC `refresh_token` grant) before expiry and rewrites the shared `auth.json` in place (`roboco/llm/providers/grok_auth.py` `refresh_if_stale`, run once per dispatch tick; the orchestrator's `~/.grok` mount is read-write so it can rewrite it). As a backstop the agent entrypoint runs `python -m roboco.llm.providers.grok_auth --check` and refuses to start (exit 78) on a missing/expired token instead of hanging.
|
||
|
||
## Self-Healing & Feature Flags
|
||
|
||
**Self-healing CI loop (default-off).** RoboCo can watch its own repository's CI (a single named workflow) and, on a detected regression, open a fix task that is held out of dispatch until the CEO approves it (it terminates at `awaiting_ceo_approval`), then dispatch it through the normal delivery flow. It is dormant by default and armed by `ROBOCO_SELF_HEAL_ENABLED` plus a second opt-in `ROBOCO_SELF_HEAL_ORIGINATE_ENABLED`; origination is bounded by `ROBOCO_SELF_HEAL_MAX_OPEN_TASKS` / `_MAX_PER_CYCLE` so it can't flood the backlog. It never auto-merges or self-deploys (`roboco/services/self_heal_engine.py`).
|
||
|
||
**Multi-repo CI-watch (default-off).** The fan-out generalization of self-heal: instead of RoboCo's single own repo, it watches every project the operator opts into (`projects.ci_watch_enabled`, migration 048) and, on a red CI conclusion on that project's default branch, opens one fix task into that project's lifecycle that rides the normal delivery flow (+ PR-review gate) and never auto-merges. It reuses the exact hardened per-project `GitService.get_latest_ci_conclusion` (a missing signal is "unknown", never a false green; per-project errors are isolated and never abort the sweep), and is bounded + deduped per repo by `git_url` (a monorepo's cell-projects share one fix task) with per-cycle / rolling caps. Armed by `ROBOCO_CI_WATCH_ENABLED` (+ `_INTERVAL_SECONDS` / `_MAX_OPEN_TASKS` / `_MAX_PER_CYCLE` / `_DEFAULT_WORKFLOW`) and per-project `ci_watch_enabled` / `ci_watch_workflow`; `MultiProjectCITelemetrySource` (`roboco/services/telemetry/source.py`) + `CiWatchEngine` (`roboco/services/ci_watch_engine.py`) + a dedicated orchestrator `_ci_watch_loop`. The single-repo self-heal loop is untouched.
|
||
|
||
**Dependency-update bot (default-off).** A per-project engine mirroring the self-heal/CI-watch shape: weekly (default) it probes whether a dependency upgrade would change a project's lockfiles and, if so, opens one "update dependencies" task that rides the normal delivery flow (+ PR-review gate) and never auto-merges. Detection is read-only — `WorkspaceService.dry_upgrade_changes_lockfile` runs the project's `dep_update_command` (e.g. `uv lock --upgrade`) in a throwaway clone of the read clone and diffs the lockfile paths (`dep_update_paths`, or inferred `uv.lock`/`pnpm-lock.yaml`); the read clone is never mutated, nothing is committed/pushed, and a null/failing command originates nothing (fail-safe). A project participates only when `projects.dep_update_command` is set (migration 049); bounded + deduped per `git_url` with per-cycle/rolling caps. Armed by `ROBOCO_DEP_UPDATE_ENABLED` (+ `_INTERVAL_SECONDS` default 604800 / `_MAX_OPEN_TASKS` / `_MAX_PER_CYCLE`); `DepUpdateEngine` (`roboco/services/dep_update_engine.py`) + a dedicated `_dep_update_loop`.
|
||
|
||
**Gated release manager (default-off).** The autonomy that automates cutting a release up to the decision. A default-off background loop (`ReleaseManagerEngine` + `_release_manager_loop`) runs the deterministic readiness sweep (`ReleaseReadinessService.assess`, `roboco/services/release_readiness.py`) — diff-since-tag → conventional-commit classification → semver bump → version-reference completeness (the missed-ref guard) → CHANGELOG completeness → docs-drift (agent count) → migration single-head → gate state — and, past a threshold (`ROBOCO_RELEASE_MIN_COMMITS`, or any feat/security) with a green gate, originates ONE **release proposal** held for the CEO. The proposal is a `source='release_manager'` task owned by the Secretary, HELD (`confirmed_by_human=False`) and skipped by every dispatcher — acted on only by the CEO-gated routes, never delivered. The CEO approves or rejects-with-changes in the panel (`release-proposal-card.tsx`; `GET/POST /api/release/proposal{,/approve,/reject}`, CEO-only); approval runs the **fail-closed** `ReleaseExecutor` (`roboco/services/release_executor.py`): write the bumps across the canonical set (derived from the previous `chore(release):` commit) + the CHANGELOG entry, run `make quality` (abort before commit on red), commit `chore(release): X.Y.Z` (signed) + push, wait for green release-commit CI (abort before publish on red), then `gh release create vX.Y.Z`. Idempotent (an already-published version is a no-op) and never publishes without the CEO. Correctness is deterministic code, not agent judgment; the only generative step is the CHANGELOG prose, which the CEO reviews. Armed by `ROBOCO_RELEASE_MANAGER_ENABLED` (+ `ROBOCO_RELEASE_MIN_COMMITS` / `_INTERVAL_SECONDS`). Auto-deploy stays out of scope — publishing builds images; deploying to the NAS is the CEO's manual step.
|
||
|
||
**Organizational memory loop (default-off).** Closes the learn→reuse loop so agents stop cold-respawning blind. Three parts, all gated by `ROBOCO_ORG_MEMORY_ENABLED`: ① **capture** — at task completion `TaskService._completion_learnings_for` distills ONE high-signal lesson (Problem→Approach→Gotcha, ≤120 words) via the local model (`MemoryDistiller`, `roboco/services/memory_distiller.py`) instead of the noisy raw-notes/duration capture (flag-off keeps the legacy capture); journal indexing excludes `is_private` reflections from the shared corpus. ② **retrieve (keystone)** — on claim, `_briefing_for` injects `context_briefing["institutional_memory"]`: top-K (`ROBOCO_ORG_MEMORY_TOP_K`) relevance-floored (`ROBOCO_ORG_MEMORY_MIN_SCORE`) lessons + approved playbooks from a role-shaped query (`EvidenceRepo.similar_memory` over the LEARNINGS + PLAYBOOKS pgvector indexes); below the floor nothing is injected (no briefing bloat). ③ **playbooks** — a first-class curated procedure store: `PlaybookTable` (migration 050), the `PLAYBOOKS` OptimalService index, the `draft_playbook` content verb (delivery roles), Auditor `approve_playbook`/`reject_playbook`/`archive_playbook` curation (approval indexes it), and the panel review queue (`playbook-review-queue.tsx`; `/api/playbooks` Auditor/CEO routes). Distillation runs on the local model only — never a cloud LLM in the hot path; every step is best-effort (a failure never blocks completion or the briefing).
|
||
|
||
**Feature flags / company-in-a-box.** Env-gated, default-off subsystems toggle from the panel's Settings → Feature Flags card (`panel/src/components/settings/feature-flags-card.tsx`) instead of hand-editing env: web research (`ROBOCO_RESEARCH_ENABLED`), the strategy engine (`ROBOCO_STRATEGY_ENGINE_ENABLED`), pitch provisioning (`ROBOCO_PROVISIONING_*`), external / internal PR review, the agent-runtime toolchain match (`ROBOCO_TOOLCHAIN_MATCH_ENABLED`), the architectural-conventions standard (`ROBOCO_CONVENTIONS_ENABLED`), gateway-health recovery (`ROBOCO_GATEWAY_HEALTH_ENABLED`), multi-repo CI-watch (`ROBOCO_CI_WATCH_ENABLED`), the dependency-update bot (`ROBOCO_DEP_UPDATE_ENABLED`), the gated release manager (`ROBOCO_RELEASE_MANAGER_ENABLED`), the organizational memory loop (`ROBOCO_ORG_MEMORY_ENABLED`), and the self-heal flags above. A toggle persists in the settings store and takes effect on the next backend restart; an unset flag falls back to its environment / config default.
|
||
|
||
## Architectural Conventions Standard
|
||
|
||
**Per-project architectural standard (default-off).** Beyond the `make`-style gates (which check syntax/types/tests, not *where code lives*), each project can carry a repo-canonical `.roboco/conventions.yml` — an architecture map (which definition *kinds* belong in which modules), a toggleable rule set, custom regex rules, and waivers — so an agent cannot land a Pydantic model defined inside a router or a `# noqa` / `# type: ignore`. Placement of a *helper* (any top-level function) only **warns** — too blunt to hard-block; `thin_routes` doesn't count an explicit `db.commit()`; and a small allowlist of unavoidable framework suppressions (ruff `TC001`–`TC003`, pydantic `prop-decorator`) is exempt. Gated by `ROBOCO_CONVENTIONS_ENABLED`; fully inert when off. RoboCo itself ships a canonical `.roboco/conventions.yml`.
|
||
|
||
**Effective map.** Consumers read the *effective* map — auto-derived defaults (from a repo scan + `BUILTIN_RULES`, excluding `tests/`/`docs/` trees) overlaid by the committed file — so behaviour is identical whether the file is present, absent, or partial. `ConventionsService` (`roboco/services/conventions.py`) builds it, caches it per `(project, HEAD sha)` in `project_conventions_cache` (migration `043`), renders the per-task baseline constraints + the ambient prompt block, and scaffolds/restores the file via a PR (`GitService.open_conventions_pr`). The committed file + scan are read from a dedicated project-level **read clone** the service ensures on demand (`WorkspaceService.ensure_read_clone`, pinned to the default branch's HEAD) — the backfill that makes the standard resolve even for a project created before it existed, with no manual `workspace_path`. The schema lives in `roboco/foundation/policy/conventions/` (pure).
|
||
|
||
**Validator.** A single Python CLI, `python -m roboco.conventions check --root <repo> --files <a> <b> ...` (`roboco/conventions/`), uses tree-sitter (Python + TypeScript grammars, shipped in the agent image) to classify each changed definition and flag forbidden placements + hygiene + custom-rule matches as JSONL findings, after waiver filtering. Precision over recall (it abstains when uncertain so a `block` gate can't false-positive-strand a task) and fail-loud (a validator that cannot run exits 3 so the gate blocks, never silently passes).
|
||
|
||
**Threading + enforcement.** The standard reaches the work two ways: an ambient "Architectural Standard" block injected at spawn (`compose_prompt`) and an auto-attached `## Constraints` section on every project task (`TaskService.create`). Enforcement is deterministic: a `block`-level finding refuses `i_am_done` (dev pre-submit) and `pr_pass` (the in-path PR gate) with the offending `file:line` + fix hint; findings also surface in QA's `claim_review` evidence (`convention_findings`). A false positive is relieved by a `waiver` the dev commits in their branch — accountable, reviewed in the PR. The panel's per-project Conventions tab (in the edit-project dialog) shows the map + health and offers Save / Restore.
|
||
|
||
## MegaTask (sequenced batch intake)
|
||
|
||
**MegaTask** lets the CEO describe several tasks in one Intake chat and ship them as one collision-aware, sequenced batch — even across projects that don't share a codebase (the motivating case: a SaaS app + its OSS core engine + a framework adapter). It is a **core capability, not a feature flag** (additive + opt-in by nature: proposed only when the CEO asks for several tasks; single-task intake is byte-for-byte unchanged), branded "MegaTask" on every user-facing surface while internal names stay technical (`batch_id`, `SequencingService`).
|
||
|
||
**The umbrella model.** A MegaTask's identity is a real **umbrella** task — branchless, no PR of its own — over N **root-subtasks**, each a real Main-PM coordination root with its own `project_id`, branch, and PR. Hierarchy: Umbrella (Main PM) → N Root-subtasks (Main PM) → Cell tasks (cell PMs) → Dev subtasks. One extra Main-PM layer on top of the normal model. The umbrella is the single board-review / CEO-approve / Main-PM-coordinate unit, so the batch plugs into the existing coordination-root flow for free (task tree, progress rollup, CEO queue).
|
||
|
||
**Identity predicate (single source of truth).** `roboco/foundation/policy/batch.py`: `is_batch_umbrella` (`batch_id` set AND `parent_task_id` None), `is_batch_root_subtask` (`batch_id` set AND parented), `is_branchless_coordination` ((no-project AND product) OR umbrella). Every git-exemption site consults it so the umbrella's exemptions can't drift: the orchestrator's `_is_coordination_task`, the claim→in_progress branch gate (`GitContext.is_coordination`), `_ensure_branch_for_task` (returns `""` for an umbrella), and the CEO-reject routing. `submit_root` hard-rejects an umbrella (it assembles no PR); umbrella completion reuses the existing branchless path (`all_subtasks_terminal`, PR waived → escalate to CEO).
|
||
|
||
**Sequencing.** The pure `SequencingService.analyze(surfaces, cell_of, cell_capacity)` (`roboco/services/sequencing.py`; schema in `roboco/foundation/policy/sequencing/`) turns each draft's collision surface — `intends_to_touch` (globs), `adds_migration`, `touches_shared` — into a dependency DAG + Kahn-layered **waves**: file-overlap serializes (more-important first by `(priority, idx)`), migration-adders chain serially, a shared-surface edit runs after each non-shared task it overlaps (file-overlap-conditioned), independent tasks run in parallel; cell-contention only warns. Correctness lives in code, not agent judgment. The columns `tasks.batch_id` + `intends_to_touch` / `adds_migration` / `touches_shared` are migration **046**.
|
||
|
||
**Intake + create path.** The intake chat can be scoped to a **MegaTask** (a multi-project picker → `StartLiveRequest.project_ids`); the orchestrator clones each repo (`_clone_intake_scope` / `_slugs_for_project_ids`, the multi-repo machinery products already used). The intake agent proposes the whole batch with one **`propose_batch`** tool call — wired on both runtimes (the Claude SDK driver emits one `batch` stream chunk; the grok `intake_server` POSTs a `batch` relay event). The panel's third intake scope accumulates it into a Review-MegaTask card → `POST /prompter/live/{session}/confirm-batch`. `PrompterService.confirm_live_batch` builds the umbrella + N root-subtasks (via `create_task_from_draft` + a `BatchPlacement`) and wires the analyzer edges through `add_dependency`. The Board route holds the root-subtasks in BACKLOG until `approve_and_start` releases them (`_activate_batch_root_subtasks`); the Main-PM route dispatches wave 0 at once. The Product Owner + Head of Marketing review the whole batch (their identity prompts carry a MegaTask section).
|
||
|
||
## Services
|
||
|
||
Core services in `roboco/services/`:
|
||
|
||
| Service | Purpose |
|
||
|---------|---------|
|
||
| `TaskService` | Task CRUD and state transitions |
|
||
| `WorkSessionService` | Git session management, PR lifecycle |
|
||
| `WorkspaceService` | Multi-agent workspace resolution and cloning |
|
||
| `ProjectService` | Project/repository management |
|
||
| `MessagingService` | Channels, sessions, messages |
|
||
| `NotificationService` | Formal notifications |
|
||
| `JournalService` | Agent journals and entries |
|
||
| `OptimalService` | RAG queries (in-house pgvector engine) |
|
||
| `PermissionsService` | Role-based access control |
|
||
|
||
## Configuration
|
||
|
||
Key settings in `roboco/config.py` (env prefix: `ROBOCO_`):
|
||
|
||
```bash
|
||
# Database
|
||
ROBOCO_DATABASE_HOST=localhost
|
||
ROBOCO_DATABASE_PORT=5432
|
||
ROBOCO_DATABASE_USER=roboco
|
||
ROBOCO_DATABASE_PASSWORD=roboco
|
||
ROBOCO_DATABASE_NAME=roboco
|
||
|
||
# Redis
|
||
ROBOCO_REDIS_HOST=localhost
|
||
ROBOCO_REDIS_PORT=6379
|
||
|
||
# Security (REQUIRED)
|
||
# Generate with: python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'
|
||
ROBOCO_ENCRYPTION_KEY=<your-fernet-key>
|
||
|
||
# Workspaces
|
||
ROBOCO_WORKSPACES_ROOT=/data/workspaces
|
||
ROBOCO_WORKSPACE_AUTO_CLONE=true
|
||
ROBOCO_WORKSPACE_CLONE_TIMEOUT=300
|
||
|
||
# RAG (in-house pgvector engine)
|
||
ROBOCO_RAG_CHUNK_STRATEGY=fixed
|
||
ROBOCO_RAG_CHUNK_SIZE=512
|
||
ROBOCO_RAG_USE_HYDE=true
|
||
ROBOCO_RAG_USE_HYBRID_SEARCH=true
|
||
|
||
# AI/LLM
|
||
ROBOCO_DEFAULT_EMBEDDING_MODEL=qwen3-embedding:0.6b
|
||
ROBOCO_LOCAL_LLM_MODEL=glm-5.2:cloud
|
||
ROBOCO_LOCAL_LLM_BASE_URL=http://roboco-ollama:11434/v1
|
||
ROBOCO_OLLAMA_BASE_URL=http://roboco-ollama:11434
|
||
```
|
||
|
||
## Docker Deployment
|
||
|
||
### Container Architecture
|
||
|
||
The system runs as Docker Compose services. All Dockerfiles live under `docker/` at the project root; every service uses `context: .` plus `dockerfile: docker/<name>.Dockerfile`.
|
||
|
||
| Service | Purpose | Healthcheck |
|
||
|---------|---------|-------------|
|
||
| `postgres` | PostgreSQL + pgvector | `pg_isready` |
|
||
| `redis` | Cache, sessions, event bus | `redis-cli ping` |
|
||
| `ollama` | Local LLM + embeddings | `ollama list` |
|
||
| `ollama-init` | Pulls models on startup | One-shot |
|
||
| `agent-base-image` / `agent-*-image` | Pre-built images spawned per agent | One-shot |
|
||
| `orchestrator` | API + agent spawner | Depends on all above |
|
||
| `panel` | Next.js control panel (internal, port 3000) | — |
|
||
| `nginx` | Reverse proxy fronting panel + orchestrator | — |
|
||
|
||
### Single Entry Point
|
||
|
||
`nginx` is the only externally-exposed service. It listens on `localhost:3000` and routes:
|
||
|
||
- `/api/*` and `/ws/*` → `orchestrator:8000`
|
||
- everything else → `panel:3000`
|
||
|
||
This avoids CORS since the browser sees one origin. The Next.js code uses relative URLs (`/api`, `/ws`) and lets nginx do the dispatch.
|
||
|
||
### WebSocket streams
|
||
|
||
The orchestrator exposes WebSocket endpoints under `/ws` (router in `roboco/api/websocket.py`, `ConnectionManager` + `broadcast_*` helpers):
|
||
|
||
| Endpoint | Purpose |
|
||
|----------|---------|
|
||
| `/ws/channels/{id}`, `/ws/agents/{id}`, `/ws/sessions/{id}`, `/ws/notifications/{id}` | Per-resource live streams — `/ws/channels` + `/ws/sessions` carry live `message.new` frames (from `EventType.MESSAGE_SENT`), so a session transcript updates without a manual refresh |
|
||
| `/ws/system` | Operator/system-wide stream (no per-agent keying) — the rate-limit lifecycle (`RATE_LIMIT_HIT` / `RATE_LIMIT_LIFTED`) and live usage (`USAGE_SNAPSHOT`, pushed to the usage dashboard) |
|
||
|
||
Server-side events reach these sockets through `roboco/api/websocket_bridge.py`, which subscribes to the `StreamEventBus` and forwards each event to the matching connections. To add a new live event: define an `EventType` (dotted value), publish it to the bus, add a `_handle_*` forwarder in `websocket_bridge`, and consume it on the panel via the `useWebSocket("/<endpoint>", …)` hook — do not stand up a parallel endpoint or client stack. `MESSAGE_SENT` is the worked example: `send_message` publishes it, `_handle_message_event` fans it out to `/ws/sessions/{id}` + `/ws/channels/{id}` as a `message.new` frame, and the panel's `useSessionStream` consumes it.
|
||
|
||
### Rate limiting & usage
|
||
|
||
- **Provider rate limits** are tracked in Redis (`RateLimitStateTracker`, `roboco/services/gateway/`). On a provider 429 an agent calls `i_am_blocked(reason="rate_limited")`; the spawn gate then **queues** (never drops) further work for that provider, and a background probe-and-resume loop in the orchestrator clears the limit and revives parked agents when it lifts.
|
||
- **Provider overloads** reuse the same park-and-probe break. A persistent model-API overload (HTTP 529 / 500 / 503 — the SDK already retries transient ones) parks the provider exactly like a 429 instead of crash-retrying the agent straight back into the overload and burning tokens; the overload is detected orchestrator-side from the dead container's log markers, and the background loop revives the parked work when it recovers. The same break also catches the **Claude session-limit** 429 (the org's 5-hour usage window): an agent exiting with a 0-token session-limit rejection parks the provider and is auto-revived when the window resets, instead of fleet-wide crash-respawning straight back into the limit. Gated by `ROBOCO_OVERLOAD_BREAK_ENABLED` (default-on).
|
||
- **Gateway-health recovery** closes a blind spot in the stale-claim reaper: the heartbeat is bumped only by gateway verbs, so a broken-but-alive agent (a corrupted `/app/.venv` so no gateway tool imports) goes heartbeat-stale yet keeps its container up, and the reaper's live-skip would protect it forever. On a stale-heartbeat live container the reaper now probes the gateway out-of-band (`_probe_gateway_health` → `docker exec` the gateway venv imports) and, once broken past `ROBOCO_GATEWAY_HEALTH_GRACE_SECONDS` (a transient probe miss is tolerated), kills + evicts it (`_maybe_recover_broken_gateway`) so it falls through to release + respawn; healthy or inconclusive probes spare it. Gated by `ROBOCO_GATEWAY_HEALTH_ENABLED` (default-on). It is the third leg beside the shipped bash-guard `/app` block (prevents the self-corruption) and the reaper Docker-liveness fallback (stops over-reaping live containers).
|
||
- **PM coordinator concurrency.** A Main / Cell PM plans and delegates many root tasks in parallel — the actual work then runs in the delegated children/cells, not in the PM's own hands. The claim-time concurrency guards that keep a *developer* to one task at a time (`already_active` / `paused`, in `roboco/services/gateway/claim_guards.py`) are therefore **skipped for the coordinator PM roles** (`_COORDINATOR_ROLES = {main_pm, cell_pm}`, consulted in `_run_claim_guards`); only a genuine upstream **sequence dependency** (`unmet_dependency`, which parks the task back to `pending`) holds a PM's root back. Without this a single PM that claimed one root could never plan a second — it thrashed between its claimed roots and respawned forever, burning tokens for zero progress (the live `i_am_idle`-auto-paused-umbrella deadlock). The `paused` guard also excludes the target task itself, so a PM re-entering its own paused umbrella never self-blocks.
|
||
- **Orchestrator runtime-state durability.** The PM-respawn loop breaker (`_pm_respawn_tracker`, the `(agent_slug, task_id) → strike-count` circuit breaker) is **DB-durable** via the `respawn_tracker` table (migration 051): each gate mutation write-throughs fire-and-forget on the `_bg_tasks` set (`_schedule_respawn_persist` → `_persist_respawn_record`), and `restore_respawn_tracker()` repopulates it at `start()`, validating each row against live tasks (terminal/missing rows are evicted). Kept only in memory it reset to `count=1` on every restart and re-burned the whole strike threshold (4 spawns) against a still-wedged task. It mirrors the `WaitingRecordTable` / `restore_waiting_records` pattern: best-effort (a DB hiccup degrades to in-memory-only — it can only ever *suppress* a spawn, never manufacture one) and inert when the table is empty. The companion `_instances` registry is **reconciled-from-Docker** (not persisted) at startup via `_readopt_running_agents`, so the reaper's liveness path and the spawn gate's `_is_agent_active` check see surviving containers immediately after a restart.
|
||
- **Token usage** is captured per agent session from the Claude Code transcript via the SDK server's `/usage/sync` (hook → orchestrator finalize → `agent_spawn_sessions` → `daily_usage_rollups` → dashboard). Cost uses provider-aware pricing in `roboco/billing/pricing.py` (Anthropic priced; local/Ollama intentionally `$0`). The token sweep also publishes `USAGE_SNAPSHOT` to `/ws/system`, so the dashboard's "Token Usage & Cost" panel updates live and falls back to HTTP polling when the stream is down.
|
||
- **Delivery observability** (the panel's Metrics → "Delivery" tab) shows how work *flows*, computed by `MetricsService` from data already captured — no new feature flag. Per-stage cycle time and the bottleneck distribution are reconstructed from the `audit_log` transition journey (each generic `task.<status>` event marks entry into a status; the named `task.qa_fail`/`task.pr_fail` events are excluded from the reconstruction). Rework rate reads `tasks.revision_count` — incremented once per transition into `needs_revision` at the single chokepoint `TaskService._emit_status_transition_audit` — and attributes each bounce to the QA / PR-reviewer via those named audit events; rework cost joins `agent_spawn_sessions.task_id`. Read-only endpoints: `/dashboard/metrics/{cycle-time,bottlenecks,rework,scorecard/agent/{id},scorecard/team/{team}}`.
|
||
|
||
### Startup Sequence
|
||
|
||
The startup order is critical due to dependencies:
|
||
|
||
```
|
||
postgres ──┐
|
||
redis ─────┼──> ollama ──> ollama-init ──> orchestrator ──> panel ──> nginx
|
||
│ │ │
|
||
│ │ └── Pulls qwen3-embedding:0.6b, glm-5.2:cloud
|
||
│ └── Healthcheck: ollama list
|
||
└── Healthcheck: pg_isready, redis-cli ping
|
||
```
|
||
|
||
**Important timing notes:**
|
||
1. `ollama-init` pulls models (~30s for embedding model, ~2min for LLM)
|
||
2. Orchestrator waits for models before starting
|
||
3. FastAPI lifespan indexes documents using Ollama (~30-60s)
|
||
4. Orchestrator polls `/health` until API is ready before starting dispatcher
|
||
5. After orchestrator is up, `panel` (Next.js) builds/starts, then `nginx`
|
||
|
||
### Database migrations
|
||
|
||
Schema changes ship as Alembic migrations under `alembic/versions/`. Run:
|
||
|
||
```bash
|
||
docker compose exec orchestrator alembic upgrade head
|
||
```
|
||
|
||
after pulling any change that adds a new migration.
|
||
|
||
### Ollama Configuration
|
||
|
||
Ollama provides two APIs:
|
||
- `/v1/*` - OpenAI-compatible API (for LLM chat/completion)
|
||
- `/api/*` - Native Ollama API (for embeddings, model management)
|
||
|
||
The embedder uses `/api/embed` endpoint with the `qwen3-embedding:0.6b` model.
|
||
|
||
**Environment variables for Docker:**
|
||
```bash
|
||
ROBOCO_LOCAL_LLM_BASE_URL=http://roboco-ollama:11434/v1 # OpenAI-compat
|
||
ROBOCO_OLLAMA_BASE_URL=http://roboco-ollama:11434 # Native API
|
||
```
|
||
|
||
### Common Issues
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---------|-------|-----|
|
||
| `404 /api/embed` | Model not pulled | Check `docker logs roboco-ollama-init` |
|
||
| `All connection attempts failed` | API not ready | Orchestrator starts before FastAPI lifespan completes |
|
||
| Healthcheck failing | Wrong endpoint | Use `ollama list` not `curl` |
|
||
|
||
## Blueprint Reference
|
||
|
||
The organizational structure, communication matrix, role descriptions, and access-control model are documented inline above and in the user-facing documentation site (MkDocs Material; source under `docs/`, built by `mkdocs.yml`, deployed by `.github/workflows/docs.yml` via GitHub Pages Actions and served at rennf93.github.io/roboco). `docs/rag/` remains the agent-facing RAG corpus (excluded from the published site); the old root `usage.md` / `deployment.md` are now redirect stubs into the site.
|