Files
roboco/docs/map/metrics-observability.md
T
7901ea419e Retire channels/sessions/messages; A2A becomes primary agent comms (#306)
* feat(a2a): deliver latest incoming message preview into the claim briefing

list_unread_a2a now carries last_message_preview (the latest message from the
OTHER agent, never the agent's own reply), fetched via a correlated subquery in
the same query — no N+1 on the per-verb briefing path.

* feat(a2a): read_a2a verb delivers unread message bodies to the agent

A2AService.get_unread_messages returns the caller's unread INCOMING messages
(never its own sends), marking exactly those rows read atomically so a message
arriving mid-call is preserved. Wired as the read_a2a content verb (route +
do_server tool + granted to every delivery role) — the content-bearing read the
A2A inbox lacked (read_messages only zeroed the counter).

* docs(rag): document read_a2a as the A2A content-read path

* fix(task): backlog activation no longer requires a discussion session

Removes the SessionTaskTable gate in activate() (and its dangling log field),
deletes _inherit_parent_session + its create() call, and drops the now-unused
SessionTaskTable import. Coordination rides task state; the session subsystem is
being retired. Tests updated to the new (no-session) behavior.

* fix(orchestrator): drop session sweep from _run_sweep

Removes the messaging import + sweep_timed_out_sessions call. That import sat
outside the try/except, so once messaging.py is deleted it would have killed the
entire sweep cascade (budget kill-switch, token rollups, retention, image prune,
superseded-PR reconcile). Notification sweep + all maintenance sweeps unchanged.

* release-manager --no-tags read-clone fix

* test: update evidence_repo unit test for a2a last_message_preview

* refactor(gateway): drop session propagation on delegate

Removes propagate_sessions_to_subtask from delegate(), the ChoreographerDeps
messaging field + property, and the ChoreographerDeps messaging arg in deps.py
(ContentActions messaging + import stay until the verbs are removed). Deletes the
propagation test; strips the now-invalid messaging kwarg from ChoreographerDeps
test builders.

* refactor(gateway): remove say/open_session/link_session/channels verbs

Removes the four channel/session verbs across content_actions (impls +
ContentActionsDeps.messaging), do_server (tools + registry), role_config (grants
+ _CHANNEL_DISCOVERY), do.py (routes), schemas/v1/do.py (request models), and
deps.py (MessagingService import + construction). Regenerates the prompt verb
tables. dm/notify/read_messages/read_a2a stay. Tests deleted/updated accordingly.

* uv.lock Upgrade

* refactor: remove conversation RAG indexing; Secretary announces via notification

Drops the CONVERSATIONS index (index_conversation, ConversationsIndexPlugin,
IndexType.CONVERSATIONS enum, IndexConversationParams, mentor.py type-label, the
messaging index hook) and its chunk-table manifest entries. The Secretary's
ANNOUNCE/RELAY_MESSAGE now fan out a BROADCAST notification to every agent's
inbox (NotificationService.broadcast) instead of posting to a dead channel.

* fix(panel): label RAG health error lines by subsystem

A red llm_error (e.g. the glm-5.2:cloud weekly-limit 429) rendered under
the 'Embedding: ok' header with no label, reading as an embedding failure.
Prefix each error line with LLM / Embedding / Vector store.

* refactor: remove channel/message reads from metrics, dashboard, git, events

MetricsService drops get_communication_volume + the MessageTable
message-count in get_agent_metrics (and the now-dead messages_sent_week
field). DashboardService drops get_channel_feeds/_compute_channel_status
and the message read in get_recent_activity (task activity kept);
get_auditor_metrics no longer reports communication_volume.
GitService's two primary-session-id helpers always return None now
(callers already treat None as "no primary session"). events/handlers.py
drops the SESSION_CLOSED/SESSION_TIMEOUT subscriptions + the
handle_session_boundary handler.

Forced follow-on: api/routes/dashboard.py + api/schemas/dashboard.py
dropped the now-dangling live_feeds/ChannelFeed surface and the
/metrics/communication route, which wrapped the removed service calls
directly (mypy would otherwise fail on the missing attributes).

* refactor: delete MessagingService + channel seeding

Edited db/__init__.py and services/__init__.py first (drop the unconditional
Channel/Group/Message/Session table + MessagingService re-exports), then
deleted services/messaging.py, then trimmed db/seed.py to only create_agents
(create_channels/create_channel_memberships/create_initial_messages gone).

Forced expansion: api/routes/{channels,groups,sessions,messages}.py import
roboco.services.messaging directly (not through the package __init__), as
does api/routes/tasks.py (the session-links embed on GET /tasks/{id} and the
GET /{id}/sessions route). Deleting messaging.py without addressing these
breaks `import roboco.api.app` immediately, since app.py eagerly imports all
route modules at startup. Since the 4 CRUD route files are 100%
MessagingService-backed with zero independent logic (and are wholesale
deletes in the plan's later API-routes task anyway), deleted them now +
unmounted from app.py/routes/__init__.py; tasks.py got the same surgical
trim its later task already specified (drop session-links embed +
TaskSessionLinkResponse/TaskResponse.sessions). This pulls a slice of that
later work forward — the routes/schemas for channels/groups/sessions/messages
still need their own pass, but their messaging-coupled parts are gone.

Verified with a full-suite collection sweep (12010 tests collected, zero
import errors) beyond the directly touched test dirs, given the expanded
blast radius.

* refactor: remove channel/session/message models, tables, and channel policy

Models: deleted channel.py/group.py/session.py/messaging.py wholesale
(zero external consumers besides the models/__init__.py re-export).
message.py surgically trimmed: removed MessageCreate (dead) and MessageEdit
(never instantiated; ExtractedMessage.edit_history retyped to
list[dict[str, Any]] to match how it's actually persisted — confirmed
ExtractedMessage was never written to any DB table, so MessageTable's
removal carries no functional risk to the kept extraction pipeline).
base.py: removed SessionStatus + ChannelType, kept MessageType. Also
removed the confirmed-dead channels_read/channels_write fields from
models/agent.py:AgentPermissions and models/dashboard.py:ChannelFeedData.

db/tables.py: deleted ChannelTable/GroupTable/SessionTable/SessionTaskTable/
MessageTable, TaskTable.session_links, and JournalEntryTable.session_id —
cascaded through models/journal.py, services/journal.py, and
api/schemas+routes/journals.py (22 plumbing sites).

foundation/policy/communications.py: removed the ChannelSpec/CHANNELS
catalog + TEAM_SCOPED_ROLES/_CELL_*/_AUDITOR_ONLY helpers, kept the
notification policy (Priority/parse_priority/NOTIFY_SENDER_ROLES/
ACK_REQUIRED_BY_TYPE). enforcement/channel_access.py deleted (confirmed
fully dead in production). agents_config.py: removed CHANNEL_ACCESS
(kept A2A_ALLOWED_PAIRS). seeds/initial_data.py: removed
DEFAULT_CHANNELS/CHANNEL_MEMBERSHIPS/AUDITOR_SILENT_ACCESS + the
never-consumed INITIAL_MESSAGES. config.py: removed
session_idle_timeout_seconds (zero consumers). exceptions.py: removed
dead ChannelError/ChannelAccessDeniedError/SessionClosedError.

Forced expansion beyond the original file list — ChannelType cascaded
into a live, mounted surface the plan didn't trace: agents_config.
CHANNEL_ACCESS -> services/permissions.py's channel-RBAC methods (not
models/permissions.py, which turned out to have no channel code at all)
-> two real endpoints in api/routes/stream.py (GET /permissions,
GET /permissions/channel/{name}) and two dependency factories in
api/deps.py. Removed the channel methods + fields, deleted the
channel-specific stream.py endpoint, deleted require_channel_read/write.
Also deleted api/schemas/{channels,sessions}.py (hard dependency on the
removed enums; already fully dead after the Task 10 route deletions) and
api/schemas/messages.py (a TYPE_CHECKING-only import of the deleted
MessageTable; likewise already fully dead) + its dedicated test file.

Test updates: test_permissions.py -14 channel tests (matches the planned
count exactly), test_communications.py / test_communications_consumers.py
split to keep only notification-policy coverage, test_exceptions.py -9,
test_deps.py -4, plus the journal/stream/foundation-smoke fallout. Also
fixed a pre-existing (Task 7) broken assertion in
test_foundation_phase3_smoke.py that inspected a `say()` method already
removed from ContentActions.

Verified: full-suite collection (11961 tests, zero import errors) and a
complete test run (11567 passed, 394 skipped, 0 failed) in addition to
the targeted suites.

* migration: drop channels/groups/sessions/session_tasks/messages + enum types

alembic/versions/060_drop_messaging.py: drop_column journal_entries.
session_id (sidesteps hardcoding the FK constraint name — verified
empirically against a live migrated DB that it's actually
fk_journal_entries_session_id_sessions, but drop_column doesn't care
either way); drop_table in FK order (messages -> session_tasks ->
sessions -> groups -> channels); DROP TABLE IF EXISTS chunks_conversations
(runtime-provisioned, not alembic-managed, would otherwise orphan); DROP
TYPE IF EXISTS for messagetype/sessionstatus/sessionscope/channeltype
(messagetype's Python enum stays for ExtractedMessage, but the DB type
had zero live columns left once MessageTable was dropped in the prior
commit). downgrade() raises NotImplementedError — one-way removal.

Pruned scripts/reset_runtime_state.sql + .sh: removed the DELETE/COUNT
lines for messages/session_tasks/sessions/groups/channels and the
groups.active_session_id reset block.

Verified end-to-end against a scratch Postgres DB: full migration chain
001->060 applies cleanly, alembic heads shows a single head, all 6 dropped
tables + 4 enum types + the journal_entries.session_id column are
confirmed gone, journal_entries keeps only its journal_id/task_id FKs,
downgrade correctly raises NotImplementedError without corrupting DB
state, and the pruned reset_runtime_state.sql runs clean (no errors)
against a fully-migrated DB.

* refactor(api): remove channel/session/message routes + WS streams

Most of this task's file list was already forced through in earlier
commits (routes/{channels,groups,sessions,messages}.py + app.py/__init__.py
unmounting in the MessagingService-deletion commit; tasks.py's
session-links embed + GET /{id}/sessions + schemas/tasks.py's
TaskResponse.sessions in that same commit; deps.py's require_channel_read/
write + schemas/{channels,sessions}.py in the models/tables commit). This
closes out what was left:

- api/websocket.py: deleted the channel_stream + session_stream routes,
  ConnectionManager's channel_connections/session_connections dicts,
  connect_channel/connect_session, broadcast_to_channel/broadcast_to_session,
  get_channel_subscriber_count, and their cleanup lines in disconnect().
  Agent streams, notification streams, and the operator system stream are
  untouched.
- api/websocket_bridge.py: deleted _handle_session_event +
  _handle_message_event and their SESSION_CREATED/SESSION_CLOSED/
  SESSION_TIMEOUT/MESSAGE_SENT subscriptions. The A2A live-view, rate-limit,
  usage, agent-lifecycle, and notification bridges are untouched.
- api/schemas/websocket.py: removed NewMessageBroadcast, WSMessageNew,
  WSMessageEdit, WSMessageDelete, WSSessionClosed — kept the WSMessage base
  class (still subclassed by the kept WSAgentStream/WSNotification) plus
  those two.
- api/schemas/groups.py: deleted (already fully orphaned since routes/
  groups.py was removed; its GroupResponse/GroupDetailResponse had zero
  consumers).

Updated the 5 websocket test files accordingly (removed the channel/
session-specific tests + fixed imports); test_websocket_bridge.py's
registration-coverage test dropped the SESSION_*/MESSAGE_SENT assertions.

Verified: full-suite collection (11943 tests, zero import errors) and a
complete test run (11549 passed, 394 skipped, 0 failed).

* docs: retire channels/sessions/messages from agent-facing docs + CLAUDE.md

Rewrites docs/rag (RAG-indexed) + docs/map + CLAUDE.md to reflect A2A (dm +
read_a2a) as primary agent comms; deletes the channel docs, splits messaging-tools
+ messaging-notification (renamed notification.md), swaps the WS worked example to
A2A_MESSAGE_SENT. _complete_map.md still needs regeneration (generated file).

* refactor(panel): remove Communications surface (channels/sessions)

Deletes the /communications routes, message components, task-detail Sessions tab,
use-channels + channel/session WS hooks, and the channels/sessions/messages/groups
api clients; prunes the Channel/Session/Message/Group types + mock data. (Auditor
live-feeds + dashboard.ts dead-route cleanup is a follow-up.)

* refactor(panel): drop auditor channel-feed + dead communication-metric route

* docs(map): regenerate _complete_map from updated slices

* fix(a2a): reduce get_unread_messages complexity below xenon C + stale comments

Extract the per-conversation unread-counter recompute into _reset_unread_counter
(the CI quality gate flagged get_unread_messages as rank C). Also drop the deleted
open_session from a content_actions comment and reword an evidence_repo docstring
that cited the removed messaging._notify_mentions.

---------

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-07-04 03:10:33 +02:00

28 KiB
Raw Blame History

RoboCo Map — metrics-observability slice

Purpose

The metrics & observability slice is the read-only measurement layer of RoboCo: it turns task/audit-log/spawn-session state into the numbers the panel renders and the CEO/Auditor act on. MetricsService reconstructs delivery flow (cycle-time, bottlenecks, rework, scorecards) from the audit-log transition journey plus tasks.revision_count; UsageService aggregates per-agent/per-team/per-model token spend and projections from agent_spawn_sessions / daily_usage_rollups; DashboardService adds auditor flags/reports (in-memory) and CEO overview aggregations; CockpitService fuses goals + delivery + spend + strategy signals into one CEO summary; telemetry/source.py reads CI health for self-heal / multi-repo CI-watch; billing/pricing.py is the provider-aware per-token cost function every cost field is derived from.

Files

Path Role approx LOC
roboco/services/metrics.py MetricsService — velocity, blockers, team/agent metrics, health, cycle-time/bottleneck/rework/scorecard observability 905
roboco/services/dashboard.py DashboardService — auditor flags/reports (in-memory singleton), CEO overview, audit queue, agent status, recent activity 457
roboco/services/cockpit.py CockpitService — read-only CEO "is the business winning?" summary (goals+delivery+spend+signals) 97
roboco/services/usage.py UsageService — token usage summary, time-series, by-agent/team/model, projection, cache efficiency, today summary, recent sessions 478
roboco/services/usage_events.py UsageSnapshot dataclass + publish_usage_snapshot — publishes USAGE_SNAPSHOT to the StreamEventBus 52
roboco/services/telemetry/__init__.py Re-export of CI telemetry source symbols 18
roboco/services/telemetry/source.py TelemetrySample + TelemetrySource protocol + GitHubCITelemetrySource (self-heal) + MultiProjectCITelemetrySource (CI-watch) 212
roboco/billing/__init__.py Re-export calculate_cost, CostResult, calculate_cost_result 8
roboco/billing/pricing.py calculate_cost / calculate_cost_result / CostResult — provider-aware per-token USD pricing (Anthropic + Grok priced; local/Ollama $0); CostResult exposes unpriced flag for unpriced-Anthropic detection 199

Key Symbols

Name Kind File:Line Responsibility
MetricsService class metrics.py:91 All delivery/velocity/health/observability aggregations
MetricsService.get_velocity method metrics.py:108 Period completed/created counts, avg completion hours, completion rate
MetricsService._blocked_since_map method metrics.py:176 Queries audit_log for task.blocked events to build {task_id: blocked_at} map; fixes the updated_at heuristic (#67)
MetricsService.get_blocker_metrics method metrics.py:208 Active blockers, avg/longest blocked hours, blockers by team
MetricsService.get_team_metrics method metrics.py:229 Per-team active/completed/blocked + doc-coverage (dev_notes proxy)
MetricsService.get_all_team_metrics method metrics.py:322 Loop over BACKEND/FRONTEND/UX_UI
MetricsService.get_agent_metrics method metrics.py:333 Per-agent weekly completed, avg hours, messages
MetricsService.get_health_status method metrics.py:487 ok/slow/critical from blocked ratio + stale-active heuristic
MetricsService._determine_health_status method metrics.py:451 Threshold logic (CRITICAL_BLOCKED_RATIO=0.3, SLOW=0.15, STALE=5)
MetricsService.get_cycle_time_by_stage method metrics.py:529 Per-stage dwell reconstructed from audit_log task.<status> events via LEAD window; excludes named qa_fail/pr_fail
MetricsService.get_bottleneck_distribution method metrics.py:587 Cumulative dwell per stage + live parked counts + active blockers
MetricsService.get_rework_metrics method metrics.py:747 Overall/team/agent rework rate + rework cost from spawn sessions
MetricsService._rework_by_agent method metrics.py:646 Owner bounce-rate + reviewer-attributed qa_fails/pr_fails from audit_log
MetricsService._rework_cost method metrics.py:720 Sum of estimated_cost_usd over spawn sessions of reworked tasks
MetricsService.get_scorecard method metrics.py:795 Fused per-agent or per-cell scorecard (completed, cycle, rework, tokens, cost)
MetricsService._tokens_cost_for method metrics.py:768 Sum tokens+cost from spawn sessions for an agent slug or team
MetricsService._avg_cycle_hours method metrics.py:869 Avg started→completed hours for completed tasks
_as_hours func metrics.py:47 Coerce SQL epoch aggregate to rounded float (avoids Decimal→JSON string crash)
ACTIVE_STATUSES const metrics.py:61 CLAIMED/IN_PROGRESS/VERIFYING/AWAITING_QA (BLOCKED excluded — note)
DashboardService class dashboard.py:58 Auditor flags/reports + CEO overview + agent/activity feeds
_DashboardStorageHolder class dashboard.py:35 Process-singleton in-memory flag/report store
get_storage / reset_storage func dashboard.py:41/48 Singleton accessor + test reset
DashboardService.create_flag/get_flags/resolve_flag methods dashboard.py:87/102/124 Auditor flag CRUD over in-memory store
DashboardService.create_report/send_report methods dashboard.py:146/184 Auditor report CRUD + send marking
DashboardService.get_audit_queue method dashboard.py:241 Blocked + awaiting-QA tasks as queue items
DashboardService.get_team_health_list method dashboard.py:279 Health for BACKEND/FRONTEND/UX_UI/BOARD
DashboardService.get_key_metrics method dashboard.py:299 Velocity + doc coverage + blockers summary
DashboardService.get_all_agent_status method dashboard.py:365 Agent counts by status + per-agent snapshot
DashboardService.get_recent_activity method dashboard.py:398 Merged messages+task_updates feed, sorted desc
CockpitService class cockpit.py:36 Read-only CEO summary + lightweight signals slice
CockpitService.summary method cockpit.py:41 goals+counts+delivery+spend+projection+pitches+signals, basis="proxy"
CockpitService.signals method cockpit.py:81 Strategy-engine signals only (lightweight panel slice)
UsageService class usage.py:70 Token usage analytics over spawn sessions + rollups
UsageService.get_summary method usage.py:77 Period totals + trend_pct vs previous period
UsageService.get_time_series method usage.py:166 Hourly (24h) / daily (7d/30d) buckets
UsageService._aggregate_by method usage.py:237 Shared group-by for agent/team/model with pct_of_total
UsageService.get_by_agent/get_by_team/get_by_model methods usage.py:304/310/314 Dimension breakdowns
UsageService.get_projection method usage.py:322 7-day avg daily cost × 30 projected monthly
UsageService.get_cache_efficiency method usage.py:359 cache_hit_rate + cost_saved (sonnet baseline pricing)
UsageService.get_today_summary method usage.py:420 Today's tokens/cost from daily_usage_rollups
UsageService.get_recent_sessions method usage.py:461 Raw per-session rows for the dashboard table
_parse_period func usage.py:55 "24h"/"7d"/"30d" → (start_dt, hours); defaults 24h
_row_tokens / _session_row func usage.py:24/38 Null-coalescing token extraction + session row shaping
UsageSnapshot dataclass usage_events.py:22 Aggregate token/cost payload for USAGE_SNAPSHOT events
UsageSnapshot.event_data method usage_events.py:36 Render + timestamp-stamp event payload
publish_usage_snapshot func usage_events.py:47 Publish USAGE_SNAPSHOT to StreamEventBus (lazy Event import)
TelemetrySample dataclass telemetry/source.py:39 Normalized health reading; is_breach = value ≥ threshold
TelemetrySource protocol telemetry/source.py:63 Pull-based read-only fetch() contract
GitHubCITelemetrySource class telemetry/source.py:70 Self-heal: latest CI run for self_heal_project_slug
MultiProjectCITelemetrySource class telemetry/source.py:130 CI-watch: per-project samples, isolated failures
MultiProjectCITelemetrySource._sample_for method telemetry/source.py:166 One project's sample, None on unreadable signal
FAILURE_CONCLUSIONS const telemetry/source.py:36 {failure, timed_out, startup_failure}
get_ci_telemetry_source / get_multi_ci_telemetry_source func telemetry/source.py:125/207 Factory constructors
CostResult dataclass billing/pricing.py:96 Frozen result carrying cost_usd, unpriced (True when Anthropic model has no pricing entry), is_anthropic; lets callers distinguish intentional-$0 (local) from missed-pricing (Anthropic)
calculate_cost func billing/pricing.py:113 Thin float wrapper over calculate_cost_result — returns cost_usd only; kept for existing callers (orchestrator, grok_cli_usage)
calculate_cost_result func billing/pricing.py:135 Full provider-aware cost calculation returning CostResult; authoritative for unpriced attribution
_lookup_prices func billing/pricing.py:80 Substring pricing-table lookup, longest fragment wins
_is_anthropic_model func billing/pricing.py:75 Claude/opus/sonnet/haiku fragment detection (warn gate)
_PRICING const billing/pricing.py:49 Per-model (input/output/cache_read/cache_write) USD/1M table

Data Flow

Two upstreams feed this slice: (1) the orchestrator token sweep (runtime/orchestrator.py ~line 5270) reads each active agent's Claude Code transcript via the SDK /usage/sync, calls billing.calculate_cost, persists a AgentSpawnSessionTable row (and a TokenUsageSnapshotTable row), then accumulates per-agent totals and publishes a USAGE_SNAPSHOT event via usage_events.publish_usage_snapshotStreamEventBuswebsocket_bridge/ws/system panel clients. (2) task lifecycle transitions write generic task.<status> and named task.qa_fail/task.pr_fail events to audit_log and increment tasks.revision_count at the single TaskService._emit_status_transition_audit chokepoint.

Downstream, the panel hits the API routes: /api/usage/* (summary, time-series, by-agent/team/model, projection, cache-efficiency, sessions) → UsageService; /api/dashboard/metrics/{velocity,blockers,team,communication,health,agent,cycle-time,bottlenecks,rework,scorecard/agent,scorecard/team} and /api/dashboard/{auditor,ceo,...}DashboardService + MetricsService; /api/cockpit/{summary,signals}CockpitService. MetricsService.get_cycle_time_by_stage runs a raw SQL window over audit_log filtering event_type = 'task.' || to_status to exclude named events, deriving per-stage dwell. Rework joins tasks.revision_count > 0 to completed tasks and agent_spawn_sessions.task_id for cost. Telemetry flows separately: the self-heal loop and ci-watch loop construct their source, call fetch(), and feed TelemetrySamples to their engine's regression detector, which originates a held fix task on a breach.

Mermaid

graph LR
  Transcript[Claude Code transcript] --> Sweep[orchestrator token sweep]
  Sweep --> CalcCost[billing.calculate_cost]
  CalcCost --> SpawnSess[agent_spawn_sessions row]
  Sweep --> PubSnap[publish_usage_snapshot]
  PubSnap --> Bus[StreamEventBus]
  Bus --> WS["/ws/system panel"]

  TaskLife[TaskService transitions] --> AuditLog[audit_log task.* events]
  TaskLife --> RevCount[tasks.revision_count++]

  Panel -->|/api/usage/*| UsageSvc[UsageService]
  UsageSvc --> SpawnSess
  UsageSvc --> Rollups[daily_usage_rollups]

  Panel -->|/api/dashboard/metrics/*| DashSvc[DashboardService]
  DashSvc --> MetricsSvc[MetricsService]
  MetricsSvc --> AuditLog
  MetricsSvc --> Tasks[tasks table]
  MetricsSvc --> SpawnSess
  MetricsSvc --> RevCount

  Panel -->|/api/cockpit/*| Cockpit[CockpitService]
  Cockpit --> Goals[company_goals]
  Cockpit --> TaskSvc[TaskService]
  Cockpit --> UsageSvc
  Cockpit --> Strategy[strategy_engine]
  Cockpit --> Pitch[pitch service]

  SelfHeal[self_heal_loop] --> CISrc[GitHubCITelemetrySource]
  CIWatch[ci_watch_loop] --> MultiSrc[MultiProjectCITelemetrySource]
  CISrc --> GitSvc[GitService.get_latest_ci_conclusion]
  MultiSrc --> GitSvc
  CISrc --> Samples[TelemetrySample]
  MultiSrc --> Samples
  Samples --> Engine[regression detector -> held fix task]

Logical Tree

metrics-observability
├── roboco/billing/
│   ├── __init__.py            # re-export calculate_cost
│   └── pricing.py             # _PRICING table, _lookup_prices, calculate_cost, _is_anthropic_model
├── roboco/services/
│   ├── metrics.py             # MetricsService (velocity/blockers/team/agent/comm/health/observability)
│   ├── dashboard.py           # DashboardService + _DashboardStorageHolder (flags/reports/CEO/queue)
│   ├── cockpit.py             # CockpitService (summary/signals)
│   ├── usage.py               # UsageService (summary/series/by-dim/projection/cache/today/sessions)
│   ├── usage_events.py        # UsageSnapshot + publish_usage_snapshot
│   └── telemetry/
│       ├── __init__.py        # re-exports
│       └── source.py          # TelemetrySample, TelemetrySource, GitHubCITelemetrySource, MultiProjectCITelemetrySource

Dependencies

Internal (roboco):

  • roboco.db.tablesAgentSpawnSessionTable, DailyUsageRollupTable, AgentTable, AuditLogTable, TaskTable, NotificationTable
  • roboco.models.baseTaskStatus, Team, AgentStatus
  • roboco.models.metrics — all metric result schemas (VelocityMetrics, StageTiming, ReworkReport, Scorecard, …)
  • roboco.models.dashboardFlagData, ReportData, DashboardStorage, CreateFlagParams, …
  • roboco.models.eventsEvent, EventType (lazy import in usage_events)
  • roboco.events.stream_busStreamEventBus (TYPE_CHECKING only)
  • roboco.services.baseBaseService
  • roboco.services.gitGitService.get_latest_ci_conclusion (telemetry)
  • roboco.services.company_goals, pitch, strategy_engine, task (cockpit)
  • roboco.configsettings (telemetry)
  • roboco.loggingget_logger (telemetry)
  • roboco.utils.convertersto_python_uuid, require_uuid

External:

  • sqlalchemy (func, select, and_, text, func.extract/date_trunc/percentile_cont)
  • sqlalchemy.ext.asyncioAsyncSession
  • structlog (pricing logger)
  • datetime, uuid, dataclasses, typing

Entry Points

  • HTTP routes (roboco/api/routes/usage.py): /api/usage/{summary,time-series,by-agent,by-team,by-model,projection,cache-efficiency,sessions}get_usage_service.
  • HTTP routes (roboco/api/routes/dashboard.py): /api/dashboard/auditor*, /api/dashboard/ceo*, /api/dashboard/kanban/*, /api/dashboard/agents/status, /api/dashboard/activity/recent, /api/dashboard/metrics/{velocity,blockers,team/{team},communication,health,agent/{id},cycle-time,bottlenecks,rework,scorecard/agent/{id},scorecard/team/{team}}get_dashboard_service + get_metrics_service. Auditor/CEO routes gated by _require_auditor_or_ceo.
  • HTTP routes (roboco/api/routes/cockpit.py): /api/cockpit/{summary,signals}get_cockpit_service.
  • Orchestrator loop tick: runtime/orchestrator.py token sweep (~line 5270) → calculate_cost + publish_usage_snapshot; runs per dispatch tick on active agents.
  • Self-heal / CI-watch loop ticks: services/self_heal_engine.py and services/ci_watch_engine.py construct their telemetry source and call fetch() each cycle; armed by their respective ROBOCO_* flags.
  • Grok usage path: llm/providers/grok_cli_usage.py calls calculate_cost per grok session (line 110).
  • Lifespan/CLI: none directly; this slice is pulled on-demand by routes/loops.

Config Flags

No flags live inside this slice's files, but the slice's behavior is gated/parameterized by flags held in roboco/config.py and consumed here:

  • settings.self_heal_project_slug (ROBOCO_SELF_HEAL_PROJECT_SLUG) — empty → GitHubCITelemetrySource.fetch returns no samples (telemetry/source.py:83).
  • settings.self_heal_ci_workflow (ROBOCO_SELF_HEAL_CI_WORKFLOW) — workflow filter for self-heal CI lookup (source.py:86).
  • settings.ci_watch_default_workflow (ROBOCO_CI_WATCH_DEFAULT_WORKFLOW) — fallback workflow for MultiProjectCITelemetrySource (source.py:152).
  • projects.ci_watch_enabled (DB column, migration 048) — per-project opt-in read by the CI-watch engine that drives MultiProjectCITelemetrySource.
  • ROBOCO_SELF_HEAL_ENABLED / ROBOCO_CI_WATCH_ENABLED — arm the loops that call these sources (held in config, consumed by engines, not by source.py itself).
  • Cockpit indirectly honors ROBOCO_STRATEGY_ENGINE_ENABLED / ROBOCO_PROVISIONING_* via the strategy/pitch services it composes.

Gotchas

  • usage.get_summary docstring fixed (536bbb64). The old docstring falsely claimed rollup-based reads. It was corrected in commit 536bbb64 (#66): get_summary is documented as summing raw agent_spawn_sessions rows (sub-day precise); daily_usage_rollups is the day-grain snapshot written by the sweeper and read by get_today_summary; the two can diverge for "today" until the sweeper catches up.
  • ACTIVE_STATUSES excludes BLOCKED in team metrics but get_health_status includes it. metrics.py:61 comment documents this; get_team_metrics (line 234) uses ACTIVE_STATUSES (no BLOCKED) while get_health_status (line 497) uses a local list including BLOCKED. The blocked-task ratio therefore only appears in health, not in team "active_tasks".
  • Doc-coverage is a dev_notes-not-None proxy. get_team_metrics (metrics.py:300) counts completed tasks with non-null dev_notes as "documented" — a simplified heuristic, not real documentation-phase completion.
  • Blocked-hours heuristic improved (536bbb64). get_blocker_metrics now calls _blocked_since_map (metrics.py:176) to read the task.blocked audit row (indexed on target_id/event_type/timestamp) as the authoritative "blocked since" timestamp (#67). Falls back to updated_at or created_at only when no audit row exists. The over-count on non-blocking updates is fixed for tasks that have a proper audit trail.
  • _as_hours is load-bearing for JSON. metrics.py:47 — EXTRACT(epoch …) returns Decimal on PG14+ via asyncpg, which serializes to a JSON string and crashes the panel's value.toFixed(...). Any new "hours" field must go through it.
  • Cycle-time SQL excludes named events by string equality. metrics.py:554 a.event_type = 'task.' || (a.details->>'to_status') keeps only generic transitions. If a future named event's to_status matches a status name AND is stored without the task. prefix convention, it could inject zero-length stages. Relies on the audit-log event-type naming convention being upheld.
  • Rework cost uses agent_spawn_sessions.task_id join. metrics.py:742 — only spawn sessions linked to the reworked task's id contribute; sessions missing the task_id link (e.g. early orchestrator bug) undercount cost.
  • DashboardService flag/report store is an in-memory process singleton (_DashboardStorageHolder, dashboard.py:35) — not DB-backed, not replicated, lost on restart. Flags/reports are ephemeral; do not treat as durable state.
  • DashboardService.get_reports slicing (return result[-limit:], dashboard.py:178) returns the last limit in insertion order but the list is dict-ordered (insertion), not time-ordered — fine while insertion == creation order, but fragile if reports are ever added out of order.
  • Pricing substring match, longest-wins (pricing.py:78). A model name containing both sonnet and a longer fragment (e.g. claude-3-5-sonnet) resolves to the longest fragment entry; the table includes both full names and short aliases (opus, sonnet, haiku) so accidental double-match is handled by longest-wins.
  • Unpriced Anthropic model warns + returns 0.0 (pricing.py:185) — real spend silently counted as $0 in cost panels until the table is updated, because orchestrator callers use the calculate_cost thin wrapper (not calculate_cost_result). CostResult.unpriced=True is now available from calculate_cost_result to distinguish this case, but no caller wires it yet. Unpriced non-Anthropic (Ollama/local) is intentionally $0 with no warning.
  • Cache-efficiency uses hardcoded sonnet pricing (usage.py:401-404, _FULL_INPUT_PRICE=3.00, _CACHE_READ_PRICE=0.30) for the savings estimate regardless of the actual model mix — an aggregate approximation, not per-model.
  • publish_usage_snapshot lazy-imports Event/EventType (usage_events.py:49) to avoid a circular import — callers must keep the bus passed in, not a module-level reference.
  • MultiProjectCITelemetrySource.fetch swallows per-project exceptions (source.py:172) — one bad project never aborts the sweep, but also never surfaces beyond a warning log; a persistently failing project silently contributes no sample (treated as "unknown", not "green" — correct, but invisible).
  • CockpitService.summary basis="proxy" (cockpit.py:57) — every payload is stamped proxy; the over_budget flag is only meaningful once the CEO greenlights real launch.

Drift from CLAUDE.md

  • billing/pricing.py: Grok is priced, not just "Anthropic priced; local/Ollama $0". CLAUDE.md "Cost uses provider-aware pricing in roboco/billing/pricing.py (Anthropic priced; local/Ollama intentionally $0)." omits that xAI Grok (grok-build) is now in the pricing table (pricing.py:63) as a priced non-Anthropic model. Minor incompleteness; behavior is a superset of the claim.
  • CockpitService is undocumented in CLAUDE.md. roboco/services/cockpit.py and /api/cockpit/{summary,signals} are a real CEO-facing read-only aggregation surface not mentioned anywhere in CLAUDE.md's Services table or route inventory.
  • UsageService.get_today_summary / daily_usage_rollups rollup path is not described in CLAUDE.md (which only mentions agent_spawn_sessionsdaily_usage_rollups → dashboard at a high level). The old docstring-level claim that get_summary used rollups was fixed in 536bbb64 (#66) — CLAUDE.md doesn't assert that, so no direct contradiction.
  • usage_events.py / USAGE_SNAPSHOT matches CLAUDE.md's "token sweep also publishes USAGE_SNAPSHOT to /ws/system" — no drift.
  • telemetry/source.py MultiProjectCITelemetrySource matches CLAUDE.md's "Multi-repo CI-watch" section — no drift.
  • All /dashboard/metrics/{cycle-time,bottlenecks,rework,scorecard/agent/{id},scorecard/team/{team}} endpoints exist as documented — no drift.

Changes Since Baseline

git log --oneline fd10cc862c2020b3f639cdb686d427b0198a2441..HEAD -- <scope> and git diff --stat over roboco/services/metrics.py roboco/services/dashboard.py roboco/services/cockpit.py roboco/services/usage.py roboco/services/usage_events.py roboco/services/telemetry/ roboco/billing/ both return empty — no logic-touching commits to this slice since the baseline. The slice is unchanged at HEAD relative to fd10cc86.

Post-snapshot updates (since 2026-06-29): commit 536bbb64 ("Chore/all/logical gaps sweep #286") touched billing/pricing.py, billing/__init__.py, services/metrics.py, and services/usage.py. Changes: (1) CostResult dataclass + calculate_cost_result function added to pricing.py; calculate_cost refactored to a thin wrapper; __init__.py now re-exports all three. (2) _blocked_since_map helper added to MetricsService — reads task.blocked audit row as authoritative "blocked since" (#67); get_blocker_metrics uses it with updated_at fallback. (3) get_summary docstring corrected — no longer falsely claims rollup reads (#66). (4) metrics._blocked_since_map extracted as a xenon complexity refactor (no behavior change beyond the audit-row fix).

Regression Risks

Because the slice is unchanged since baseline, there are no recent changes within the slice that plausibly broke behavior. The risks below are standing landmines (pre-existing) that a future change to upstream data could trip; severity reflects blast radius if triggered.

Title File:Line Claim Severity
Cycle-time SQL depends on audit-log event-type naming convention metrics.py:554 A future named audit event whose to_status resolves under event_type = 'task.' || to_status could inject zero-length stages or skew dwell averages across every cycle-time/bottleneck panel. high
Rework cost join on agent_spawn_sessions.task_id metrics.py:742 If spawn sessions stop populating task_id (orchestrator regression), rework cost silently drops to $0 with no warning — underreported CEO spend. high
Unpriced Anthropic model silently $0 — partially mitigated billing/pricing.py:169 CostResult.unpriced=True is now returned by calculate_cost_result for a missing Anthropic model, but both the orchestrator (orchestrator.py:5209) and grok_cli_usage.py still call the calculate_cost thin-float wrapper — cost panels still show $0. Risk remains until callers switch to calculate_cost_result. medium
get_summary docstring/code mismatch on rollups usage.py:77 RESOLVED (536bbb64 #66): docstring was corrected to accurately describe that get_summary reads raw agent_spawn_sessions, not daily_usage_rollups. low
Blocked-hours heuristic uses updated_at/created_at metrics.py:176 RESOLVED (536bbb64 #67): _blocked_since_map now reads the task.blocked audit row as primary source; falls back to updated_at/created_at only when no audit row. low
DashboardService flag/report store is in-memory singleton dashboard.py:35 Flags/reports vanish on orchestrator restart and are not replicated across instances; an operator relying on them as durable audit trail loses data. medium
ACTIVE_STATUSES excludes BLOCKED in team metrics metrics.py:61 get_team_metrics.active_tasks undercounts vs get_health_status.active_tasks for the same team — two panel cards can show different "active" numbers. low
Cache-efficiency hardcoded sonnet pricing usage.py:401 cost_saved_by_cache_usd is an aggregate approximation that diverges from real per-model savings; misleading if shown next to real cost figures. low
MultiProjectCITelemetrySource swallows per-project errors telemetry/source.py:172 A persistently failing project silently contributes no sample (correct "unknown" semantics) but is invisible beyond a warning log — could mask a config/token rot. low

Health

The slice is internally coherent and well-documented at the method level; the observability reconstruction (cycle-time/bottleneck/rework) is correctly designed around the audit-log event-naming contract and revision_count chokepoint, and the provider-aware pricing is sound. The main integrity concerns are coupling, not correctness: cycle-time and rework-cost are tightly bound to upstream audit-log event naming and agent_spawn_sessions.task_id population, so any drift there silently degrades panels without an in-slice guard. The in-memory DashboardService store is the clearest remaining local hygiene debt (the get_summary docstring mismatch and blocked-hours heuristic were resolved in 536bbb64). Commit 536bbb64 also adds CostResult.unpriced attribution — the mitigation for the silent-$0 risk — though orchestrator callers haven't switched to calculate_cost_result yet. The standing landmines above (especially cycle-time SQL naming convention and rework-cost task_id join) warrant upstream-contract tests.