mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
[F071] abort non-blocking intake/secretary spawn on mid-spawn shutdown
The non-blocking spawn (start_intake_session / start_secretary_session) schedules _spawn_intake_container_guarded / _spawn_secretary_container_guarded via _schedule_bg. Those run docker run and only register in _instances at the END. If shutdown arrived between docker run and the registration line, the container was started but the orchestrator had no handle — stop() iterates only _instances, so the container was orphaned (leaked, manual docker rm). Worse, the F070 drain could let the spawn coroutine complete the registration AFTER stop() already iterated _instances, landing a live container into a shutting-down registry nothing tears down. Add a post-docker-run shutdown guard in _spawn_intake_container and _spawn_secretary_container: re-check self._running after _run_container_cmd returns; if the orchestrator began shutting down, remove the just-started container (by its deterministic name) and raise _SpawnAbortedDuringShutdown WITHOUT registering. The guarded wrappers catch that BEFORE except Exception and close the live relay silently (shutdown is not a user-facing failure, no error pushed to the SSE stream). The F070 stop() drain awaits the bg spawn coroutine, so the abort surfaces cleanly. TOCTOU-safe: between the _running check and the _instances assignment there is no await (config + instance construction are sync), so once the check passes, registration completes before the event loop can interleave stop(). The normal running path is unchanged (sanity tests pin it).
This commit is contained in:
@@ -744,6 +744,20 @@ class AgentReadinessError(Exception):
|
||||
"""
|
||||
|
||||
|
||||
class _SpawnAbortedDuringShutdown(Exception):
|
||||
"""Raised when a non-blocking intake/secretary spawn completes ``docker run``
|
||||
after the orchestrator began shutting down.
|
||||
|
||||
The raiser has already removed the just-started container (so it isn't
|
||||
orphaned); the guarded wrapper catches this BEFORE its generic
|
||||
``except Exception`` and closes the live relay silently — shutdown is not a
|
||||
user-facing failure, so no error is pushed to the SSE stream. The F070
|
||||
``stop()`` drain awaits the bg spawn coroutine, so this surfaces cleanly
|
||||
instead of the registration landing a live container into a registry that
|
||||
``stop()`` has already finished iterating.
|
||||
"""
|
||||
|
||||
|
||||
class AgentOrchestrator:
|
||||
"""
|
||||
Manages Claude Code containers for all agents.
|
||||
@@ -3421,6 +3435,12 @@ class AgentOrchestrator:
|
||||
project_ids=project_ids,
|
||||
initial_message=initial_message,
|
||||
)
|
||||
except _SpawnAbortedDuringShutdown:
|
||||
# Shutdown began mid-spawn; the just-started container was already
|
||||
# removed by the raiser. Close the relay silently — shutdown is not a
|
||||
# user-facing failure, so no error is pushed to the SSE stream.
|
||||
get_live_registry().close(session_id)
|
||||
return
|
||||
except Exception as exc:
|
||||
logger.error(
|
||||
"Intake container spawn failed", session_id=session_id, error=str(exc)
|
||||
@@ -3500,6 +3520,16 @@ class AgentOrchestrator:
|
||||
)
|
||||
container_id = await self._run_container_cmd(cmd)
|
||||
|
||||
# Shutdown may have begun while this (non-blocking) spawn was in flight
|
||||
# — the bg coroutine runs concurrently with stop(). If so, remove the
|
||||
# just-started container and abort WITHOUT registering: stop()'s
|
||||
# _instances iteration has already run (or is running), so a registration
|
||||
# now would land a live container nothing tears down (the orphan). The
|
||||
# stop() drain awaits this coroutine, so the abort surfaces cleanly.
|
||||
if not self._running:
|
||||
await self._remove_container(container_name)
|
||||
raise _SpawnAbortedDuringShutdown(INTAKE_AGENT_ID)
|
||||
|
||||
config = AgentConfig(
|
||||
agent_id=INTAKE_AGENT_ID,
|
||||
blueprint_path=prompt_path,
|
||||
@@ -3595,6 +3625,12 @@ class AgentOrchestrator:
|
||||
await self._spawn_secretary_container(
|
||||
session_id, initial_message=initial_message
|
||||
)
|
||||
except _SpawnAbortedDuringShutdown:
|
||||
# Shutdown began mid-spawn; the just-started container was already
|
||||
# removed by the raiser. Close the relay silently — shutdown is not
|
||||
# a user-facing failure, so no error is pushed to the SSE stream.
|
||||
get_live_registry().close(session_id)
|
||||
return
|
||||
except Exception as exc:
|
||||
logger.error(
|
||||
"Secretary container spawn failed",
|
||||
@@ -3666,6 +3702,14 @@ class AgentOrchestrator:
|
||||
)
|
||||
container_id = await self._run_container_cmd(cmd)
|
||||
|
||||
# Shutdown may have begun while this (non-blocking) spawn was in flight
|
||||
# — see the matching guard in _spawn_intake_container. Remove the
|
||||
# just-started container and abort WITHOUT registering, so it isn't
|
||||
# orphaned by a stop() that has already iterated _instances.
|
||||
if not self._running:
|
||||
await self._remove_container(container_name)
|
||||
raise _SpawnAbortedDuringShutdown(SECRETARY_AGENT_ID)
|
||||
|
||||
config = AgentConfig(
|
||||
agent_id=SECRETARY_AGENT_ID,
|
||||
blueprint_path=prompt_path,
|
||||
|
||||
Reference in New Issue
Block a user