[F071] abort non-blocking intake/secretary spawn on mid-spawn shutdown

The non-blocking spawn (start_intake_session / start_secretary_session)
schedules _spawn_intake_container_guarded / _spawn_secretary_container_guarded
via _schedule_bg. Those run docker run and only register in _instances at the
END. If shutdown arrived between docker run and the registration line, the
container was started but the orchestrator had no handle — stop() iterates
only _instances, so the container was orphaned (leaked, manual docker rm).
Worse, the F070 drain could let the spawn coroutine complete the
registration AFTER stop() already iterated _instances, landing a live
container into a shutting-down registry nothing tears down.

Add a post-docker-run shutdown guard in _spawn_intake_container and
_spawn_secretary_container: re-check self._running after _run_container_cmd
returns; if the orchestrator began shutting down, remove the just-started
container (by its deterministic name) and raise _SpawnAbortedDuringShutdown
WITHOUT registering. The guarded wrappers catch that BEFORE except Exception
and close the live relay silently (shutdown is not a user-facing failure,
no error pushed to the SSE stream). The F070 stop() drain awaits the bg
spawn coroutine, so the abort surfaces cleanly.

TOCTOU-safe: between the _running check and the _instances assignment there
is no await (config + instance construction are sync), so once the check
passes, registration completes before the event loop can interleave stop().
The normal running path is unchanged (sanity tests pin it).
This commit is contained in:
Renn F
2026-06-28 18:09:47 +02:00
parent 2e3bfb7bbb
commit e15c4e3415
3 changed files with 307 additions and 0 deletions
+44
View File
@@ -744,6 +744,20 @@ class AgentReadinessError(Exception):
"""
class _SpawnAbortedDuringShutdown(Exception):
"""Raised when a non-blocking intake/secretary spawn completes ``docker run``
after the orchestrator began shutting down.
The raiser has already removed the just-started container (so it isn't
orphaned); the guarded wrapper catches this BEFORE its generic
``except Exception`` and closes the live relay silently shutdown is not a
user-facing failure, so no error is pushed to the SSE stream. The F070
``stop()`` drain awaits the bg spawn coroutine, so this surfaces cleanly
instead of the registration landing a live container into a registry that
``stop()`` has already finished iterating.
"""
class AgentOrchestrator:
"""
Manages Claude Code containers for all agents.
@@ -3421,6 +3435,12 @@ class AgentOrchestrator:
project_ids=project_ids,
initial_message=initial_message,
)
except _SpawnAbortedDuringShutdown:
# Shutdown began mid-spawn; the just-started container was already
# removed by the raiser. Close the relay silently — shutdown is not a
# user-facing failure, so no error is pushed to the SSE stream.
get_live_registry().close(session_id)
return
except Exception as exc:
logger.error(
"Intake container spawn failed", session_id=session_id, error=str(exc)
@@ -3500,6 +3520,16 @@ class AgentOrchestrator:
)
container_id = await self._run_container_cmd(cmd)
# Shutdown may have begun while this (non-blocking) spawn was in flight
# — the bg coroutine runs concurrently with stop(). If so, remove the
# just-started container and abort WITHOUT registering: stop()'s
# _instances iteration has already run (or is running), so a registration
# now would land a live container nothing tears down (the orphan). The
# stop() drain awaits this coroutine, so the abort surfaces cleanly.
if not self._running:
await self._remove_container(container_name)
raise _SpawnAbortedDuringShutdown(INTAKE_AGENT_ID)
config = AgentConfig(
agent_id=INTAKE_AGENT_ID,
blueprint_path=prompt_path,
@@ -3595,6 +3625,12 @@ class AgentOrchestrator:
await self._spawn_secretary_container(
session_id, initial_message=initial_message
)
except _SpawnAbortedDuringShutdown:
# Shutdown began mid-spawn; the just-started container was already
# removed by the raiser. Close the relay silently — shutdown is not
# a user-facing failure, so no error is pushed to the SSE stream.
get_live_registry().close(session_id)
return
except Exception as exc:
logger.error(
"Secretary container spawn failed",
@@ -3666,6 +3702,14 @@ class AgentOrchestrator:
)
container_id = await self._run_container_cmd(cmd)
# Shutdown may have begun while this (non-blocking) spawn was in flight
# — see the matching guard in _spawn_intake_container. Remove the
# just-started container and abort WITHOUT registering, so it isn't
# orphaned by a stop() that has already iterated _instances.
if not self._running:
await self._remove_container(container_name)
raise _SpawnAbortedDuringShutdown(SECRETARY_AGENT_ID)
config = AgentConfig(
agent_id=SECRETARY_AGENT_ID,
blueprint_path=prompt_path,