* fix(db): bound lock waits and idle transactions so a parked coroutine can't wedge the pool
2026-07-29 production incident: verb/evidence handlers and background
dispatch coroutines held an open DB transaction across minutes of git
subprocess work and asyncio lock queues (per-workspace ensure locks).
Early writes in those transactions held tasks/agents row locks, every
other write convoyed behind them, and blocked statements camped on pool
connections until all 30 were waiters — 1000+ QueuePool timeouts per
hour, one transaction open 1h20m.
Two layers:
- get_engine now passes asyncpg server_settings:
idle_in_transaction_session_timeout (default 120s) kills any session
parked mid-transaction on non-DB work, releasing its locks and pool
slot; lock_timeout (default 30s) makes a statement queued on someone
else's row lock give up instead of holding a connection for the wait.
Both env-tunable (ROBOCO_DATABASE_IDLE_IN_TRANSACTION_TIMEOUT_MS /
ROBOCO_DATABASE_LOCK_TIMEOUT_MS), 0 disables. Alembic runs its own
sync engine and is untouched; best-effort writers (proactive-context
injection) already swallow errors and now fail in 30s instead of
camping for an hour.
- ContentActions.evidence commits the request session before its
fetch/diff git work, so a multi-minute evidence call no longer pins a
pool connection for the duration (expire_on_commit=False keeps the
loaded task usable; later reads reopen a transaction on demand).
The deeper restructuring — claim flows committing their transition
before briefing/workspace assembly — is scoped to the existing
evidence-assembly-timeout task and not attempted here.
* fix(db): bound lock waits and idle transactions so a parked coroutine can't wedge the pool
2026-07-29 production incident: verb/evidence handlers and background
dispatch coroutines held an open DB transaction across minutes of git
subprocess work and asyncio lock queues (per-workspace ensure locks).
Early writes in those transactions held tasks/agents row locks, every
other write convoyed behind them, and blocked statements camped on pool
connections until all 30 were waiters — 1000+ QueuePool timeouts per
hour, one transaction open 1h20m.
Two layers:
- get_engine now passes asyncpg server_settings:
idle_in_transaction_session_timeout (default 20 min) kills any session
parked mid-transaction on non-DB work, releasing its locks and pool
slot; lock_timeout (default 60s) makes a statement queued on someone
else's row lock give up with a clean retryable error instead of
camping on a pool connection for the wait. Both env-tunable
(ROBOCO_DATABASE_IDLE_IN_TRANSACTION_TIMEOUT_MS /
ROBOCO_DATABASE_LOCK_TIMEOUT_MS), 0 disables. The idle default
deliberately clears the longest LEGITIMATE in-transaction window — a
cold-workspace claim holds its transaction across the clone (300s
budget) + dep install (600s budget) under the 900s slow-verb wall —
so routine claims never trip it while today's 80-minute parked
transaction dies at 20 min. Alembic's env.py builds its own engine
and never carries these; best-effort writers (proactive-context
injection) already swallow errors and now fail in 60s instead of
camping for an hour.
- ContentActions.evidence ends the request transaction (commit, or
rollback on a poisoned session) before its fetch/diff git work, so a
multi-minute evidence call no longer pins a pool connection for the
duration (expire_on_commit=False keeps the loaded task usable; later
reads reopen a transaction on demand).
The deeper restructuring — claim flows committing their transition
before briefing/workspace assembly — is scoped to the existing
evidence-assembly-timeout task and not attempted here.
---------
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
