mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
Renn F
aaf6b19303
_on_probe_failure only incremented the failure counter and, at 10 failures, sent a one-shot CEO notification. It never cleared the tracker, never gave up, never fell back to time-expiry. _do_probe returns False for any non-2xx AND any httpx error, so a permanently unreachable probe endpoint (removed API key, network partition to the probe host, misconfigured base URL) kept the provider parked forever — every agent on it gated by _provider_spawn_parked, their tasks reaped to pending but the spawn gate queuing every spawn, sitting pending forever. The only recovery was the operator manually clearing the Redis key. Past _PROBE_GIVE_UP_THRESHOLD (30) persistent failures, fall back to the same time-expiry optimism the unprobeable-provider path uses (_do_probe returns True when there is no probe URL): clear the park and resume parked agents. If the provider is genuinely still down the real workload attempts re-park via the 429/5xx path, so this is bounded burn — strictly better than a silent forever-strand. Kept above the CEO-notify threshold (10) so the operator still gets the notification first.