feat(lambda): cold-start hardening + handler-side retry orchestration (#180)

* feat(lambda): cold-start hardening + handler-side retry orchestration

Two related improvements based on benchmarking the integration at scale
(3454-site sample, multiple iterations).

Cold-start hardening (lambda-entrypoint.sh + lambda_handler.py):
  - Clean stale Xvfb lock file before starting the X server. We observed
    that under cold-start storms, a previous Xvfb sometimes died and left
    /tmp/.X99-lock + /tmp/.X11-unix/X99 behind, so the next start failed
    with "Server is already active for display 99". Removing both files
    makes Xvfb start cleanly every time.
  - Replace `sleep 0.5` with a poll-for-X11-socket loop (up to 10s) plus
    a 200ms post-socket buffer for listen()/accept() to settle. The
    fixed sleep lost the race during concurrent cold inits, surfacing as
    "Looks like you launched a headed browser without having a XServer
    running" failures (~10% rate at 100-concurrent cold-start storm).
  - Add _launch_with_retry helper in the handler: 3 attempts with linear
    backoff (0.3s, 0.6s) on launch_context_async failures. Belt-and-
    suspenders for whatever the entrypoint fix doesn't catch — a retry on
    a now-warm container almost always succeeds.

Handler-side retry orchestration (lambda_handler.py):
  - Add _classify_error() — maps Playwright errors to retry-strategy
    overrides:
      ERR_CERT_*                -> --ignore-certificate-errors + 60s goto
      Timeout exceeded          -> 90s goto + 25s smart_wait cap
      ERR_CONNECTION_TIMED_OUT  -> same as Timeout
    Returns None for unrecoverable site issues (DNS, SSL, refused, HTTP
    4xx/5xx) — those bail immediately without burning a retry slot.
  - Add _attempt_scrape() — extracted scrape body so the retry loop can
    call it with overridden event dicts. Each attempt relaunches the
    browser; uniform behavior across strategies.
  - Rewrite _run() as a retry loop: first attempt uses event verbatim;
    on a classifiable failure, merge the strategy's overrides into the
    event and retry. Bounded by the new `retries` event field (default 1;
    set to 0 to disable retry).
  - Add _raise_with_history() — surfaces a final failure with a
    retry_history block embedded in the error message so callers see
    exactly what was tried before bailing. Successful invocations return
    the standard response shape unchanged — no surprise fields.

INSTRUCTIONS.md updates:
  - Bump function timeout recommendation from 60-120s to 120-180s. Under
    retry, a Timeout-class first failure (30s) plus a longer-budget retry
    (90s) plus cleanup can total ~120-130s; 180s leaves headroom.
  - Document the new `retries` event field in the schema.
  - Add a "Retry orchestration" subsection covering both layers (launch
    retries and strategy retries) with the full strategy table.

Bench results on the 3454-site sample (seed=1):
  v1 baseline (no fixes, c=100):           13.5% failure rate, $1.07
  v2 (entrypoint Xvfb poll only, c=100):    9.9% failure rate, $1.11
  v3 (cold-start fix + bench-side retry):   3.3% failure rate, $1.32
  This change (handler retry, c=250):       2.1% failure rate, $1.13

The remaining 2.1% are all genuinely unrecoverable: DNS doesn't exist,
broken SSL, connection refused, 4xx/5xx responses, payload >6MB Lambda
limit. No retry logic can fix those.

* fix(lambda): merge extra_args on strategy retry instead of clobbering

A flat dict spread replaced caller-supplied extra_args (e.g.
--proxy-server=...) with the strategy's extra_args on a cert retry.
Append both lists so caller flags survive the merge.

examples/integrations/aws_lambda/INSTRUCTIONS.md

@@ -113,6 +113,29 @@ Only `url` is required. Everything else is optional.
 | `screenshot` | bool | `true` |
 | `full_page_screenshot` | bool | `false` |
 
+### Retry orchestration
+
+The handler retries transient navigation failures inline within the same Lambda invocation. Two layers, both built-in:
+
+- **Launch retries** — 3 attempts with 0.3 s + 0.6 s backoff. Recovers Xvfb / Chromium spawn races at cold start. Fast and cheap; not configurable.
+- **Strategy retries** — default 1 attempt, configurable via the `retries` event field. Recovers specific post-launch error classes by relaunching with adjusted Chromium args / page-load budgets.
+
+| Field | Type | Default |
+|---|---|---|
+| `retries` | int | `1` — number of strategy-retry attempts after the first failure. Set to `0` to disable retry entirely. |
+
+Strategies (priority order — first match wins):
+
+| Error pattern | Strategy applied |
+|---|---|
+| `ERR_CERT_*` (any cert error) | `extra_args: ["--ignore-certificate-errors"]`, `goto_timeout_ms: 60000` |
+| `Timeout … exceeded` | `goto_timeout_ms: 90000`, `max_settle_ms: 25000` |
+| `ERR_CONNECTION_TIMED_OUT` | same as `Timeout … exceeded` |
+
+Errors that are **not retried** (no anonymous scraper can recover): `ERR_NAME_NOT_RESOLVED`, `ERR_SSL_PROTOCOL_ERROR`, `ERR_CONNECTION_REFUSED`, `ERR_HTTP_RESPONSE_CODE_FAILURE`. These bail immediately.
+
+On final failure, the raised `RuntimeError`'s message includes a `retry_history` block listing every attempt (strategy applied + error seen). Successful invocations return the standard response shape unchanged — no surprise fields when retries didn't fire.
+
 ### Response
 
 ```json
@@ -140,7 +163,7 @@ Whatever tool you use to create the Lambda function (CLI, CDK, Terraform, SAM, c
 | Package type | Image | Required — this is a container image, not a zip. |
 | Architecture | `arm64` | Roughly 20% cheaper than x86_64. Native build on Apple Silicon. Match the architecture you built for. |
 | Memory | 3008 MB | Memory in Lambda is tied to vCPU. Below ~1769 MB Chromium starts noticeably slower. |
-| Timeout | 60–120 s | Cold start can hit 80+ s on this image; warm invocations are 3–15 s depending on site. |
+| Timeout | 120–180 s | Single-attempt scrapes complete in 3–15 s warm; under retry, a `Timeout`-class first failure (30 s default) plus a longer-budget retry (90 s) plus cleanup can total ~120-130 s. 180 s leaves headroom; below 120 s the function will time out before the retry completes. Cold-start init adds 5-10 s on top. |
 | Ephemeral storage (`/tmp`) | 1024 MB | Chromium profile dirs and screenshots can fill the 512 MB default. |
 | Networking | Default (no VPC) | Binary is baked in, no network needed at cold start. Add VPC + NAT only if your proxy egress requires it. |
 | Execution role | `AWSLambdaBasicExecutionRole` | Just CloudWatch Logs. Add more permissions only if your handler needs them. |

examples/integrations/aws_lambda/lambda-entrypoint.sh

@@ -15,8 +15,25 @@ set -e
 mkdir -p /tmp/.X11-unix
 chmod 1777 /tmp/.X11-unix 2>/dev/null || true
 
+# Clean any stale Xvfb state. If a previous Xvfb died and left its lock file
+# behind (we observed this in cold-start storms), a new Xvfb refuses to start
+# with "Server is already active for display 99". Removing both files makes
+# Xvfb start cleanly every time.
+rm -f /tmp/.X99-lock /tmp/.X11-unix/X99
+
 Xvfb :99 -screen 0 1920x1080x24 -nolisten tcp >/tmp/Xvfb.log 2>&1 &
-sleep 0.5
+
+# Wait for the X11 socket to appear AND for Xvfb to be ready to serve. The
+# socket file appears at bind(), but listen() and the first accept() come
+# slightly later — under cold-start CPU contention this gap matters.
+i=0
+while [ ! -e /tmp/.X11-unix/X99 ] && [ "$i" -lt 200 ]; do
+    i=$((i + 1))
+    sleep 0.05
+done
+# Small buffer after the socket appears so Xvfb has a moment to call listen()
+# and start accepting clients. Cheap insurance against the bind/listen gap.
+sleep 0.2
 
 # Lambda handler shape: exactly one arg, dotted identifier (no spaces, no slashes,
 # no leading dot). `python`, `cloakserve`, `cloaktest`, `bash`, `node` all fail

examples/integrations/aws_lambda/lambda_handler.py

@@ -43,6 +43,19 @@ Event schema (all fields except `url` are optional):
         screenshot              bool         True
         full_page_screenshot    bool         False — capture entire scrollable page
 
+    Retry orchestration:
+        retries     int  default 1. Number of retry attempts after the first
+                          failure. Set to 0 to disable retries entirely (the
+                          handler will fail fast on the first error).
+                          Retried errors:
+                            ERR_CERT_*                -> retry with --ignore-certificate-errors
+                            Timeout exceeded          -> retry with goto_timeout_ms=90000, max_settle_ms=25000
+                            ERR_CONNECTION_TIMED_OUT  -> same as Timeout
+                          Not retried (unrecoverable): ERR_NAME_NOT_RESOLVED,
+                          ERR_SSL_PROTOCOL_ERROR, generic ERR_CONNECTION_REFUSED.
+                          On final failure, the error message includes a
+                          retry_history block with strategy + error per attempt.
+
 Returns:
     {"title": ..., "url": ..., "html": ..., "screenshot_b64"?: ...}
 """
@@ -51,6 +64,7 @@ from __future__ import annotations
 
 import asyncio
 import base64
+import json
 import logging
 import subprocess
 from pathlib import Path
@@ -179,16 +193,69 @@ async def _post_nav_waits(page, event: dict) -> None:
         await page.wait_for_timeout(event["wait_ms"])
 
 
-async def _run(event: dict) -> dict:
+async def _launch_with_retry(event: dict, attempts: int = 3, backoff_s: float = 0.3):
-    url = event["url"]
+    """Retry launch_context_async up to `attempts` times with linear backoff.
 
-    try:
+    Lambda cold-start storms occasionally race Xvfb readiness or hit transient
-        ctx = await launch_context_async(**_build_launch_kwargs(event))
+    Chromium spawn failures — both surface as "Target page, context or browser
-    except Exception as e:
+    has been closed" at launch. The failure is fast (~0.5s) so retries are
-        diag = _diag_snapshot()
+    cheap, and a retry on a now-warm container almost always succeeds.
-        logger.error("launch_context_async failed: %s\nDIAG:\n%s", e, diag)
-        raise RuntimeError(f"launch failed: {e}\n--- DIAG ---\n{diag}") from e
 
+    Pairs with the lock-cleanup + socket-poll in lambda-entrypoint.sh: the
+    entrypoint catches the common case at container init; this catches the
+    residual race when the first invocation hits before Xvfb is fully ready.
+    """
+    last_err: Exception | None = None
+    for i in range(attempts):
+        try:
+            return await launch_context_async(**_build_launch_kwargs(event))
+        except Exception as e:
+            last_err = e
+            logger.warning("launch attempt %d/%d failed: %s",
+                           i + 1, attempts, str(e)[:200])
+            if i + 1 < attempts:
+                await asyncio.sleep(backoff_s * (i + 1))  # 0.3s, 0.6s
+    raise last_err  # type: ignore[misc]
+
+
+def _classify_error(err: Exception) -> dict | None:
+    """Map a Playwright error to a retry-strategy override dict, or None
+    if the error is unrecoverable.
+
+    Match on str(e) because Playwright errors carry their codes inside the
+    message (Error.__str__ includes ERR_CERT_AUTHORITY_INVALID etc.); there
+    is no stable structured `.error_code` attribute to rely on.
+
+    Strategies (priority order — first match wins):
+      ERR_CERT_*                  -> --ignore-certificate-errors + 60s goto budget
+      Timeout exceeded            -> 90s goto budget + 25s smart_wait cap
+      ERR_CONNECTION_TIMED_OUT    -> same as Timeout
+    Returns None for unrecoverable site issues (DNS, SSL, refused, HTTP 4xx/5xx).
+    """
+    msg = str(err)
+    if "ERR_CERT" in msg:
+        return {
+            "extra_args": ["--ignore-certificate-errors"],
+            "goto_timeout_ms": 60000,
+        }
+    if ("Timeout" in msg and "exceeded" in msg) or "ERR_CONNECTION_TIMED_OUT" in msg:
+        return {
+            "goto_timeout_ms": 90000,
+            "max_settle_ms": 25000,
+        }
+    return None
+
+
+async def _attempt_scrape(url: str, event: dict) -> dict:
+    """One self-contained scrape attempt: launch, navigate, wait, capture, close.
+
+    Extracted from `_run` so the retry loop can call it repeatedly with an
+    overridden event dict. Each attempt relaunches the browser — uniform
+    behavior across strategies (the cert-bypass strategy *requires* a relaunch
+    because `--ignore-certificate-errors` is a Chromium CLI arg, not a per-
+    context switch), and the ~3-5s relaunch cost is fine on the slow path.
+    """
+    ctx = await _launch_with_retry(event)
     try:
         page = await ctx.new_page()
         await page.goto(
@@ -217,3 +284,53 @@ async def _run(event: dict) -> dict:
             await ctx.close()
         except Exception:
             pass
+
+
+def _raise_with_history(err: Exception, history: list[dict]) -> None:
+    """Surface a final failure with a retry_history block embedded in the
+    error message, so callers see what was tried before bailing."""
+    diag = _diag_snapshot()
+    if history:
+        diag = "retry_history: " + json.dumps(history, default=str) + "\n\n" + diag
+    logger.error("scrape failed (after %d retries): %s\nDIAG:\n%s",
+                 len(history), err, diag)
+    raise RuntimeError(f"scrape failed: {err}\n--- DIAG ---\n{diag}") from err
+
+
+async def _run(event: dict) -> dict:
+    """Top-level scrape with strategy-based retry orchestration.
+
+    First attempt uses the event verbatim. If it fails with a classifiable
+    error (cert / timeout), retry with that strategy's overrides merged into
+    the event. `retries` bounds the number of strategy retries (default 1;
+    set to 0 to disable retry entirely).
+    """
+    url = event["url"]
+    retries_left = max(0, int(event.get("retries", 1)))
+    history: list[dict] = []
+    current_event = event
+
+    while True:
+        try:
+            return await _attempt_scrape(url, current_event)
+        except Exception as e:
+            if retries_left <= 0:
+                _raise_with_history(e, history)
+            strategy = _classify_error(e)
+            if strategy is None:
+                _raise_with_history(e, history)
+            history.append({
+                "attempt": len(history) + 1,
+                "error": str(e)[:300],
+                "strategy": strategy,
+            })
+            logger.warning("attempt %d failed (%s); retrying with strategy=%s",
+                           len(history), str(e)[:120], strategy)
+            merged_args = list(current_event.get("extra_args", [])) + list(strategy.get("extra_args", []))
+            current_event = {**current_event, **strategy, "extra_args": merged_args}
+            retries_left -= 1
+            # No backoff: strategy overrides change goto budget directly;
+            # the prior failure was either fast (cert reject) or already
+            # waited its full timeout. Container is warm.
+
+