From 9eb90da012e7279060cf939640f64bef2e6e12f7 Mon Sep 17 00:00:00 2001 From: Alex Stepansky Date: Mon, 27 Apr 2026 21:33:04 -0400 Subject: [PATCH] feat(lambda): cold-start hardening + handler-side retry orchestration (#180) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(lambda): cold-start hardening + handler-side retry orchestration Two related improvements based on benchmarking the integration at scale (3454-site sample, multiple iterations). Cold-start hardening (lambda-entrypoint.sh + lambda_handler.py): - Clean stale Xvfb lock file before starting the X server. We observed that under cold-start storms, a previous Xvfb sometimes died and left /tmp/.X99-lock + /tmp/.X11-unix/X99 behind, so the next start failed with "Server is already active for display 99". Removing both files makes Xvfb start cleanly every time. - Replace `sleep 0.5` with a poll-for-X11-socket loop (up to 10s) plus a 200ms post-socket buffer for listen()/accept() to settle. The fixed sleep lost the race during concurrent cold inits, surfacing as "Looks like you launched a headed browser without having a XServer running" failures (~10% rate at 100-concurrent cold-start storm). - Add _launch_with_retry helper in the handler: 3 attempts with linear backoff (0.3s, 0.6s) on launch_context_async failures. Belt-and- suspenders for whatever the entrypoint fix doesn't catch — a retry on a now-warm container almost always succeeds. Handler-side retry orchestration (lambda_handler.py): - Add _classify_error() — maps Playwright errors to retry-strategy overrides: ERR_CERT_* -> --ignore-certificate-errors + 60s goto Timeout exceeded -> 90s goto + 25s smart_wait cap ERR_CONNECTION_TIMED_OUT -> same as Timeout Returns None for unrecoverable site issues (DNS, SSL, refused, HTTP 4xx/5xx) — those bail immediately without burning a retry slot. - Add _attempt_scrape() — extracted scrape body so the retry loop can call it with overridden event dicts. Each attempt relaunches the browser; uniform behavior across strategies. - Rewrite _run() as a retry loop: first attempt uses event verbatim; on a classifiable failure, merge the strategy's overrides into the event and retry. Bounded by the new `retries` event field (default 1; set to 0 to disable retry). - Add _raise_with_history() — surfaces a final failure with a retry_history block embedded in the error message so callers see exactly what was tried before bailing. Successful invocations return the standard response shape unchanged — no surprise fields. INSTRUCTIONS.md updates: - Bump function timeout recommendation from 60-120s to 120-180s. Under retry, a Timeout-class first failure (30s) plus a longer-budget retry (90s) plus cleanup can total ~120-130s; 180s leaves headroom. - Document the new `retries` event field in the schema. - Add a "Retry orchestration" subsection covering both layers (launch retries and strategy retries) with the full strategy table. Bench results on the 3454-site sample (seed=1): v1 baseline (no fixes, c=100): 13.5% failure rate, $1.07 v2 (entrypoint Xvfb poll only, c=100): 9.9% failure rate, $1.11 v3 (cold-start fix + bench-side retry): 3.3% failure rate, $1.32 This change (handler retry, c=250): 2.1% failure rate, $1.13 The remaining 2.1% are all genuinely unrecoverable: DNS doesn't exist, broken SSL, connection refused, 4xx/5xx responses, payload >6MB Lambda limit. No retry logic can fix those. * fix(lambda): merge extra_args on strategy retry instead of clobbering A flat dict spread replaced caller-supplied extra_args (e.g. --proxy-server=...) with the strategy's extra_args on a cert retry. Append both lists so caller flags survive the merge. --- .../integrations/aws_lambda/INSTRUCTIONS.md | 25 +++- .../aws_lambda/lambda-entrypoint.sh | 19 ++- .../integrations/aws_lambda/lambda_handler.py | 133 ++++++++++++++++-- 3 files changed, 167 insertions(+), 10 deletions(-) diff --git a/examples/integrations/aws_lambda/INSTRUCTIONS.md b/examples/integrations/aws_lambda/INSTRUCTIONS.md index f68f2e7..df44d0e 100644 --- a/examples/integrations/aws_lambda/INSTRUCTIONS.md +++ b/examples/integrations/aws_lambda/INSTRUCTIONS.md @@ -113,6 +113,29 @@ Only `url` is required. Everything else is optional. | `screenshot` | bool | `true` | | `full_page_screenshot` | bool | `false` | +### Retry orchestration + +The handler retries transient navigation failures inline within the same Lambda invocation. Two layers, both built-in: + +- **Launch retries** — 3 attempts with 0.3 s + 0.6 s backoff. Recovers Xvfb / Chromium spawn races at cold start. Fast and cheap; not configurable. +- **Strategy retries** — default 1 attempt, configurable via the `retries` event field. Recovers specific post-launch error classes by relaunching with adjusted Chromium args / page-load budgets. + +| Field | Type | Default | +|---|---|---| +| `retries` | int | `1` — number of strategy-retry attempts after the first failure. Set to `0` to disable retry entirely. | + +Strategies (priority order — first match wins): + +| Error pattern | Strategy applied | +|---|---| +| `ERR_CERT_*` (any cert error) | `extra_args: ["--ignore-certificate-errors"]`, `goto_timeout_ms: 60000` | +| `Timeout … exceeded` | `goto_timeout_ms: 90000`, `max_settle_ms: 25000` | +| `ERR_CONNECTION_TIMED_OUT` | same as `Timeout … exceeded` | + +Errors that are **not retried** (no anonymous scraper can recover): `ERR_NAME_NOT_RESOLVED`, `ERR_SSL_PROTOCOL_ERROR`, `ERR_CONNECTION_REFUSED`, `ERR_HTTP_RESPONSE_CODE_FAILURE`. These bail immediately. + +On final failure, the raised `RuntimeError`'s message includes a `retry_history` block listing every attempt (strategy applied + error seen). Successful invocations return the standard response shape unchanged — no surprise fields when retries didn't fire. + ### Response ```json @@ -140,7 +163,7 @@ Whatever tool you use to create the Lambda function (CLI, CDK, Terraform, SAM, c | Package type | Image | Required — this is a container image, not a zip. | | Architecture | `arm64` | Roughly 20% cheaper than x86_64. Native build on Apple Silicon. Match the architecture you built for. | | Memory | 3008 MB | Memory in Lambda is tied to vCPU. Below ~1769 MB Chromium starts noticeably slower. | -| Timeout | 60–120 s | Cold start can hit 80+ s on this image; warm invocations are 3–15 s depending on site. | +| Timeout | 120–180 s | Single-attempt scrapes complete in 3–15 s warm; under retry, a `Timeout`-class first failure (30 s default) plus a longer-budget retry (90 s) plus cleanup can total ~120-130 s. 180 s leaves headroom; below 120 s the function will time out before the retry completes. Cold-start init adds 5-10 s on top. | | Ephemeral storage (`/tmp`) | 1024 MB | Chromium profile dirs and screenshots can fill the 512 MB default. | | Networking | Default (no VPC) | Binary is baked in, no network needed at cold start. Add VPC + NAT only if your proxy egress requires it. | | Execution role | `AWSLambdaBasicExecutionRole` | Just CloudWatch Logs. Add more permissions only if your handler needs them. | diff --git a/examples/integrations/aws_lambda/lambda-entrypoint.sh b/examples/integrations/aws_lambda/lambda-entrypoint.sh index 7674a9c..0118084 100644 --- a/examples/integrations/aws_lambda/lambda-entrypoint.sh +++ b/examples/integrations/aws_lambda/lambda-entrypoint.sh @@ -15,8 +15,25 @@ set -e mkdir -p /tmp/.X11-unix chmod 1777 /tmp/.X11-unix 2>/dev/null || true +# Clean any stale Xvfb state. If a previous Xvfb died and left its lock file +# behind (we observed this in cold-start storms), a new Xvfb refuses to start +# with "Server is already active for display 99". Removing both files makes +# Xvfb start cleanly every time. +rm -f /tmp/.X99-lock /tmp/.X11-unix/X99 + Xvfb :99 -screen 0 1920x1080x24 -nolisten tcp >/tmp/Xvfb.log 2>&1 & -sleep 0.5 + +# Wait for the X11 socket to appear AND for Xvfb to be ready to serve. The +# socket file appears at bind(), but listen() and the first accept() come +# slightly later — under cold-start CPU contention this gap matters. +i=0 +while [ ! -e /tmp/.X11-unix/X99 ] && [ "$i" -lt 200 ]; do + i=$((i + 1)) + sleep 0.05 +done +# Small buffer after the socket appears so Xvfb has a moment to call listen() +# and start accepting clients. Cheap insurance against the bind/listen gap. +sleep 0.2 # Lambda handler shape: exactly one arg, dotted identifier (no spaces, no slashes, # no leading dot). `python`, `cloakserve`, `cloaktest`, `bash`, `node` all fail diff --git a/examples/integrations/aws_lambda/lambda_handler.py b/examples/integrations/aws_lambda/lambda_handler.py index 7f5c583..5c6f3c6 100644 --- a/examples/integrations/aws_lambda/lambda_handler.py +++ b/examples/integrations/aws_lambda/lambda_handler.py @@ -43,6 +43,19 @@ Event schema (all fields except `url` are optional): screenshot bool True full_page_screenshot bool False — capture entire scrollable page + Retry orchestration: + retries int default 1. Number of retry attempts after the first + failure. Set to 0 to disable retries entirely (the + handler will fail fast on the first error). + Retried errors: + ERR_CERT_* -> retry with --ignore-certificate-errors + Timeout exceeded -> retry with goto_timeout_ms=90000, max_settle_ms=25000 + ERR_CONNECTION_TIMED_OUT -> same as Timeout + Not retried (unrecoverable): ERR_NAME_NOT_RESOLVED, + ERR_SSL_PROTOCOL_ERROR, generic ERR_CONNECTION_REFUSED. + On final failure, the error message includes a + retry_history block with strategy + error per attempt. + Returns: {"title": ..., "url": ..., "html": ..., "screenshot_b64"?: ...} """ @@ -51,6 +64,7 @@ from __future__ import annotations import asyncio import base64 +import json import logging import subprocess from pathlib import Path @@ -179,16 +193,69 @@ async def _post_nav_waits(page, event: dict) -> None: await page.wait_for_timeout(event["wait_ms"]) -async def _run(event: dict) -> dict: - url = event["url"] +async def _launch_with_retry(event: dict, attempts: int = 3, backoff_s: float = 0.3): + """Retry launch_context_async up to `attempts` times with linear backoff. - try: - ctx = await launch_context_async(**_build_launch_kwargs(event)) - except Exception as e: - diag = _diag_snapshot() - logger.error("launch_context_async failed: %s\nDIAG:\n%s", e, diag) - raise RuntimeError(f"launch failed: {e}\n--- DIAG ---\n{diag}") from e + Lambda cold-start storms occasionally race Xvfb readiness or hit transient + Chromium spawn failures — both surface as "Target page, context or browser + has been closed" at launch. The failure is fast (~0.5s) so retries are + cheap, and a retry on a now-warm container almost always succeeds. + Pairs with the lock-cleanup + socket-poll in lambda-entrypoint.sh: the + entrypoint catches the common case at container init; this catches the + residual race when the first invocation hits before Xvfb is fully ready. + """ + last_err: Exception | None = None + for i in range(attempts): + try: + return await launch_context_async(**_build_launch_kwargs(event)) + except Exception as e: + last_err = e + logger.warning("launch attempt %d/%d failed: %s", + i + 1, attempts, str(e)[:200]) + if i + 1 < attempts: + await asyncio.sleep(backoff_s * (i + 1)) # 0.3s, 0.6s + raise last_err # type: ignore[misc] + + +def _classify_error(err: Exception) -> dict | None: + """Map a Playwright error to a retry-strategy override dict, or None + if the error is unrecoverable. + + Match on str(e) because Playwright errors carry their codes inside the + message (Error.__str__ includes ERR_CERT_AUTHORITY_INVALID etc.); there + is no stable structured `.error_code` attribute to rely on. + + Strategies (priority order — first match wins): + ERR_CERT_* -> --ignore-certificate-errors + 60s goto budget + Timeout exceeded -> 90s goto budget + 25s smart_wait cap + ERR_CONNECTION_TIMED_OUT -> same as Timeout + Returns None for unrecoverable site issues (DNS, SSL, refused, HTTP 4xx/5xx). + """ + msg = str(err) + if "ERR_CERT" in msg: + return { + "extra_args": ["--ignore-certificate-errors"], + "goto_timeout_ms": 60000, + } + if ("Timeout" in msg and "exceeded" in msg) or "ERR_CONNECTION_TIMED_OUT" in msg: + return { + "goto_timeout_ms": 90000, + "max_settle_ms": 25000, + } + return None + + +async def _attempt_scrape(url: str, event: dict) -> dict: + """One self-contained scrape attempt: launch, navigate, wait, capture, close. + + Extracted from `_run` so the retry loop can call it repeatedly with an + overridden event dict. Each attempt relaunches the browser — uniform + behavior across strategies (the cert-bypass strategy *requires* a relaunch + because `--ignore-certificate-errors` is a Chromium CLI arg, not a per- + context switch), and the ~3-5s relaunch cost is fine on the slow path. + """ + ctx = await _launch_with_retry(event) try: page = await ctx.new_page() await page.goto( @@ -217,3 +284,53 @@ async def _run(event: dict) -> dict: await ctx.close() except Exception: pass + + +def _raise_with_history(err: Exception, history: list[dict]) -> None: + """Surface a final failure with a retry_history block embedded in the + error message, so callers see what was tried before bailing.""" + diag = _diag_snapshot() + if history: + diag = "retry_history: " + json.dumps(history, default=str) + "\n\n" + diag + logger.error("scrape failed (after %d retries): %s\nDIAG:\n%s", + len(history), err, diag) + raise RuntimeError(f"scrape failed: {err}\n--- DIAG ---\n{diag}") from err + + +async def _run(event: dict) -> dict: + """Top-level scrape with strategy-based retry orchestration. + + First attempt uses the event verbatim. If it fails with a classifiable + error (cert / timeout), retry with that strategy's overrides merged into + the event. `retries` bounds the number of strategy retries (default 1; + set to 0 to disable retry entirely). + """ + url = event["url"] + retries_left = max(0, int(event.get("retries", 1))) + history: list[dict] = [] + current_event = event + + while True: + try: + return await _attempt_scrape(url, current_event) + except Exception as e: + if retries_left <= 0: + _raise_with_history(e, history) + strategy = _classify_error(e) + if strategy is None: + _raise_with_history(e, history) + history.append({ + "attempt": len(history) + 1, + "error": str(e)[:300], + "strategy": strategy, + }) + logger.warning("attempt %d failed (%s); retrying with strategy=%s", + len(history), str(e)[:120], strategy) + merged_args = list(current_event.get("extra_args", [])) + list(strategy.get("extra_args", [])) + current_event = {**current_event, **strategy, "extra_args": merged_args} + retries_left -= 1 + # No backoff: strategy overrides change goto budget directly; + # the prior failure was either fast (cert reject) or already + # waited its full timeout. Container is warm. + +