mirror of
https://github.com/CloakHQ/CloakBrowser.git
synced 2026-06-23 11:41:46 +02:00
Alex StepanskyandGitHub
9eb90da012
* feat(lambda): cold-start hardening + handler-side retry orchestration
Two related improvements based on benchmarking the integration at scale
(3454-site sample, multiple iterations).
Cold-start hardening (lambda-entrypoint.sh + lambda_handler.py):
- Clean stale Xvfb lock file before starting the X server. We observed
that under cold-start storms, a previous Xvfb sometimes died and left
/tmp/.X99-lock + /tmp/.X11-unix/X99 behind, so the next start failed
with "Server is already active for display 99". Removing both files
makes Xvfb start cleanly every time.
- Replace `sleep 0.5` with a poll-for-X11-socket loop (up to 10s) plus
a 200ms post-socket buffer for listen()/accept() to settle. The
fixed sleep lost the race during concurrent cold inits, surfacing as
"Looks like you launched a headed browser without having a XServer
running" failures (~10% rate at 100-concurrent cold-start storm).
- Add _launch_with_retry helper in the handler: 3 attempts with linear
backoff (0.3s, 0.6s) on launch_context_async failures. Belt-and-
suspenders for whatever the entrypoint fix doesn't catch — a retry on
a now-warm container almost always succeeds.
Handler-side retry orchestration (lambda_handler.py):
- Add _classify_error() — maps Playwright errors to retry-strategy
overrides:
ERR_CERT_* -> --ignore-certificate-errors + 60s goto
Timeout exceeded -> 90s goto + 25s smart_wait cap
ERR_CONNECTION_TIMED_OUT -> same as Timeout
Returns None for unrecoverable site issues (DNS, SSL, refused, HTTP
4xx/5xx) — those bail immediately without burning a retry slot.
- Add _attempt_scrape() — extracted scrape body so the retry loop can
call it with overridden event dicts. Each attempt relaunches the
browser; uniform behavior across strategies.
- Rewrite _run() as a retry loop: first attempt uses event verbatim;
on a classifiable failure, merge the strategy's overrides into the
event and retry. Bounded by the new `retries` event field (default 1;
set to 0 to disable retry).
- Add _raise_with_history() — surfaces a final failure with a
retry_history block embedded in the error message so callers see
exactly what was tried before bailing. Successful invocations return
the standard response shape unchanged — no surprise fields.
INSTRUCTIONS.md updates:
- Bump function timeout recommendation from 60-120s to 120-180s. Under
retry, a Timeout-class first failure (30s) plus a longer-budget retry
(90s) plus cleanup can total ~120-130s; 180s leaves headroom.
- Document the new `retries` event field in the schema.
- Add a "Retry orchestration" subsection covering both layers (launch
retries and strategy retries) with the full strategy table.
Bench results on the 3454-site sample (seed=1):
v1 baseline (no fixes, c=100): 13.5% failure rate, $1.07
v2 (entrypoint Xvfb poll only, c=100): 9.9% failure rate, $1.11
v3 (cold-start fix + bench-side retry): 3.3% failure rate, $1.32
This change (handler retry, c=250): 2.1% failure rate, $1.13
The remaining 2.1% are all genuinely unrecoverable: DNS doesn't exist,
broken SSL, connection refused, 4xx/5xx responses, payload >6MB Lambda
limit. No retry logic can fix those.
* fix(lambda): merge extra_args on strategy retry instead of clobbering
A flat dict spread replaced caller-supplied extra_args (e.g.
--proxy-server=...) with the strategy's extra_args on a cert retry.
Append both lists so caller flags survive the merge.
182 lines
11 KiB
Markdown
182 lines
11 KiB
Markdown
# CloakBrowser on AWS Lambda
|
||
|
||
Run stealth Chromium one-shot scrapes inside an AWS Lambda function (container image package type). The image derives directly from the official CloakHQ Docker Hub image (`cloakhq/cloakbrowser`) and adds Lambda runtime support on top — Lambda is an additional invocation surface, not a replacement. Every other surface from the canonical image (`python`, `cloakserve`, `cloaktest`, `node`, `bash`, examples) keeps working.
|
||
|
||
This document covers what the image is, how to build and locally test it, and the event/response contract. **It does not prescribe a deployment method** — push the resulting image to ECR and create the Lambda function however you prefer (AWS CLI, CDK, Terraform, SAM, console, etc.). Configuration tips for whichever tool you use are at the bottom.
|
||
|
||
## Files in this directory
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `Dockerfile` | `FROM cloakhq/cloakbrowser` plus a thin Lambda layer. Self-contained — no files outside this directory are referenced. |
|
||
| `lambda-entrypoint.sh` | Dual-mode entrypoint. Starts Xvfb, then routes `module.func` CMDs through `awslambdaric` (via the bundled `aws-lambda-rie` locally, or the AWS Runtime API in production), and execs everything else (`python`, `cloakserve`, `cloaktest`, `node`, `bash`) directly. |
|
||
| `lambda_handler.py` | Default handler. Takes `{url, ...}`, returns `{title, url, html, screenshot_b64?}`. Always headed via Xvfb. |
|
||
| `INSTRUCTIONS.md` | This file. |
|
||
|
||
The Lambda layer is ~30 lines on top of the official image — no apt list, no Node install, no JS-wrapper build, no Chromium download. The canonical CloakHQ image owns those.
|
||
|
||
This directory is **standalone**: copy or clone it anywhere (its own repo, a subdirectory of an existing project, a CI artifact bundle) and the build still works. It depends only on the upstream `cloakhq/cloakbrowser` image on Docker Hub and the `aws-lambda-rie` binary on GitHub Releases — both fetched at build time.
|
||
|
||
## Build
|
||
|
||
From inside this directory:
|
||
|
||
```bash
|
||
docker buildx build --platform linux/arm64 -t cloakbrowser-lambda:arm64 --load .
|
||
```
|
||
|
||
Or from anywhere, pointing at this directory as the build context:
|
||
|
||
```bash
|
||
docker buildx build --platform linux/arm64 \
|
||
-f path/to/aws_lambda/Dockerfile \
|
||
-t cloakbrowser-lambda:arm64 --load \
|
||
path/to/aws_lambda
|
||
```
|
||
|
||
The build pulls `cloakhq/cloakbrowser:latest` from Docker Hub and adds the Lambda layer on top. Pin a specific tag (e.g. `cloakhq/cloakbrowser:0.3.25`) in the `FROM` line for reproducible builds; `latest` floats with the upstream release cadence.
|
||
|
||
For x86_64, switch `--platform linux/amd64` (slower on Apple Silicon under emulation).
|
||
|
||
## Local smoke test (no AWS account needed)
|
||
|
||
> **What's the RIE?** Lambda container images can't be run with a plain `docker run` — they expect to talk to AWS's Runtime API (the HTTP service Lambda exposes inside its sandbox to deliver events and collect responses). AWS publishes a small binary called the **Runtime Interface Emulator** that stands up a fake Runtime API on localhost so you can test the container exactly the way Lambda will invoke it, without deploying. We bake the RIE into the image, and the dual-mode entrypoint uses it automatically when `AWS_LAMBDA_RUNTIME_API` isn't set (i.e. you're not running in real Lambda).
|
||
|
||
The image bakes in `aws-lambda-rie`, so the standard Lambda local-invoke endpoint works without mounting anything:
|
||
|
||
```bash
|
||
docker run --rm -p 9000:8080 cloakbrowser-lambda:arm64
|
||
|
||
# In another shell:
|
||
curl -sS -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \
|
||
-d '{"url":"https://example.com"}'
|
||
```
|
||
|
||
Other invocation surfaces stay intact (these match the canonical CloakHQ image):
|
||
|
||
```bash
|
||
docker run --rm -it cloakbrowser-lambda:arm64 python # REPL
|
||
docker run --rm cloakbrowser-lambda:arm64 python examples/basic.py # examples
|
||
docker run --rm -p 9222:9222 cloakbrowser-lambda:arm64 cloakserve --port=9222 # CDP server
|
||
docker run --rm cloakbrowser-lambda:arm64 cloaktest # stealth tests
|
||
docker run --rm -it cloakbrowser-lambda:arm64 node # JS wrapper
|
||
```
|
||
|
||
## Event schema
|
||
|
||
Only `url` is required. Everything else is optional.
|
||
|
||
### Launch options (forwarded to `cloakbrowser.launch_context_async`)
|
||
|
||
| Field | Type | Default |
|
||
|---|---|---|
|
||
| `url` | str | required |
|
||
| `proxy` | str / dict | none — `http://user:pass@host:port` or a Playwright proxy dict |
|
||
| `humanize` | bool | `false` — enable human-like mouse / keyboard / scroll |
|
||
| `human_preset` | str | `"default"` or `"careful"` |
|
||
| `geoip` | bool | `false` — auto timezone+locale from proxy IP |
|
||
| `timezone` | str | none — IANA tz, e.g. `"America/New_York"` |
|
||
| `locale` | str | none — BCP-47, e.g. `"en-US"` |
|
||
| `viewport` | `{width,height}` | `1920x947` (cloakbrowser default) |
|
||
| `user_agent` | str | none |
|
||
| `extra_args` | `list[str]` | `[]` — extra Chromium CLI flags |
|
||
|
||
### Navigation
|
||
|
||
| Field | Type | Default |
|
||
|---|---|---|
|
||
| `wait_until` | str | `"domcontentloaded"` — `load` / `domcontentloaded` / `networkidle` / `commit` |
|
||
| `goto_timeout_ms` | int | `30000` |
|
||
|
||
### Post-navigation waits
|
||
|
||
`smart_wait` is the default when no other wait is specified. It polls `document.documentElement.outerHTML.length` and returns when the size hasn't changed for `dom_stable_ms`. Robust for at-scale scraping because it ignores network activity (analytics beacons, long-poll, websockets) that doesn't mutate the DOM — `wait_until: "networkidle"` is unreliable on modern SPAs for exactly this reason.
|
||
|
||
| Field | Type | Default |
|
||
|---|---|---|
|
||
| `smart_wait` | bool | `true` if no other wait is set |
|
||
| `dom_stable_ms` | int | `1500` |
|
||
| `max_settle_ms` | int | `15000` |
|
||
| `wait_for_load_state` | str | none — `load` / `domcontentloaded` / `networkidle` |
|
||
| `wait_for_load_state_timeout_ms` | int | `30000` |
|
||
| `wait_for_selector` | str | none — CSS or XPath |
|
||
| `wait_for_selector_state` | str | `"visible"` — also `attached` / `detached` / `hidden` |
|
||
| `wait_for_selector_timeout_ms` | int | `30000` |
|
||
| `wait_for_function` | str | none — JS expression returning truthy when ready |
|
||
| `wait_for_function_timeout_ms` | int | `30000` |
|
||
| `wait_ms` | int | none — fixed pause |
|
||
|
||
### Capture
|
||
|
||
| Field | Type | Default |
|
||
|---|---|---|
|
||
| `screenshot` | bool | `true` |
|
||
| `full_page_screenshot` | bool | `false` |
|
||
|
||
### Retry orchestration
|
||
|
||
The handler retries transient navigation failures inline within the same Lambda invocation. Two layers, both built-in:
|
||
|
||
- **Launch retries** — 3 attempts with 0.3 s + 0.6 s backoff. Recovers Xvfb / Chromium spawn races at cold start. Fast and cheap; not configurable.
|
||
- **Strategy retries** — default 1 attempt, configurable via the `retries` event field. Recovers specific post-launch error classes by relaunching with adjusted Chromium args / page-load budgets.
|
||
|
||
| Field | Type | Default |
|
||
|---|---|---|
|
||
| `retries` | int | `1` — number of strategy-retry attempts after the first failure. Set to `0` to disable retry entirely. |
|
||
|
||
Strategies (priority order — first match wins):
|
||
|
||
| Error pattern | Strategy applied |
|
||
|---|---|
|
||
| `ERR_CERT_*` (any cert error) | `extra_args: ["--ignore-certificate-errors"]`, `goto_timeout_ms: 60000` |
|
||
| `Timeout … exceeded` | `goto_timeout_ms: 90000`, `max_settle_ms: 25000` |
|
||
| `ERR_CONNECTION_TIMED_OUT` | same as `Timeout … exceeded` |
|
||
|
||
Errors that are **not retried** (no anonymous scraper can recover): `ERR_NAME_NOT_RESOLVED`, `ERR_SSL_PROTOCOL_ERROR`, `ERR_CONNECTION_REFUSED`, `ERR_HTTP_RESPONSE_CODE_FAILURE`. These bail immediately.
|
||
|
||
On final failure, the raised `RuntimeError`'s message includes a `retry_history` block listing every attempt (strategy applied + error seen). Successful invocations return the standard response shape unchanged — no surprise fields when retries didn't fire.
|
||
|
||
### Response
|
||
|
||
```json
|
||
{
|
||
"title": "...",
|
||
"url": "https://example.com/",
|
||
"html": "<!DOCTYPE html>...",
|
||
"screenshot_b64": "<base64 PNG>"
|
||
}
|
||
```
|
||
|
||
## Lambda-specific Chromium hardening (baked in, do not remove)
|
||
|
||
Two flags are forced on every launch by `lambda_handler.py`:
|
||
|
||
- `--disable-dev-shm-usage` — Lambda's `/dev/shm` is ~64 MB; Chromium's renderer crashes mid-paint without this.
|
||
- `--no-zygote` — Lambda's restricted process model can't fork from Chromium's zygote process; without this the browser launches but child renderers fail to spawn and the first `page.new_page()` raises `TargetClosedError`.
|
||
|
||
## Function configuration recommendations
|
||
|
||
Whatever tool you use to create the Lambda function (CLI, CDK, Terraform, SAM, console), apply these settings:
|
||
|
||
| Setting | Value | Why |
|
||
|---|---|---|
|
||
| Package type | Image | Required — this is a container image, not a zip. |
|
||
| Architecture | `arm64` | Roughly 20% cheaper than x86_64. Native build on Apple Silicon. Match the architecture you built for. |
|
||
| Memory | 3008 MB | Memory in Lambda is tied to vCPU. Below ~1769 MB Chromium starts noticeably slower. |
|
||
| Timeout | 120–180 s | Single-attempt scrapes complete in 3–15 s warm; under retry, a `Timeout`-class first failure (30 s default) plus a longer-budget retry (90 s) plus cleanup can total ~120-130 s. 180 s leaves headroom; below 120 s the function will time out before the retry completes. Cold-start init adds 5-10 s on top. |
|
||
| Ephemeral storage (`/tmp`) | 1024 MB | Chromium profile dirs and screenshots can fill the 512 MB default. |
|
||
| Networking | Default (no VPC) | Binary is baked in, no network needed at cold start. Add VPC + NAT only if your proxy egress requires it. |
|
||
| Execution role | `AWSLambdaBasicExecutionRole` | Just CloudWatch Logs. Add more permissions only if your handler needs them. |
|
||
|
||
## Cold start
|
||
|
||
First invocation in a new container takes ~80–90 s (image extraction, Chromium binary mmap, JS engine warmup, no DNS/TLS caches). Subsequent warm invocations on the same container are 3–15 s.
|
||
|
||
For latency-sensitive use cases: provision concurrency, schedule a CloudWatch/EventBridge warmer ping, or accept the cold tail.
|
||
|
||
If you see empty/missing dynamic content on cold-start invocations, raise `max_settle_ms` in the event payload (e.g. `25000`) — the default `15000` is tuned for warm runs.
|
||
|
||
## License
|
||
|
||
The patched Chromium binary inside the upstream `cloakhq/cloakbrowser` image is governed by the **CloakBrowser Binary License** (published at https://github.com/CloakHQ/CloakBrowser/blob/main/BINARY-LICENSE.md). Internal organizational use (private ECR, your own scraping pipelines, your own business) is free. Exposing this Lambda as a paid API to third-party customers — i.e. browser-as-a-service — requires an OEM/SaaS license from CloakHQ (`cloakhq@pm.me`). Do not push the resulting image to a public registry; that would be redistribution and is prohibited.
|