diff --git a/apps/docs/deployment/docker-compose.md b/apps/docs/deployment/docker-compose.md index 7da8e17..b67eeed 100644 --- a/apps/docs/deployment/docker-compose.md +++ b/apps/docs/deployment/docker-compose.md @@ -96,6 +96,35 @@ RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080 For supported endpoint formats, pools, and mounted list files, see [Configuration → Proxies](/getting-started/configuration#proxies). +## Route TRAWL through Gluetun + +To route all TRAWL traffic through a Gluetun VPN, share Gluetun's network namespace and wait for +its healthcheck before starting TRAWL. Because TRAWL no longer has its own network namespace, +publish both TRAWL ports on the `gluetun` service: + +```yaml +services: + gluetun: + image: qmcgaw/gluetun + # Configure your VPN provider and credentials here. + ports: + - "8191:8191" + - "8192:8192" + + trawl: + image: ghcr.io/germondai/trawl:latest + network_mode: service:gluetun + shm_size: 1gb + depends_on: + gluetun: + condition: service_healthy +``` + +Do not also publish `8191` or `8192` on `trawl`; Compose does not allow port publishing with +`network_mode: service:gluetun`. Gluetun routing applies to all outbound container traffic. +`PROXY_URL`, by contrast, is optional application-level proxy configuration used by TRAWL's +escalation tiers; it is not required to send the container through the VPN. + ## Logs ```bash diff --git a/apps/docs/deployment/troubleshooting.md b/apps/docs/deployment/troubleshooting.md index 895705b..09f543f 100644 --- a/apps/docs/deployment/troubleshooting.md +++ b/apps/docs/deployment/troubleshooting.md @@ -15,6 +15,31 @@ description: Common issues and how to fix them. 2. **Camoufox binary not installed** — The API Dockerfile runs `bunx camoufox-js fetch`. If this step was skipped (e.g. build cache reuse), rebuild: `docker compose build --no-cache api`. 3. **shm_size too small** — Ensure `shm_size: 1gb` is set on the API service. +### Startup timeout behind Gluetun + +**Symptom:** TRAWL logs `browser launch exceeded ...` during startup and never becomes healthy. + +Camoufox performs outbound work, including GeoIP lookup, while launching. If TRAWL starts before +Gluetun has established its VPN connection, that work can hit the browser **launch timeout**. Set +`depends_on.gluetun.condition: service_healthy` as shown in +[Docker Compose → Route TRAWL through Gluetun](/deployment/docker-compose#route-trawl-through-gluetun). + +This differs from the **acquire timeout** (`BROWSER_ACQUIRE_TIMEOUT_MS`): a launch timeout means a +browser could not start, often because outbound networking or a GeoIP endpoint was unavailable; an +acquire timeout means the browsers started but all pool capacity remained busy, and `/v1` returns +HTTP 429. + +### Requests through the VPN return HTTP 403 + +If TRAWL starts successfully but a target returns HTTP 403, the VPN is already routing traffic and +the target is likely rejecting the VPN exit IP. Waiting longer for startup will not fix that. Try a +different Gluetun server or exit region, or configure an appropriate application proxy. + +Gluetun and `PROXY_URL` operate at different layers: `network_mode: service:gluetun` routes all +container traffic through the VPN, while the optional `PROXY_URL` is used explicitly by TRAWL's +proxy escalation tier. Setting `PROXY_URL` is not necessary for Gluetun routing, and setting it +means that proxied requests still reach that proxy through Gluetun's network namespace. + ## Container crash-loops with `EISDIR` or `Cannot find module` errors **Symptom:** The container restarts continuously, logging one of: diff --git a/packages/browser/src/pool.ts b/packages/browser/src/pool.ts index 2420bdc..010b6ab 100644 --- a/packages/browser/src/pool.ts +++ b/packages/browser/src/pool.ts @@ -477,7 +477,9 @@ export class BrowserPool { ]).finally(() => { if (timer) clearTimeout(timer) }) - if (!result) throw new Error(`browser launch exceeded ${ms}ms`) + if (!result) { + throw new Error(`browser launch exceeded ${ms}ms; check outbound network and GeoIP access`) + } return result } diff --git a/packages/browser/tests/pool.test.ts b/packages/browser/tests/pool.test.ts index 65f7553..6600ffb 100644 --- a/packages/browser/tests/pool.test.ts +++ b/packages/browser/tests/pool.test.ts @@ -63,6 +63,16 @@ function makeFactory() { } describe("BrowserPool recycling", () => { + test("launch timeout diagnoses outbound network and GeoIP availability", async () => { + const pool = createPool({ + poolSize: 1, + launchTimeoutMs: 20, + browserFactory: NEVER, + }) + + await expect(pool.init()).rejects.toThrow("browser launch exceeded 20ms; check outbound network and GeoIP access") + }) + test("restarts the browser after the temporary context threshold", async () => { const { factory, browsers, contexts } = makeFactory()