trawl/apps/docs/deployment/troubleshooting.md

title, description

title	description
Troubleshooting	Common issues and how to fix them.

Troubleshooting

API never becomes ready

Symptom: docker compose logs api shows browser launches but the API never prints ready — all N browsers warm.

Causes:

1. Redis not reachable — Check REDIS_URL. From inside Docker, use redis://redis:6379 not redis://localhost:6379.
2. Camoufox binary not installed — The API Dockerfile runs bunx camoufox-js fetch. If this step was skipped (e.g. build cache reuse), rebuild: docker compose build --no-cache api.
3. shm_size too small — Ensure shm_size: 1gb is set on the API service.

Startup timeout behind Gluetun

Symptom: TRAWL logs browser launch exceeded ... during startup and never becomes healthy.

Camoufox performs outbound work, including GeoIP lookup, while launching. If TRAWL starts before Gluetun has established its VPN connection, that work can hit the browser launch timeout. Set depends_on.gluetun.condition: service_healthy as shown in Docker Compose → Route TRAWL through Gluetun.

This differs from the acquire timeout (BROWSER_ACQUIRE_TIMEOUT_MS): a launch timeout means a browser could not start, often because outbound networking or a GeoIP endpoint was unavailable; an acquire timeout means the browsers started but all pool capacity remained busy, and /v1 returns HTTP 429.

Requests through the VPN return HTTP 403

If TRAWL starts successfully but a target returns HTTP 403, the VPN is already routing traffic and the target is likely rejecting the VPN exit IP. Waiting longer for startup will not fix that. Try a different Gluetun server or exit region, or configure an appropriate application proxy.

Gluetun and PROXY_URL operate at different layers: network_mode: service:gluetun routes all container traffic through the VPN, while the optional PROXY_URL is used explicitly by TRAWL's proxy escalation tier. Setting PROXY_URL is not necessary for Gluetun routing, and setting it means that proxied requests still reach that proxy through Gluetun's network namespace.

Container crash-loops with EISDIR or Cannot find module errors

Symptom: The container restarts continuously, logging one of:

error: EISDIR reading "/app/packages/browser/node_modules/camoufox-js"

error: Cannot find module '@sinclair/typebox' from '/app/apps/api/node_modules/elysia/dist/index.mjs'

or, on older images, error: Cannot find package 'memoirist' from '/app/apps/api/node_modules/elysia/dist/index.mjs'.

Cause: A bug in Bun's default "isolated" install linker corrupted node_modules during the Docker build, leaving transitive dependencies (camoufox-js, @sinclair/typebox) missing or broken inside the image (oven-sh/bun#23524, oven-sh/bun#29489).

::: tip Already fixed — just re-pull This was fixed by switching the image build to bun install --linker=hoisted. Every :latest and :baseline image published after the fix is unaffected. If you're hitting this, re-pull rather than patching your container:

docker pull ghcr.io/germondai/trawl:latest   # or :baseline
docker compose up -d --force-recreate

:::

All requests return Tier 3 (never hitting cache)

Symptom: every request takes 10–30s, even for the same domain.

Causes:

1. Redis session data is not persisting — Run docker compose exec redis redis-cli keys "session:*" after a successful scrape. If empty, the session cache write is failing. Check API logs for Redis connection errors.
2. SESSION_TTL_SECONDS set too low — If it's shorter than Cloudflare's challenge interval, the cache expires before the next request.
3. Domain key mismatch — The key is the hostname only. sub.example.com and www.example.com are separate sessions.

POST /v1 returns HTTP 429 with status: "error"

Symptom: Request returns HTTP 429 (not 500) with a FlareSolverr v2 envelope and message: "Browser pool saturated, retry shortly".

Cause: TRAWL polled for BROWSER_ACQUIRE_TIMEOUT_MS (default 15s) without finding an idle browser. With BROWSER_POOL_SIZE=3 and 10 concurrent requests, this only fires under sustained burst pressure.

Fixes (in order of preference):

1. Raise BROWSER_ACQUIRE_TIMEOUT_MS if your upstream target legitimately takes >5s per scrape — bumps the queue wait before 429 fires.
2. Raise BROWSER_POOL_SIZE if you're consistently saturating — each browser uses ~350–500 MB RAM.
3. Reduce incoming request rate if you control the client (Prowlarr's indexer interval, etc.).

POST /v1 returns status: "error" with message "timeout"

Symptom: maxTimeout exceeded (per-request timeout set by the client).

Causes:

1. Cloudflare introduced a harder challenge — Some sites use Turnstile or WAF rules that are harder to bypass. Check the API logs for the actual error.
2. Pool exhausted — All browsers are busy. Increase BROWSER_POOL_SIZE.
3. Proxy not working — If PROXY_URL is configured and invalid, Tier 3 will fail consistently. Test the proxy directly: curl --proxy $PROXY_URL https://nowsecure.nl.

Prowlarr FlareSolverr test fails

Symptom: Green test in isolation but Prowlarr reports the FlareSolverr test as failed.

Check:

1. The URL in Prowlarr includes no trailing slash: http://trawl:8191
2. Prowlarr can reach the TRAWL container. If they're in different Docker networks, add TRAWL to Prowlarr's network (see Prowlarr docs).
3. Run docker exec prowlarr curl -s http://trawl:8191/health to verify network reachability from inside the Prowlarr container.

High memory usage / OOM kills

Each Camoufox instance uses 350–500 MB. With 3 browsers, expect ~1.5 GB total. If the API is being killed:

1. Reduce BROWSER_POOL_SIZE to 1 or 2
2. Upgrade the server (more RAM or more cores)
3. Ensure shm_size: 1gb is set — Firefox uses /dev/shm heavily

Debugging tips

# Live API logs
docker compose logs -f api

# Check Redis keys
docker compose exec redis redis-cli keys "*"

# Test scrape endpoint
curl -s -X POST http://localhost:8191/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' | jq '{tier, totalMs}'

# Verify FlareSolverr compat response shape
curl -s -X POST http://localhost:8191/v1 \
  -H "Content-Type: application/json" \
  -d '{"cmd":"request.get","url":"https://nowsecure.nl"}' | jq '{status, version}'