5.0 KiB
title, description
| title | description |
|---|---|
| Troubleshooting | Common issues and how to fix them. |
Troubleshooting
API never becomes ready
Symptom: docker compose logs api shows browser launches but the API never prints ready — all N browsers warm.
Causes:
- Redis not reachable — Check
REDIS_URL. From inside Docker, useredis://redis:6379notredis://localhost:6379. - Camoufox binary not installed — The API Dockerfile runs
bunx camoufox-js fetch. If this step was skipped (e.g. build cache reuse), rebuild:docker compose build --no-cache api. - shm_size too small — Ensure
shm_size: 1gbis set on the API service.
Container crash-loops with EISDIR or Cannot find module errors
Symptom: The container restarts continuously, logging one of:
error: EISDIR reading "/app/packages/browser/node_modules/camoufox-js"
error: Cannot find module '@sinclair/typebox' from '/app/apps/api/node_modules/elysia/dist/index.mjs'
or, on older images, error: Cannot find package 'memoirist' from '/app/apps/api/node_modules/elysia/dist/index.mjs'.
Cause: A bug in Bun's default "isolated" install linker corrupted node_modules during the Docker build, leaving transitive dependencies (camoufox-js, @sinclair/typebox) missing or broken inside the image (oven-sh/bun#23524, oven-sh/bun#29489).
::: tip Already fixed — just re-pull
This was fixed by switching the image build to bun install --linker=hoisted. Every :latest and :baseline image published after the fix is unaffected. If you're hitting this, re-pull rather than patching your container:
docker pull ghcr.io/germondai/trawl:latest # or :baseline
docker compose up -d --force-recreate
:::
All requests return Tier 3 (never hitting cache)
Symptom: every request takes 10–30s, even for the same domain.
Causes:
- Redis session data is not persisting — Run
docker exec trawl-redis redis-cli keys "session:*"after a successful scrape. If empty, the session cache write is failing. Check API logs for Redis errors. SESSION_TTL_SECONDSset too low — If it's shorter than Cloudflare's challenge interval, the cache expires before the next request.- Domain key mismatch — The key is the hostname only.
sub.example.comandwww.example.comare separate sessions.
POST /v1 returns HTTP 429 with status: "error"
Symptom: Request returns HTTP 429 (not 500) with a FlareSolverr v2 envelope and message: "Browser pool saturated, retry shortly".
Cause: TRAWL polled for BROWSER_ACQUIRE_TIMEOUT_MS (default 15s) without finding an idle browser. With BROWSER_POOL_SIZE=3 and 10 concurrent requests, this only fires under sustained burst pressure.
Fixes (in order of preference):
- Raise
BROWSER_ACQUIRE_TIMEOUT_MSif your upstream target legitimately takes >5s per scrape — bumps the queue wait before 429 fires. - Raise
BROWSER_POOL_SIZEif you're consistently saturating — each browser uses ~350–500 MB RAM. - Reduce incoming request rate if you control the client (Prowlarr's indexer interval, etc.).
POST /v1 returns status: "error" with message "timeout"
Symptom: maxTimeout exceeded (per-request timeout set by the client).
Causes:
- Cloudflare introduced a harder challenge — Some sites use Turnstile or WAF rules that are harder to bypass. Check the API logs for the actual error.
- Pool exhausted — All browsers are busy. Increase
BROWSER_POOL_SIZE. - Proxy not working — If
PROXY_URLis configured and invalid, Tier 3 will fail consistently. Test the proxy directly:curl --proxy $PROXY_URL https://nowsecure.nl.
Prowlarr FlareSolverr test fails
Symptom: Green test in isolation but Prowlarr reports the FlareSolverr test as failed.
Check:
- The URL in Prowlarr includes no trailing slash:
http://trawl:8191 - Prowlarr can reach the TRAWL container. If they're in different Docker networks, add TRAWL to Prowlarr's network (see Prowlarr docs).
- Run
docker exec prowlarr curl -s http://trawl:8191/healthto verify network reachability from inside the Prowlarr container.
High memory usage / OOM kills
Each Camoufox instance uses 350–500 MB. With 3 browsers, expect ~1.5 GB total. If the API is being killed:
- Reduce
BROWSER_POOL_SIZEto 1 or 2 - Upgrade the server (more RAM or more cores)
- Ensure
shm_size: 1gbis set — Firefox uses/dev/shmheavily
Debugging tips
# Live API logs
docker compose logs -f api
# Check Redis keys
docker exec trawl-redis redis-cli keys "*"
# Test scrape endpoint
curl -s -X POST http://localhost:8191/scrape \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}' | jq '{tier, totalMs}'
# Verify FlareSolverr compat response shape
curl -s -X POST http://localhost:8191/v1 \
-H "Content-Type: application/json" \
-d '{"cmd":"request.get","url":"https://nowsecure.nl"}' | jq '{status, version}'