trawl/apps/docs/getting-started/configuration.md

title, description

title	description
Configuration	All environment variables for TRAWL, with defaults and examples.

Configuration

All configuration is via environment variables. Copy .env.example to .env and edit before starting.

Redis

REDIS_URL

Default: redis://localhost:6379

Standard Redis connection URL. When running inside Docker Compose use the service name:

REDIS_URL=redis://redis:6379

With authentication:

REDIS_URL=redis://:yourpassword@redis:6379

With a specific database index:

REDIS_URL=redis://redis:6379/1

Browser Pool

BROWSER_POOL_SIZE

Default: 3

Number of Camoufox Firefox instances to keep warm. Each instance uses ~350–500 MB RAM under load. Start conservative and raise if you need higher concurrency.

BROWSER_POOL_SIZE=1   # minimal (1 GB host RAM)
BROWSER_POOL_SIZE=3   # default — good for most self-hosted setups
BROWSER_POOL_SIZE=8   # high-throughput (6+ GB host RAM)

Note: The API container sets shm_size: 1gb by default. If you raise BROWSER_POOL_SIZE above 5, also raise shm_size in your docker-compose.yml to at least 2gb.

BROWSER_ACQUIRE_TIMEOUT_MS

Default: 15000 (15 seconds)

How long BrowserPool.acquire() will poll for a free browser before rejecting with PoolExhaustedError. With BROWSER_POOL_SIZE=3 and a typical Cloudflare challenge taking 5–8s per request, the 15s default lets a full burst of 10 concurrent requests drain without any 429s.

Lower it for fail-fast client feedback (Prowlarr will see 429s sooner and retry on its own). Raise it for very heavy upstream targets or when you've bumped BROWSER_POOL_SIZE higher.

BROWSER_ACQUIRE_TIMEOUT_MS=5000    # fail fast — 429s after 5s
BROWSER_ACQUIRE_TIMEOUT_MS=15000   # default — absorbs a full burst on pool=3
BROWSER_ACQUIRE_TIMEOUT_MS=30000   # tolerate longer queueing on slow targets

When the timeout fires, both /v1 and /scrape return HTTP 429 with the FlareSolverr v2 error envelope (not a 500).

BROWSER_RECYCLE_AFTER_CONTEXTS

Default: 8

How many blocked / needs-js outcomes a pooled browser can produce before TRAWL restarts the full browser process. The recycle counter only increments when Tier 3 or Tier 4 reports the upstream actively rejected the browser's profile — successful solves preserve cookies, cf_clearance, and warm fingerprint state. This avoids the HTTP-429 storm that occurred when the pool preemptively recycled mid-flight (issue #17).

BROWSER_RECYCLE_AFTER_CONTEXTS=8   # default - recycle after 8 blocked/needs-js outcomes
BROWSER_RECYCLE_AFTER_CONTEXTS=0   # disable browser recycling entirely

BROWSER_CONTENT_PROCESSES

Default: 2

Caps Firefox content processes per pooled browser via the dom.ipc.processCount Firefox pref. Firefox's default of 8 lets thread count climb when Tier 3 / Tier 4 churn disposable contexts (see #13). The cap bounds the leak at the source without paying the recycle cost. Raise if specific targets fail with empty content (rare).

BROWSER_CONTENT_PROCESSES=2   # default - conservative cap, lowest RAM/CPU
BROWSER_CONTENT_PROCESSES=4   # raise if CF/Imperva challenges stall

Session Cache

SESSION_TTL_SECONDS

Default: 3600 (1 hour)

How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.

Cloudflare's cf_clearance cookie typically has a 30-minute expiry. Setting SESSION_TTL_SECONDS below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3).

SESSION_TTL_SECONDS=3600   # default — safe for most sites
SESSION_TTL_SECONDS=1800   # more conservative

Proxies

PROXY_URL

Default: (empty — no proxy)

Datacenter proxy pool used for Tier 3 (fresh challenge solve). Format: protocol://user:pass@host:port, or a comma-separated list for multiple proxies:

PROXY_URL=http://user:pass@dc-proxy.example.com:8080
PROXY_URL=http://user:pass@dc1.example.com:8080,http://user:pass@dc2.example.com:8080

Leave empty to run Tier 3 without a proxy (your server's real IP is used).

RESIDENTIAL_PROXY_URL

Default: (empty — Tier 4 disabled)

Residential proxy pool used for Tier 4 (when the datacenter IP is flagged). Same format as PROXY_URL — single URL or comma-separated list. Tier 4 is completely skipped if this variable is not set and no per-request proxy override is supplied.

RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080

PROXY_LIST_FILE / RESIDENTIAL_PROXY_LIST_FILE

Default: (empty)

Alternative to cramming a large proxy list into PROXY_URL/RESIDENTIAL_PROXY_URL — path to a file with one proxy URL per line (# comments allowed). Merged with the corresponding *_URL env var if both are set.

PROXY_LIST_FILE=/etc/trawl/datacenter-proxies.txt
RESIDENTIAL_PROXY_LIST_FILE=/etc/trawl/residential-proxies.txt

Rotation and failure handling

When more than one proxy is configured, TRAWL picks proxies sticky-per-domain — repeat requests to the same hostname keep reusing the same proxy (helps avoid re-triggering challenges), while different domains spread round-robin across the pool. If a tier attempt comes back "blocked" using a pool-sourced proxy, that proxy is put in a 5-minute cooldown and the request retries once with the next available proxy before falling through (Tier 3 → Tier 4, or Tier 4 failing outright) — bounded to 2 attempts per tier so a long list can't blow the request's maxTimeout.

Per-request override

Both POST /scrape and POST /v1 accept an optional proxy field in the request body — when present, it's used directly for that request's Tier 3/4 attempts instead of the configured pool (and isn't retried against other pool proxies on failure, since it's caller-supplied):

{ "url": "https://example.com", "proxy": "http://user:pass@my-proxy.example.com:8080" }

Note: proxy on /v1 is a TRAWL-specific extension — it is not part of the real FlareSolverr v2 contract, so other FlareSolverr-compatible clients simply won't send it.

Ports

PORT

Default: 8191

Host port the Docker port mapping forwards to TRAWL's internal listener. Defaults to 8191 — the same port FlareSolverr and Byparr use, so you can swap TRAWL in without changing any *arr app settings. The container itself always listens on 8191 internally; PORT only changes the host-side port (e.g. "${PORT:-8191}:8191" in every compose file).

To run TRAWL alongside FlareSolverr (or any other service that already binds 8191 on the host), set PORT in your shell or .env to any free port before running docker compose up:

PORT=9191 docker compose up -d
# TRAWL reachable at http://localhost:9191, while port 8191 stays free for FlareSolverr.

PORT_WEB

Default: 3000

Port the Nuxt landing page listens on.

---

Full .env.example

# ── Redis ─────────────────────────────────────
REDIS_URL=redis://localhost:6379

# ── Browser pool ──────────────────────────────
BROWSER_POOL_SIZE=3
BROWSER_ACQUIRE_TIMEOUT_MS=15000
SESSION_TTL_SECONDS=3600

# ── Proxies (optional, comma-separated lists) ─
PROXY_URL=
RESIDENTIAL_PROXY_URL=
PROXY_LIST_FILE=
RESIDENTIAL_PROXY_LIST_FILE=

# ── Ports ─────────────────────────────────────
PORT=8191
PORT_WEB=3000