--- title: Configuration description: All environment variables for TRAWL, with defaults and examples. --- # Configuration All configuration is via environment variables. Copy `.env.example` to `.env` and edit before starting. ## Redis ### `REDIS_URL` **Default:** `redis://localhost:6379` Standard Redis connection URL — TRAWL's cache backend is Redis 8.8. When running inside Docker Compose use the service name: ```ini REDIS_URL=redis://redis:6379 ``` With authentication: ```ini REDIS_URL=redis://:yourpassword@redis:6379 ``` With a specific database index: ```ini REDIS_URL=redis://redis:6379/1 ``` ## Browser Pool ### `BROWSER_POOL_SIZE` **Default:** `3` Number of Camoufox Firefox instances to keep warm. Each instance uses ~350–500 MB RAM under load. Start conservative and raise if you need higher concurrency. ```ini BROWSER_POOL_SIZE=1 # minimal (1 GB host RAM) BROWSER_POOL_SIZE=3 # default — good for most self-hosted setups BROWSER_POOL_SIZE=8 # high-throughput (6+ GB host RAM) ``` > **Note:** The API container sets `shm_size: 1gb` by default. If you raise `BROWSER_POOL_SIZE` above 5, also raise `shm_size` in your `docker-compose.yml` to at least `2gb`. ### `BROWSER_ACQUIRE_TIMEOUT_MS` **Default:** `15000` (15 seconds) How long `BrowserPool.acquire()` will poll for a free browser before rejecting with `PoolExhaustedError`. With `BROWSER_POOL_SIZE=3` and a typical Cloudflare challenge taking 5–8s per request, the 15s default lets a full burst of 10 concurrent requests drain without any 429s. Lower it for fail-fast client feedback (Prowlarr will see 429s sooner and retry on its own). Raise it for very heavy upstream targets or when you've bumped `BROWSER_POOL_SIZE` higher. ```ini BROWSER_ACQUIRE_TIMEOUT_MS=5000 # fail fast — 429s after 5s BROWSER_ACQUIRE_TIMEOUT_MS=15000 # default — absorbs a full burst on pool=3 BROWSER_ACQUIRE_TIMEOUT_MS=30000 # tolerate longer queueing on slow targets ``` When the timeout fires, both `/v1` and `/scrape` return **HTTP 429** with the FlareSolverr v2 error envelope (not a 500). ### `BROWSER_RECYCLE_AFTER_CONTEXTS` **Default:** `8` How many Tier 3 or Tier 4 temporary contexts a pooled browser can create before TRAWL rolling-replaces the full browser process. Every context counts, regardless of whether the attempt succeeds, times out, errors, or is blocked. TRAWL warms one replacement while the existing browser remains available, installs it when the entry is idle, then closes the retired browser. This briefly raises the pool by one browser, and replacements are serialized pool-wide to bound that peak. ```ini BROWSER_RECYCLE_AFTER_CONTEXTS=8 # default - replace after 8 Tier 3/4 contexts BROWSER_RECYCLE_AFTER_CONTEXTS=0 # disable browser recycling entirely ``` ### `BROWSER_CONTENT_PROCESSES` **Default:** `2` Caps Firefox content processes per pooled browser via the `dom.ipc.processCount` Firefox pref. Firefox's default of 8 lets thread count climb when Tier 3 / Tier 4 churn disposable contexts (see #13). The cap bounds the leak at the source without paying the recycle cost. Raise if specific targets fail with empty content (rare). ```ini BROWSER_CONTENT_PROCESSES=2 # default - conservative cap, lowest RAM/CPU BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall ``` ### Browser recovery timeouts | Variable | Default | Purpose | | --- | ---: | --- | | `BROWSER_STALL_TIMEOUT_MS` | `180000` | Grace period after a request's own timeout before its browser checkout is reclaimed | | `BROWSER_CLOSE_TIMEOUT_MS` | `10000` | Maximum wait for a wedged browser or context to close | | `BROWSER_LAUNCH_TIMEOUT_MS` | `90000` | Maximum wait for Camoufox to launch | These bounds keep an unresponsive Firefox process from permanently consuming a pool slot. The defaults are suitable for most installations. ## Session Cache ### `SESSION_TTL_SECONDS` **Default:** `3600` (1 hour) How long solved browser cookies and user-agent state are cached in Redis per domain. After this TTL the next protected request triggers a fresh challenge solve (Tier 3) and refreshes the cache. Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3). ```ini SESSION_TTL_SECONDS=3600 # default — safe for most sites SESSION_TTL_SECONDS=1800 # more conservative ``` ## Proxies ### `PROXY_URL` **Default:** _(empty — no proxy)_ Datacenter proxy pool used for Tier 3 (fresh challenge solve). TRAWL passes these endpoints to the browser and supports HTTP and SOCKS5 forms: ```ini PROXY_URL=http://dc-proxy.example.com:8080 PROXY_URL=http://user:pass@dc-proxy.example.com:8080 PROXY_URL=socks5://dc-proxy.example.com:1080 ``` HTTP credentials can be embedded in the URL. For multiple endpoints, use a comma-separated list: ```ini PROXY_URL=http://user:pass@dc1.example.com:8080,http://user:pass@dc2.example.com:8080 ``` Leave empty to run Tier 3 without a proxy (your server's real IP is used). ### `RESIDENTIAL_PROXY_URL` **Default:** _(empty — Tier 4 disabled)_ Residential proxy pool used for Tier 4 (when the datacenter IP is flagged). Same format as `PROXY_URL` — single URL or comma-separated list. Tier 4 is completely skipped if this variable is not set and no per-request `proxy` override is supplied. ```ini RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080 RESIDENTIAL_PROXY_URL=socks5://residential.example.com:1080 ``` Provider labels such as "rotating", "sticky", "country", or "session" do not change the TRAWL format. Use the hostname, port, and credentials supplied by the provider. ### `PROXY_LIST_FILE` / `RESIDENTIAL_PROXY_LIST_FILE` **Default:** _(empty)_ Alternative to cramming a large proxy list into `PROXY_URL`/`RESIDENTIAL_PROXY_URL` — path to a file with one proxy URL per line (`#` comments allowed). Merged with the corresponding `*_URL` env var if both are set. ```ini PROXY_LIST_FILE=/etc/trawl/datacenter-proxies.txt RESIDENTIAL_PROXY_LIST_FILE=/etc/trawl/residential-proxies.txt ``` Example file: ```text # /etc/trawl/residential-proxies.txt http://user:pass@residential-1.example.com:8080 http://user:pass@residential-2.example.com:8080 socks5://residential-3.example.com:1080 ``` When using Docker, the path is inside the TRAWL container. Mount the file and pass the same in-container path: ```yaml services: trawl: environment: RESIDENTIAL_PROXY_LIST_FILE: /etc/trawl/residential-proxies.txt volumes: - ./residential-proxies.txt:/etc/trawl/residential-proxies.txt:ro ``` For a single endpoint or a short pool, a local `.env` beside `docker-compose.yml` is enough: ```ini RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080 ``` The supplied Compose files pass all four proxy variables into the container. ### Rotation and failure handling When more than one proxy is configured, TRAWL picks proxies **sticky-per-domain** — repeat requests to the same hostname keep reusing the same proxy (helps avoid re-triggering challenges), while different domains spread round-robin across the pool. If a tier attempt comes back `"blocked"` using a pool-sourced proxy, that proxy is put in a 5-minute cooldown and the request retries once with the next available proxy before falling through (Tier 3 → Tier 4, or Tier 4 failing outright) — bounded to 2 attempts per tier so a long list can't blow the request's `maxTimeout`. ### Per-request override Both `POST /scrape` and `POST /v1` accept an optional `proxy` field in the request body — when present, it's used directly for that request's Tier 3/4 attempts instead of the configured pool (and isn't retried against other pool proxies on failure, since it's caller-supplied): ```json { "url": "https://example.com", "proxy": "http://user:pass@my-proxy.example.com:8080" } ``` Note: `proxy` on `/v1` is a TRAWL-specific extension — it is not part of the real FlareSolverr v2 contract, so other FlareSolverr-compatible clients simply won't send it. ### Test an endpoint Test an HTTP proxy independently before starting TRAWL: ```bash curl --proxy http://user:pass@proxy.example.com:8080 https://api.ipify.org ``` For SOCKS5 with proxy-side DNS resolution: ```bash curl --proxy socks5h://proxy.example.com:1080 https://api.ipify.org ``` Use the provider's exact endpoint and authentication details. A working `curl` test confirms connectivity, but the destination can still reject that proxy IP during a browser challenge. ## Ports ### `PORT` **Default:** `8191` API listener port. It defaults to `8191`, the same port used by FlareSolverr and Byparr. The supplied Compose files use `${PORT:-8191}` for the host side of the `8191` container mapping. To run TRAWL alongside FlareSolverr (or any other service that already binds `8191` on the host), set `PORT` in your shell or `.env` to any free port **before** running `docker compose up`: ```bash PORT=9191 docker compose up -d # TRAWL reachable at http://localhost:9191, while port 8191 stays free for FlareSolverr. ``` ## Forward proxy The optional general HTTP/HTTPS proxy has its own listener, CA, tier cap, and debug settings. See [Proxy Configuration](/proxy/configuration) for all `MITM_PROXY_*` variables and deployment examples. --- The repository's [`.env.example`](https://github.com/germondai/trawl/blob/main/.env.example) is the canonical copyable environment template.