4.1 KiB
title, description
| title | description |
|---|---|
| Docker Compose | Run TRAWL with Docker Compose — scraper only or full stack. |
Docker Compose
Four compose files live in the repo root, matching the setups shown on the landing page.
Scraper only
Minimal
docker-compose.minimal.yml — single service, no Redis. Fastest to get started, no session caching.
docker compose -f docker-compose.minimal.yml up -d
Cached (default)
docker-compose.yml — scraper + Redis session cache. Repeat requests to the same domain return in ~500ms.
docker compose up -d
Production
docker-compose.prod.yml — same as cached but with restart: always, a memory limit, and a healthcheck.
docker compose -f docker-compose.prod.yml up -d
trawl:
restart: always
mem_limit: 3g
environment:
BROWSER_POOL_SIZE: 5
healthcheck:
test: wget -qO- http://localhost:8191/health
interval: 30s
To update to the latest image:
docker compose pull && docker compose up -d
Baseline (older CPUs / Synology NAS)
If your CPU doesn't support AVX2 — older Synology NAS units, Atom/Celeron-era hardware — override the image tag to :baseline in any compose file above. Nothing else changes:
services:
trawl:
image: ghcr.io/germondai/trawl:baseline
# ...rest of the service definition unchanged
::: tip See Standalone Containers → Older CPUs & Synology NAS for how to tell if you need this, and the README for the full tag comparison. :::
Full stack
docker-compose.full.yml adds the landing page and docs on top of the scraper. Web and docs are built from source.
docker compose -f docker-compose.full.yml up -d
| Service | URL | Description |
|---|---|---|
trawl |
localhost:8191 |
Scraper API |
web |
localhost:3000 |
Landing page |
docs |
localhost:3001 |
Documentation |
redis |
internal | Session cache |
First run builds the web and docs images locally — takes a couple of minutes. Subsequent runs are fast (layers cached).
Environment variables
| Variable | Default | Description |
|---|---|---|
BROWSER_POOL_SIZE |
3 |
Warm browser instances |
BROWSER_ACQUIRE_TIMEOUT_MS |
15000 |
How long acquire() polls for a free browser before returning HTTP 429 |
BROWSER_RECYCLE_AFTER_CONTEXTS |
8 |
Restart a browser after this many fresh/proxy contexts; set 0 to disable |
REDIS_URL |
redis://redis:6379 |
Redis connection (set automatically in compose) |
RESIDENTIAL_PROXY_URL |
— | Enables Tier 4 proxy escalation |
Logs
docker compose logs -f trawl
docker compose logs -f redis
Memory guide
BROWSER_POOL_SIZE |
Approx. RAM | Recommended host RAM |
|---|---|---|
| 1 | ~500 MB | 1 GB |
| 3 | ~1.2 GB | 2 GB |
| 5 | ~2 GB | 3 GB |
| 10 | ~4 GB | 6 GB |
Each Camoufox Firefox instance uses ~350–500 MB under load.
Reverse proxy
To expose TRAWL over HTTPS, proxy port 8191. Set proxy_read_timeout longer than your maxTimeout — challenge solves can take up to 15s.
server {
listen 443 ssl;
server_name trawl.yourdomain.com;
location / {
proxy_pass http://localhost:8191;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_read_timeout 120s;
}
}