diff --git a/.env.example b/.env.example index 1d7c112..ccbcdab 100644 --- a/.env.example +++ b/.env.example @@ -37,6 +37,9 @@ PROXY_URL= # Optional file with one proxy URL per line. PROXY_LIST_FILE= # Tier 4 residential proxy. Uses the same URL and pool formats. +# Examples: +# RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080 +# RESIDENTIAL_PROXY_URL=socks5://residential.example.com:1080 RESIDENTIAL_PROXY_URL= # Optional file with one residential proxy URL per line. RESIDENTIAL_PROXY_LIST_FILE= diff --git a/CHANGELOG.md b/CHANGELOG.md index 22c8ea9..15bd312 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - Detect and resolve Akamai Bot Manager behavioral interstitials across scraper tiers and the HTTP/HTTPS proxy (#33). +- Document supported upstream proxy formats, local Compose configuration, and residential proxy pools (#26). ### Fixed - Reject non-object request bodies and missing, non-string, or blank `url` values with HTTP 400 before scraper-tier execution (#34). diff --git a/README.md b/README.md index e0c1824..0bbb490 100644 --- a/README.md +++ b/README.md @@ -347,6 +347,26 @@ git push origin v1.0.1 ## Configuration +TRAWL supports HTTP proxies, authenticated HTTP proxies, and SOCKS5 proxies. The standard Compose +files read proxy settings from the local `.env` file: + +```ini +# Optional Tier 3 datacenter proxy +PROXY_URL=http://user:pass@datacenter.example.com:8080 + +# Optional Tier 4 residential proxy +RESIDENTIAL_PROXY_URL=socks5://user:pass@residential.example.com:1080 +``` + +```bash +docker compose up -d +``` + +Leave either value empty to disable that proxy tier. Multiple endpoints can be separated with +commas; larger pools can use the corresponding `*_LIST_FILE` variable. See +[Configuration → Proxies](./apps/docs/getting-started/configuration.md#proxies) +for pool and mounted-file examples. + | Variable | Default | Description | | -------------------------------- | ------------------------ | ----------------------------------------------------------------------------------- | | `BROWSER_POOL_SIZE` | `3` | Warm Camoufox Firefox instances | @@ -355,7 +375,10 @@ git push origin v1.0.1 | `BROWSER_CONTENT_PROCESSES` | `2` | Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU | | `SESSION_TTL_SECONDS` | `3600` | Redis session cache TTL (seconds) | | `REDIS_URL` | `redis://localhost:6379` | Redis connection string | +| `PROXY_URL` | — | Optional Tier 3 HTTP or SOCKS5 proxy, or comma-separated pool | +| `PROXY_LIST_FILE` | — | File containing one Tier 3 proxy URL per line | | `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation | +| `RESIDENTIAL_PROXY_LIST_FILE` | — | File containing one Tier 4 proxy URL per line | | `STT_URL` | — | Local Whisper endpoint for reCAPTCHA (optional) | | `PORT` | `8191` | API listen port | | `MITM_PROXY_ENABLED` | `false` | Enable the challenge-bypassing HTTP/HTTPS proxy |