fix(compose): pass upstream proxy configuration

2026-08-17 12:11:23 +02:00 · 2026-07-26 16:39:52 +02:00
parent f08f5a2b44
commit f397815fe7
6 changed files with 91 additions and 3 deletions
@@ -73,6 +73,7 @@ See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone
 | `BROWSER_ACQUIRE_TIMEOUT_MS`     | `15000`              | How long `acquire()` polls for a free browser before returning HTTP 429 |
 | `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8`                  | Restart after this many blocked/needs-js outcomes; set `0` to disable   |
 | `REDIS_URL`                      | `redis://redis:6379` | Redis connection (set automatically in compose)                         |
+| `PROXY_URL`                      | —                    | Optional Tier 3 datacenter proxy or pool                                |
 | `RESIDENTIAL_PROXY_URL`          | —                    | Enables Tier 4 proxy escalation                                         |
 | `MITM_PROXY_ENABLED`             | `false`              | Starts the general HTTP/HTTPS proxy                                     |
 | `MITM_PROXY_PORT`                | `8192`               | Proxy listen and published port                                         |
@@ -82,6 +83,18 @@ See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone
 All supplied Compose files publish port `8192` and mount the `trawl_proxy_ca` volume. The listener
 does not start until `MITM_PROXY_ENABLED=true`. See [Proxy Configuration](/proxy/configuration).

+All supplied Compose files also pass `PROXY_URL`, `PROXY_LIST_FILE`, `RESIDENTIAL_PROXY_URL`, and
+`RESIDENTIAL_PROXY_LIST_FILE` from the local environment or `.env` file. For a single residential
+endpoint:
+
+```ini
+# .env
+RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080
+```
+
+For supported endpoint formats, pools, and mounted list files, see
+[Configuration → Proxies](/getting-started/configuration#proxies).
+
 ## Logs

 ```bash
@@ -106,10 +106,18 @@ SESSION_TTL_SECONDS=1800   # more conservative

 **Default:** _(empty — no proxy)_

-Datacenter proxy pool used for Tier 3 (fresh challenge solve). Format: `protocol://user:pass@host:port`, or a **comma-separated list** for multiple proxies:
+Datacenter proxy pool used for Tier 3 (fresh challenge solve). TRAWL passes these endpoints to the
+browser and supports HTTP and SOCKS5 forms:

 ```ini
+PROXY_URL=http://dc-proxy.example.com:8080
 PROXY_URL=http://user:pass@dc-proxy.example.com:8080
+PROXY_URL=socks5://dc-proxy.example.com:1080
+```
+
+HTTP credentials can be embedded in the URL. For multiple endpoints, use a comma-separated list:
+
+```ini
 PROXY_URL=http://user:pass@dc1.example.com:8080,http://user:pass@dc2.example.com:8080
 ```

@@ -123,8 +131,12 @@ Residential proxy pool used for Tier 4 (when the datacenter IP is flagged). Same

 ```ini
 RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080
+RESIDENTIAL_PROXY_URL=socks5://residential.example.com:1080
 ```

+Provider labels such as "rotating", "sticky", "country", or "session" do not change the TRAWL
+format. Use the hostname, port, and credentials supplied by the provider.
+
 ### `PROXY_LIST_FILE` / `RESIDENTIAL_PROXY_LIST_FILE`

 **Default:** _(empty)_
@@ -136,6 +148,35 @@ PROXY_LIST_FILE=/etc/trawl/datacenter-proxies.txt
 RESIDENTIAL_PROXY_LIST_FILE=/etc/trawl/residential-proxies.txt
 ```

+Example file:
+
+```text
+# /etc/trawl/residential-proxies.txt
+http://user:pass@residential-1.example.com:8080
+http://user:pass@residential-2.example.com:8080
+socks5://residential-3.example.com:1080
+```
+
+When using Docker, the path is inside the TRAWL container. Mount the file and pass the same
+in-container path:
+
+```yaml
+services:
+  trawl:
+    environment:
+      RESIDENTIAL_PROXY_LIST_FILE: /etc/trawl/residential-proxies.txt
+    volumes:
+      - ./residential-proxies.txt:/etc/trawl/residential-proxies.txt:ro
+```
+
+For a single endpoint or a short pool, a local `.env` beside `docker-compose.yml` is enough:
+
+```ini
+RESIDENTIAL_PROXY_URL=http://user:pass@residential.example.com:8080
+```
+
+The supplied Compose files pass all four proxy variables into the container.
+
 ### Rotation and failure handling

 When more than one proxy is configured, TRAWL picks proxies **sticky-per-domain** — repeat requests to the same hostname keep reusing the same proxy (helps avoid re-triggering challenges), while different domains spread round-robin across the pool. If a tier attempt comes back `"blocked"` using a pool-sourced proxy, that proxy is put in a 5-minute cooldown and the request retries once with the next available proxy before falling through (Tier 3 → Tier 4, or Tier 4 failing outright) — bounded to 2 attempts per tier so a long list can't blow the request's `maxTimeout`.
@@ -150,6 +191,23 @@ Both `POST /scrape` and `POST /v1` accept an optional `proxy` field in the reque

 Note: `proxy` on `/v1` is a TRAWL-specific extension — it is not part of the real FlareSolverr v2 contract, so other FlareSolverr-compatible clients simply won't send it.

+### Test an endpoint
+
+Test an HTTP proxy independently before starting TRAWL:
+
+```bash
+curl --proxy http://user:pass@proxy.example.com:8080 https://api.ipify.org
+```
+
+For SOCKS5 with proxy-side DNS resolution:
+
+```bash
+curl --proxy socks5h://proxy.example.com:1080 https://api.ipify.org
+```
+
+Use the provider's exact endpoint and authentication details. A working `curl` test confirms
+connectivity, but the destination can still reject that proxy IP during a browser challenge.
+
 ## Ports

 ### `PORT`