mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
docs(proxy): add setup and operation guides
This commit is contained in:
germondai
@@ -53,6 +53,15 @@ export default defineConfig({
|
||||
{ text: "*arr Apps", link: "/integrations/arr-apps" },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: "Forward Proxy",
|
||||
items: [
|
||||
{ text: "Overview & Traffic Flow", link: "/proxy/overview" },
|
||||
{ text: "Configuration", link: "/proxy/configuration" },
|
||||
{ text: "Install the CA", link: "/proxy/ca-installation" },
|
||||
{ text: "Client Setup", link: "/proxy/client-setup" },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: "API Reference",
|
||||
items: [
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Docker Compose
|
||||
description: Run TRAWL with Docker Compose — scraper only or full stack.
|
||||
description: Run TRAWL with the supplied minimal, cached, or production Docker Compose setup.
|
||||
---
|
||||
|
||||
# Docker Compose
|
||||
|
||||
Four compose files live in the repo root, matching the setups shown on the landing page.
|
||||
Three Compose files live in the repository root.
|
||||
|
||||
## Scraper only
|
||||
|
||||
@@ -65,32 +65,22 @@ services:
|
||||
See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone#older-cpus-synology-nas) for how to tell if you need this, and the [README](https://github.com/germondai/trawl#docker-images-one-ghcr-package-two-tags) for the full tag comparison.
|
||||
:::
|
||||
|
||||
## Full stack
|
||||
|
||||
`docker-compose.full.yml` adds the landing page and docs on top of the scraper. Web and docs are built from source.
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.full.yml up -d
|
||||
```
|
||||
|
||||
| Service | URL | Description |
|
||||
| ------- | ---------------- | ------------- |
|
||||
| `trawl` | `localhost:8191` | Scraper API |
|
||||
| `web` | `localhost:3000` | Landing page |
|
||||
| `docs` | `localhost:3001` | Documentation |
|
||||
| `redis` | internal | Session cache |
|
||||
|
||||
First run builds the web and docs images locally — takes a couple of minutes. Subsequent runs are fast (layers cached).
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| -------------------------------- | -------------------- | -------------------------------------------------------------------------- |
|
||||
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
|
||||
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
|
||||
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable |
|
||||
| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) |
|
||||
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
|
||||
| Variable | Default | Description |
|
||||
| -------------------------------- | -------------------- | ----------------------------------------------------------------------- |
|
||||
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
|
||||
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
|
||||
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart after this many blocked/needs-js outcomes; set `0` to disable |
|
||||
| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) |
|
||||
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
|
||||
| `MITM_PROXY_ENABLED` | `false` | Starts the general HTTP/HTTPS proxy |
|
||||
| `MITM_PROXY_PORT` | `8192` | Proxy listen and published port |
|
||||
| `MITM_PROXY_HOST` | `0.0.0.0` | Proxy bind address |
|
||||
| `MITM_PROXY_CA_DIR` | `/data/proxy-ca` | Persistent root CA directory |
|
||||
|
||||
All supplied Compose files publish port `8192` and mount the `trawl_proxy_ca` volume. The listener
|
||||
does not start until `MITM_PROXY_ENABLED=true`. See [Proxy Configuration](/proxy/configuration).
|
||||
|
||||
## Logs
|
||||
|
||||
|
||||
@@ -156,7 +156,8 @@ Note: `proxy` on `/v1` is a TRAWL-specific extension — it is not part of the r
|
||||
|
||||
**Default:** `8191`
|
||||
|
||||
Host port the Docker port mapping forwards to TRAWL's internal listener. Defaults to `8191` — the same port FlareSolverr and Byparr use, so you can swap TRAWL in without changing any *arr app settings. The container itself always listens on `8191` internally; `PORT` only changes the **host-side** port (e.g. `"${PORT:-8191}:8191"` in every compose file).
|
||||
API listener port. It defaults to `8191`, the same port used by FlareSolverr and Byparr.
|
||||
The supplied Compose files use `${PORT:-8191}` for the host side of the `8191` container mapping.
|
||||
|
||||
To run TRAWL alongside FlareSolverr (or any other service that already binds `8191` on the host), set `PORT` in your shell or `.env` to any free port **before** running `docker compose up`:
|
||||
|
||||
@@ -165,32 +166,13 @@ PORT=9191 docker compose up -d
|
||||
# TRAWL reachable at http://localhost:9191, while port 8191 stays free for FlareSolverr.
|
||||
```
|
||||
|
||||
### `PORT_WEB`
|
||||
## Forward proxy
|
||||
|
||||
**Default:** `3000`
|
||||
|
||||
Port the Nuxt landing page listens on.
|
||||
The optional general HTTP/HTTPS proxy has its own listener, CA, tier cap, and debug settings.
|
||||
See [Proxy Configuration](/proxy/configuration) for all `MITM_PROXY_*` variables and deployment
|
||||
examples.
|
||||
|
||||
---
|
||||
|
||||
## Full `.env.example`
|
||||
|
||||
```ini
|
||||
# ── Redis ─────────────────────────────────────
|
||||
REDIS_URL=redis://localhost:6379
|
||||
|
||||
# ── Browser pool ──────────────────────────────
|
||||
BROWSER_POOL_SIZE=3
|
||||
BROWSER_ACQUIRE_TIMEOUT_MS=15000
|
||||
SESSION_TTL_SECONDS=3600
|
||||
|
||||
# ── Proxies (optional, comma-separated lists) ─
|
||||
PROXY_URL=
|
||||
RESIDENTIAL_PROXY_URL=
|
||||
PROXY_LIST_FILE=
|
||||
RESIDENTIAL_PROXY_LIST_FILE=
|
||||
|
||||
# ── Ports ─────────────────────────────────────
|
||||
PORT=8191
|
||||
PORT_WEB=3000
|
||||
```
|
||||
The repository's [`.env.example`](https://github.com/germondai/trawl/blob/main/.env.example) is the
|
||||
canonical copyable environment template.
|
||||
|
||||
@@ -94,8 +94,7 @@ trawl/
|
||||
│ └── tiers/ Tier 1–4 executors + orchestrator
|
||||
├── docker-compose.yml scraper + Redis (default)
|
||||
├── docker-compose.minimal.yml scraper only, no Redis
|
||||
├── docker-compose.prod.yml production with restart + healthcheck
|
||||
└── docker-compose.full.yml full stack including web + docs
|
||||
└── docker-compose.prod.yml production with restart + healthcheck
|
||||
```
|
||||
|
||||
## Cross-package TypeScript
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: Install the Proxy CA
|
||||
description: Trust TRAWL's generated root CA on operating systems, browsers, Java, and containers.
|
||||
---
|
||||
|
||||
# Install the proxy CA
|
||||
|
||||
TRAWL decrypts proxied HTTPS connections to detect challenge pages. On first proxy startup it
|
||||
creates:
|
||||
|
||||
- `ca.crt` — the root certificate clients install;
|
||||
- `ca.key` — the private signing key, which must remain secret.
|
||||
|
||||
Both files live in `MITM_PROXY_CA_DIR`. Per-host certificates are generated in memory and signed by
|
||||
this root. Persist the directory so clients only need to install the root once.
|
||||
|
||||
::: danger
|
||||
Anyone with `ca.key` can issue certificates trusted by clients that installed this CA. Keep the
|
||||
directory private, do not publish it, and never distribute `ca.key`.
|
||||
:::
|
||||
|
||||
## Download the certificate
|
||||
|
||||
The route is available when the proxy is enabled:
|
||||
|
||||
```bash
|
||||
curl http://<trawl-host>:8191/proxy-ca.crt -o trawl-ca.crt
|
||||
```
|
||||
|
||||
From the TRAWL container:
|
||||
|
||||
```bash
|
||||
docker cp trawl:/data/proxy-ca/ca.crt ./trawl-ca.crt
|
||||
```
|
||||
|
||||
## macOS
|
||||
|
||||
Install into the system keychain:
|
||||
|
||||
```bash
|
||||
sudo security add-trusted-cert -d -r trustRoot \
|
||||
-k /Library/Keychains/System.keychain ./trawl-ca.crt
|
||||
```
|
||||
|
||||
Verify or remove it:
|
||||
|
||||
```bash
|
||||
security find-certificate -c "TRAWL MITM Proxy CA"
|
||||
sudo security delete-certificate -c "TRAWL MITM Proxy CA" \
|
||||
/Library/Keychains/System.keychain
|
||||
```
|
||||
|
||||
## Debian and Ubuntu
|
||||
|
||||
```bash
|
||||
sudo cp trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
|
||||
sudo update-ca-certificates
|
||||
```
|
||||
|
||||
## RHEL, Fedora, and Amazon Linux
|
||||
|
||||
```bash
|
||||
sudo cp trawl-ca.crt /etc/pki/ca-trust/source/anchors/trawl-ca.crt
|
||||
sudo update-ca-trust
|
||||
```
|
||||
|
||||
## Windows
|
||||
|
||||
Run PowerShell as Administrator:
|
||||
|
||||
```powershell
|
||||
Import-Certificate -FilePath .\trawl-ca.crt `
|
||||
-CertStoreLocation Cert:\LocalMachine\Root
|
||||
```
|
||||
|
||||
Remove it later:
|
||||
|
||||
```powershell
|
||||
Get-ChildItem Cert:\LocalMachine\Root |
|
||||
Where-Object { $_.Subject -like "*TRAWL MITM*" } |
|
||||
Remove-Item
|
||||
```
|
||||
|
||||
## Firefox and NSS stores
|
||||
|
||||
Firefox installations that do not use the operating-system roots need an NSS import:
|
||||
|
||||
```bash
|
||||
certutil -A -n "TRAWL MITM" -t "CT,C,C" -i trawl-ca.crt \
|
||||
-d sql:$HOME/.mozilla/firefox/<profile-directory>
|
||||
```
|
||||
|
||||
Alternatively use **Settings → Privacy & Security → Certificates → View Certificates →
|
||||
Authorities → Import**.
|
||||
|
||||
## Java and JDownloader
|
||||
|
||||
Java applications use their own `cacerts` store:
|
||||
|
||||
```bash
|
||||
keytool -importcert -noprompt -trustcacerts \
|
||||
-alias trawl-mitm-ca \
|
||||
-file trawl-ca.crt \
|
||||
-keystore "<java-home>/lib/security/cacerts" \
|
||||
-storepass changeit
|
||||
```
|
||||
|
||||
JDownloader bundles a JRE. Locate its active Java path in JDownloader's advanced settings, import
|
||||
the certificate into that JRE's `lib/security/cacerts`, and restart JDownloader.
|
||||
|
||||
Prowlarr, Sonarr, and Radarr are .NET applications. Their containers normally use the Linux system
|
||||
trust store, not Java `cacerts`.
|
||||
|
||||
## Docker clients
|
||||
|
||||
Mount the certificate into the client container and install it during container initialization.
|
||||
For Debian-based images:
|
||||
|
||||
```bash
|
||||
cp /config/trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
|
||||
update-ca-certificates
|
||||
```
|
||||
|
||||
The exact startup-hook directory depends on the image. LinuxServer images support
|
||||
`/custom-cont-init.d/`; other images may require a derived Dockerfile.
|
||||
|
||||
## Rotation and recovery
|
||||
|
||||
Do not delete or replace `ca.crt` or `ca.key` during normal upgrades. If either is lost, TRAWL
|
||||
generates a new root on the next startup and every client must install the new certificate.
|
||||
|
||||
To intentionally rotate the CA:
|
||||
|
||||
1. stop TRAWL;
|
||||
2. back up and remove both CA files;
|
||||
3. start TRAWL and download the new `ca.crt`;
|
||||
4. remove the old root from every client;
|
||||
5. install the new root.
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: Proxy Client Setup
|
||||
description: Configure curl, browsers, Prowlarr, JDownloader, and system-wide proxy settings.
|
||||
---
|
||||
|
||||
# Client setup
|
||||
|
||||
Install the [TRAWL root CA](/proxy/ca-installation) before enabling HTTPS proxying. Configure both
|
||||
the HTTP and HTTPS proxy address as `http://<trawl-host>:8192`; the proxy endpoint itself uses
|
||||
plain HTTP, including for HTTPS `CONNECT`.
|
||||
|
||||
## curl
|
||||
|
||||
```bash
|
||||
curl --proxy http://127.0.0.1:8192 https://example.com/
|
||||
```
|
||||
|
||||
Before installing the CA system-wide, point curl at it explicitly:
|
||||
|
||||
```bash
|
||||
curl --proxy http://127.0.0.1:8192 \
|
||||
--cacert ./trawl-ca.crt \
|
||||
https://example.com/
|
||||
```
|
||||
|
||||
Use verbose output to see `CONNECT`, certificate validation, response headers, and redirects:
|
||||
|
||||
```bash
|
||||
curl -v --proxy http://127.0.0.1:8192 https://example.com/
|
||||
```
|
||||
|
||||
## Browser or operating-system proxy
|
||||
|
||||
Set:
|
||||
|
||||
```text
|
||||
HTTP proxy: <trawl-host>:8192
|
||||
HTTPS proxy: <trawl-host>:8192
|
||||
```
|
||||
|
||||
Do not configure TRAWL as a SOCKS proxy. Install the CA in every trust store used by the browser.
|
||||
Some Firefox profiles use their own NSS store even when the operating-system store is configured.
|
||||
|
||||
Using TRAWL system-wide sends sensitive application traffic through a TLS-terminating service.
|
||||
Prefer per-application or per-domain proxy rules when possible.
|
||||
|
||||
## Prowlarr
|
||||
|
||||
For indexers that support FlareSolverr, the normal `/v1` integration remains the simplest option.
|
||||
Use the forward proxy when the indexer performs its own follow-up fetch and the solved cookie is not
|
||||
portable.
|
||||
|
||||
1. Open **Settings → Indexer Proxies**.
|
||||
2. Add an **HTTP** proxy.
|
||||
3. Set the host to the TRAWL hostname and port to `8192`.
|
||||
4. Assign a tag if only selected indexers should use the proxy.
|
||||
5. Add that tag to the intended indexers.
|
||||
|
||||
The Prowlarr host or container must trust the TRAWL CA. Prowlarr is a .NET application; install the
|
||||
CA into its operating-system or container trust store.
|
||||
|
||||
## Jackett
|
||||
|
||||
Jackett normally uses TRAWL through its FlareSolverr-compatible API. If a tracker plugin permits a
|
||||
general HTTP proxy and needs connection-bound clearance, configure `<trawl-host>:8192` there and
|
||||
install the CA in Jackett's host/container trust store.
|
||||
|
||||
## JDownloader
|
||||
|
||||
1. Import the CA into JDownloader's bundled Java trust store.
|
||||
2. Open **Settings → Connection Manager**.
|
||||
3. Add an HTTP proxy using the TRAWL host and port `8192`.
|
||||
4. Restart JDownloader after changing its Java trust store.
|
||||
|
||||
TRAWL forwards normal downloads, Range requests, redirects, and binary responses. Challenge
|
||||
escalation is designed primarily for navigation/document requests. On a challenged request with a
|
||||
binary upload body, the browser fallback is not byte-transparent because `/scrape` accepts a text
|
||||
body.
|
||||
|
||||
## changedetection.io and other services
|
||||
|
||||
Use `http://<trawl-host>:8192` as the service's HTTP and HTTPS proxy and add the CA to its container
|
||||
trust store. Environment variable conventions differ, but many command-line applications support:
|
||||
|
||||
```ini
|
||||
HTTP_PROXY=http://trawl:8192
|
||||
HTTPS_PROXY=http://trawl:8192
|
||||
NO_PROXY=localhost,127.0.0.1,trawl
|
||||
```
|
||||
|
||||
Avoid setting these globally inside the TRAWL container itself; that can create a proxy loop.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause |
|
||||
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| `CERTIFICATE_VERIFY_FAILED`, `PKIX path building failed` | The application does not trust TRAWL's root CA |
|
||||
| `400 Bad Request` | Invalid proxy request framing, URL, method, or `Content-Length` |
|
||||
| Challenge HTML is returned | Challenge was not recognized, solving failed, or the response was intentionally streamed |
|
||||
| A small website downloads slowly | Host is cached as challenged and is entering the browser tiers |
|
||||
| Range request returns `200` | The destination ignored `Range`; TRAWL does not synthesize partial responses |
|
||||
| WebSocket handshake is rejected | The direct handshake lacked required cookies or authorization |
|
||||
| Proxy works in curl but not the application | The application uses a separate trust store or ignores system proxy settings |
|
||||
|
||||
Enable `MITM_PROXY_DEBUG=true`, reproduce one request, and inspect the TRAWL logs to see whether it
|
||||
used Tier 0, streaming, or the scrape fallback.
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
title: Proxy Configuration
|
||||
description: Enable and configure TRAWL's challenge-bypassing HTTP/HTTPS proxy.
|
||||
---
|
||||
|
||||
# Proxy configuration
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --------------------- | ---------------- | -------------------------------------------------------- |
|
||||
| `MITM_PROXY_ENABLED` | `false` | Starts the forward-proxy listener |
|
||||
| `MITM_PROXY_HOST` | `0.0.0.0` | Listener bind address |
|
||||
| `MITM_PROXY_PORT` | `8192` | Listener port |
|
||||
| `MITM_PROXY_CA_DIR` | `/data/proxy-ca` | Persistent root CA certificate and private-key directory |
|
||||
| `MITM_PROXY_MAX_TIER` | `4` | Highest solver tier available to proxy escalation |
|
||||
| `MITM_PROXY_DEBUG` | `false` | Logs proxied requests and tier attempts |
|
||||
|
||||
Example:
|
||||
|
||||
```ini
|
||||
MITM_PROXY_ENABLED=true
|
||||
MITM_PROXY_HOST=127.0.0.1
|
||||
MITM_PROXY_PORT=8192
|
||||
MITM_PROXY_CA_DIR=/data/proxy-ca
|
||||
MITM_PROXY_MAX_TIER=4
|
||||
MITM_PROXY_DEBUG=false
|
||||
```
|
||||
|
||||
Use `127.0.0.1` for a local installation. Docker clients on a bridge network normally require
|
||||
`0.0.0.0`; restrict access with container networking or a host firewall.
|
||||
|
||||
`MITM_PROXY_MAX_TIER=3` prevents proxy requests from consuming a configured residential Tier 4
|
||||
proxy. An empty or invalid value uses the normal maximum of Tier 4.
|
||||
|
||||
## Docker Compose
|
||||
|
||||
The supplied Compose files publish the API and proxy ports and persist the root CA:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
trawl:
|
||||
ports:
|
||||
- "8191:8191"
|
||||
- "8192:8192"
|
||||
environment:
|
||||
MITM_PROXY_ENABLED: "true"
|
||||
MITM_PROXY_HOST: 0.0.0.0
|
||||
MITM_PROXY_PORT: 8192
|
||||
MITM_PROXY_CA_DIR: /data/proxy-ca
|
||||
volumes:
|
||||
- trawl_proxy_ca:/data/proxy-ca
|
||||
|
||||
volumes:
|
||||
trawl_proxy_ca:
|
||||
```
|
||||
|
||||
Start or recreate the service after changing proxy variables:
|
||||
|
||||
```bash
|
||||
docker compose up -d --force-recreate trawl
|
||||
```
|
||||
|
||||
## Upstream proxy interaction
|
||||
|
||||
Tier 0 direct traffic leaves from the TRAWL host directly. `PROXY_URL` and
|
||||
`RESIDENTIAL_PROXY_URL` apply when a challenged request escalates into the scrape tiers; they do
|
||||
not turn the entire forward proxy into a chain through another proxy.
|
||||
|
||||
The normal sticky-per-domain rotation and failure cooldown rules apply during escalation. Use
|
||||
`MITM_PROXY_MAX_TIER` to cap which tiers the forward proxy may reach.
|
||||
|
||||
## Verify the listener
|
||||
|
||||
Download the CA and test an HTTPS request:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8191/proxy-ca.crt -o trawl-ca.crt
|
||||
curl --proxy http://127.0.0.1:8192 \
|
||||
--cacert ./trawl-ca.crt \
|
||||
https://example.com/
|
||||
```
|
||||
|
||||
Test plain HTTP:
|
||||
|
||||
```bash
|
||||
curl --proxy http://127.0.0.1:8192 http://neverssl.com/
|
||||
```
|
||||
|
||||
Test Range forwarding:
|
||||
|
||||
```bash
|
||||
curl --proxy http://127.0.0.1:8192 \
|
||||
--cacert ./trawl-ca.crt \
|
||||
-H 'Range: bytes=0-99' \
|
||||
-D - https://httpbin.org/range/1024
|
||||
```
|
||||
|
||||
The Range request should return `206` and a 100-byte body when the upstream supports it.
|
||||
|
||||
## Debug logging
|
||||
|
||||
Set `MITM_PROXY_DEBUG=true` to log direct forwarding, streaming decisions, challenge escalation,
|
||||
winning scrape tiers, statuses, content types, and payload sizes. Disable it after troubleshooting;
|
||||
general proxy clients can generate a large volume of requests.
|
||||
|
||||
The proxy has no authentication layer. Never publish port `8192` directly to the internet.
|
||||
@@ -0,0 +1,125 @@
|
||||
---
|
||||
title: Proxy Overview
|
||||
description: How TRAWL's general HTTP/HTTPS proxy forwards traffic and escalates detected challenges.
|
||||
---
|
||||
|
||||
# Proxy overview
|
||||
|
||||
TRAWL can expose a general HTTP/HTTPS forward proxy, normally on port `8192`. It behaves like a
|
||||
direct proxy for ordinary traffic and invokes the existing `/scrape` tier pipeline only when a
|
||||
small buffered response looks like a supported challenge.
|
||||
|
||||
The proxy is disabled by default. It is intended for trusted clients on localhost, a private LAN,
|
||||
or a private container network.
|
||||
|
||||
## Traffic flow
|
||||
|
||||
```text
|
||||
Client
|
||||
│
|
||||
├─ HTTP request ─────────────────────────────┐
|
||||
│ │
|
||||
└─ HTTPS CONNECT → local TLS termination ────┤
|
||||
▼
|
||||
Tier 0 direct forward
|
||||
│
|
||||
┌────────────────────────┼────────────────────────┐
|
||||
│ │ │
|
||||
normal small response challenge detected large/media response
|
||||
│ │ │
|
||||
buffer + return run scrape tiers stream
|
||||
│
|
||||
Tier 1 → Tier 2 → Tier 3 → Tier 4
|
||||
│
|
||||
return solved HTML or raw bytes
|
||||
```
|
||||
|
||||
HTTPS requires local TLS termination so TRAWL can inspect the response body. TRAWL generates a
|
||||
per-host certificate signed by its own root CA. The client must trust that root certificate.
|
||||
|
||||
## Direct forwarding
|
||||
|
||||
Tier 0 opens a normal TCP or TLS connection to the destination and forwards the request method,
|
||||
body, and end-to-end headers. It preserves headers such as:
|
||||
|
||||
- `Authorization`, `Cookie`, `User-Agent`, `Referer`, and `Origin`
|
||||
- custom API headers
|
||||
- `Range`, `If-Range`, cache validators, and content metadata
|
||||
|
||||
Hop-by-hop headers are rebuilt or removed at the proxy boundary.
|
||||
|
||||
Responses not selected by the streaming policy—including small HTML, JSON, XML, text, and unknown
|
||||
small payloads—are buffered. Compressed `gzip`, `deflate`, and Brotli responses are decoded only
|
||||
for challenge inspection; the original bytes remain the response body. Chunked buffered responses
|
||||
are de-chunked before being returned.
|
||||
|
||||
## Challenge escalation
|
||||
|
||||
When Tier 0 detects a supported challenge wall, the request is passed to the same `scrape()`
|
||||
orchestrator used by `POST /scrape`:
|
||||
|
||||
1. plain HTTP fetch;
|
||||
2. cached browser session;
|
||||
3. fresh browser challenge solve;
|
||||
4. residential proxy solve, when configured.
|
||||
|
||||
For browser-tier HTML responses, the proxy returns the rendered solved DOM rather than the original
|
||||
challenge response. Binary responses use the raw response bytes when the winning tier exposes them.
|
||||
|
||||
A hostname that produced a Cloudflare challenge is cached for five minutes. During that window,
|
||||
later requests for the hostname skip Tier 0 and go directly to the tiered solver.
|
||||
|
||||
Challenge and CAPTCHA solving is best effort. A site can still reject the browser, require user
|
||||
interaction, bind clearance to an unsupported signal, or change its challenge implementation.
|
||||
|
||||
## Buffering and streaming
|
||||
|
||||
The proxy buffers ordinary responses by default so challenge detection sees the complete body.
|
||||
It streams when at least one of these applies:
|
||||
|
||||
- `Content-Length` is at least 8 MiB;
|
||||
- the content type is `video/*` or `audio/*`;
|
||||
- an unknown-length response has a known binary content type;
|
||||
- the URL has a known media, archive, installer, disk-image, PDF, or font extension.
|
||||
|
||||
Streaming keeps large files out of TRAWL's memory. Streamed responses bypass body-based challenge
|
||||
detection, so the policy intentionally targets content that is unlikely to be an HTML challenge.
|
||||
|
||||
`Range` requests are forwarded unchanged. A compliant upstream `206 Partial Content` response,
|
||||
including `Content-Range` and `Accept-Ranges`, is passed back to the client. If the Range response
|
||||
is instead a detectable challenge page, it can escalate through the normal solver pipeline.
|
||||
|
||||
## WebSockets
|
||||
|
||||
HTTP and HTTPS WebSocket upgrade requests use a bidirectional byte relay after the upstream
|
||||
`101 Switching Protocols` response. WebSocket frames are not buffered or interpreted.
|
||||
|
||||
The WebSocket handshake does not escalate through the browser solver. If a WebSocket endpoint
|
||||
requires a clearance cookie, the client must already have suitable credentials or obtain them
|
||||
through a preceding solved HTTP flow.
|
||||
|
||||
## Current protocol boundaries
|
||||
|
||||
The current listener is an HTTP/1.1 proxy:
|
||||
|
||||
- one request is served per proxied TLS connection;
|
||||
- HTTP keep-alive and pipelining are not reused;
|
||||
- request bodies use `Content-Length`; chunked uploads are not decoded;
|
||||
- duplicate upstream response headers are represented by the first observed value;
|
||||
- proxy authentication is not implemented;
|
||||
- HTTP/2 and HTTP/3 are not terminated between the client and TRAWL.
|
||||
|
||||
On a challenged request, the browser fallback converts a request body to UTF-8 because the public
|
||||
scrape contract currently accepts a string body. Ordinary direct forwarding preserves binary
|
||||
request bodies exactly.
|
||||
|
||||
## Proxy versus API endpoints
|
||||
|
||||
| Interface | Best for | Response handling |
|
||||
| ------------------ | -------------------------------------------- | ------------------------------------------------------- |
|
||||
| `POST /v1` | FlareSolverr-compatible integrations | Returns HTML, cookies, and user agent |
|
||||
| `POST /scrape` | Native programmatic scraping | Returns tier metadata and solved content |
|
||||
| HTTP proxy `:8192` | Applications that perform their own requests | Forwards HTTP traffic and escalates detected challenges |
|
||||
|
||||
Use `/v1` or `/scrape` when the application explicitly supports a solver API. Use the proxy when the
|
||||
application only knows how to make ordinary HTTP requests or when cookie handoff is not sufficient.
|
||||
Reference in New Issue
Block a user