diff --git a/apps/docs/.vitepress/config.ts b/apps/docs/.vitepress/config.ts index 3789164..1b402ba 100644 --- a/apps/docs/.vitepress/config.ts +++ b/apps/docs/.vitepress/config.ts @@ -53,6 +53,15 @@ export default defineConfig({ { text: "*arr Apps", link: "/integrations/arr-apps" }, ], }, + { + text: "Forward Proxy", + items: [ + { text: "Overview & Traffic Flow", link: "/proxy/overview" }, + { text: "Configuration", link: "/proxy/configuration" }, + { text: "Install the CA", link: "/proxy/ca-installation" }, + { text: "Client Setup", link: "/proxy/client-setup" }, + ], + }, { text: "API Reference", items: [ diff --git a/apps/docs/deployment/docker-compose.md b/apps/docs/deployment/docker-compose.md index 9406320..7781432 100644 --- a/apps/docs/deployment/docker-compose.md +++ b/apps/docs/deployment/docker-compose.md @@ -1,11 +1,11 @@ --- title: Docker Compose -description: Run TRAWL with Docker Compose — scraper only or full stack. +description: Run TRAWL with the supplied minimal, cached, or production Docker Compose setup. --- # Docker Compose -Four compose files live in the repo root, matching the setups shown on the landing page. +Three Compose files live in the repository root. ## Scraper only @@ -65,32 +65,22 @@ services: See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone#older-cpus-synology-nas) for how to tell if you need this, and the [README](https://github.com/germondai/trawl#docker-images-one-ghcr-package-two-tags) for the full tag comparison. ::: -## Full stack - -`docker-compose.full.yml` adds the landing page and docs on top of the scraper. Web and docs are built from source. - -```bash -docker compose -f docker-compose.full.yml up -d -``` - -| Service | URL | Description | -| ------- | ---------------- | ------------- | -| `trawl` | `localhost:8191` | Scraper API | -| `web` | `localhost:3000` | Landing page | -| `docs` | `localhost:3001` | Documentation | -| `redis` | internal | Session cache | - -First run builds the web and docs images locally — takes a couple of minutes. Subsequent runs are fast (layers cached). - ## Environment variables -| Variable | Default | Description | -| -------------------------------- | -------------------- | -------------------------------------------------------------------------- | -| `BROWSER_POOL_SIZE` | `3` | Warm browser instances | -| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 | -| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable | -| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) | -| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation | +| Variable | Default | Description | +| -------------------------------- | -------------------- | ----------------------------------------------------------------------- | +| `BROWSER_POOL_SIZE` | `3` | Warm browser instances | +| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 | +| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart after this many blocked/needs-js outcomes; set `0` to disable | +| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) | +| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation | +| `MITM_PROXY_ENABLED` | `false` | Starts the general HTTP/HTTPS proxy | +| `MITM_PROXY_PORT` | `8192` | Proxy listen and published port | +| `MITM_PROXY_HOST` | `0.0.0.0` | Proxy bind address | +| `MITM_PROXY_CA_DIR` | `/data/proxy-ca` | Persistent root CA directory | + +All supplied Compose files publish port `8192` and mount the `trawl_proxy_ca` volume. The listener +does not start until `MITM_PROXY_ENABLED=true`. See [Proxy Configuration](/proxy/configuration). ## Logs diff --git a/apps/docs/getting-started/configuration.md b/apps/docs/getting-started/configuration.md index c9e09d4..f6945e6 100644 --- a/apps/docs/getting-started/configuration.md +++ b/apps/docs/getting-started/configuration.md @@ -156,7 +156,8 @@ Note: `proxy` on `/v1` is a TRAWL-specific extension — it is not part of the r **Default:** `8191` -Host port the Docker port mapping forwards to TRAWL's internal listener. Defaults to `8191` — the same port FlareSolverr and Byparr use, so you can swap TRAWL in without changing any *arr app settings. The container itself always listens on `8191` internally; `PORT` only changes the **host-side** port (e.g. `"${PORT:-8191}:8191"` in every compose file). +API listener port. It defaults to `8191`, the same port used by FlareSolverr and Byparr. +The supplied Compose files use `${PORT:-8191}` for the host side of the `8191` container mapping. To run TRAWL alongside FlareSolverr (or any other service that already binds `8191` on the host), set `PORT` in your shell or `.env` to any free port **before** running `docker compose up`: @@ -165,32 +166,13 @@ PORT=9191 docker compose up -d # TRAWL reachable at http://localhost:9191, while port 8191 stays free for FlareSolverr. ``` -### `PORT_WEB` +## Forward proxy -**Default:** `3000` - -Port the Nuxt landing page listens on. +The optional general HTTP/HTTPS proxy has its own listener, CA, tier cap, and debug settings. +See [Proxy Configuration](/proxy/configuration) for all `MITM_PROXY_*` variables and deployment +examples. --- -## Full `.env.example` - -```ini -# ── Redis ───────────────────────────────────── -REDIS_URL=redis://localhost:6379 - -# ── Browser pool ────────────────────────────── -BROWSER_POOL_SIZE=3 -BROWSER_ACQUIRE_TIMEOUT_MS=15000 -SESSION_TTL_SECONDS=3600 - -# ── Proxies (optional, comma-separated lists) ─ -PROXY_URL= -RESIDENTIAL_PROXY_URL= -PROXY_LIST_FILE= -RESIDENTIAL_PROXY_LIST_FILE= - -# ── Ports ───────────────────────────────────── -PORT=8191 -PORT_WEB=3000 -``` +The repository's [`.env.example`](https://github.com/germondai/trawl/blob/main/.env.example) is the +canonical copyable environment template. diff --git a/apps/docs/getting-started/local-development.md b/apps/docs/getting-started/local-development.md index 94948d6..1bb210d 100644 --- a/apps/docs/getting-started/local-development.md +++ b/apps/docs/getting-started/local-development.md @@ -94,8 +94,7 @@ trawl/ │ └── tiers/ Tier 1–4 executors + orchestrator ├── docker-compose.yml scraper + Redis (default) ├── docker-compose.minimal.yml scraper only, no Redis -├── docker-compose.prod.yml production with restart + healthcheck -└── docker-compose.full.yml full stack including web + docs +└── docker-compose.prod.yml production with restart + healthcheck ``` ## Cross-package TypeScript diff --git a/apps/docs/proxy/ca-installation.md b/apps/docs/proxy/ca-installation.md new file mode 100644 index 0000000..427913b --- /dev/null +++ b/apps/docs/proxy/ca-installation.md @@ -0,0 +1,138 @@ +--- +title: Install the Proxy CA +description: Trust TRAWL's generated root CA on operating systems, browsers, Java, and containers. +--- + +# Install the proxy CA + +TRAWL decrypts proxied HTTPS connections to detect challenge pages. On first proxy startup it +creates: + +- `ca.crt` — the root certificate clients install; +- `ca.key` — the private signing key, which must remain secret. + +Both files live in `MITM_PROXY_CA_DIR`. Per-host certificates are generated in memory and signed by +this root. Persist the directory so clients only need to install the root once. + +::: danger +Anyone with `ca.key` can issue certificates trusted by clients that installed this CA. Keep the +directory private, do not publish it, and never distribute `ca.key`. +::: + +## Download the certificate + +The route is available when the proxy is enabled: + +```bash +curl http://:8191/proxy-ca.crt -o trawl-ca.crt +``` + +From the TRAWL container: + +```bash +docker cp trawl:/data/proxy-ca/ca.crt ./trawl-ca.crt +``` + +## macOS + +Install into the system keychain: + +```bash +sudo security add-trusted-cert -d -r trustRoot \ + -k /Library/Keychains/System.keychain ./trawl-ca.crt +``` + +Verify or remove it: + +```bash +security find-certificate -c "TRAWL MITM Proxy CA" +sudo security delete-certificate -c "TRAWL MITM Proxy CA" \ + /Library/Keychains/System.keychain +``` + +## Debian and Ubuntu + +```bash +sudo cp trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt +sudo update-ca-certificates +``` + +## RHEL, Fedora, and Amazon Linux + +```bash +sudo cp trawl-ca.crt /etc/pki/ca-trust/source/anchors/trawl-ca.crt +sudo update-ca-trust +``` + +## Windows + +Run PowerShell as Administrator: + +```powershell +Import-Certificate -FilePath .\trawl-ca.crt ` + -CertStoreLocation Cert:\LocalMachine\Root +``` + +Remove it later: + +```powershell +Get-ChildItem Cert:\LocalMachine\Root | + Where-Object { $_.Subject -like "*TRAWL MITM*" } | + Remove-Item +``` + +## Firefox and NSS stores + +Firefox installations that do not use the operating-system roots need an NSS import: + +```bash +certutil -A -n "TRAWL MITM" -t "CT,C,C" -i trawl-ca.crt \ + -d sql:$HOME/.mozilla/firefox/ +``` + +Alternatively use **Settings → Privacy & Security → Certificates → View Certificates → +Authorities → Import**. + +## Java and JDownloader + +Java applications use their own `cacerts` store: + +```bash +keytool -importcert -noprompt -trustcacerts \ + -alias trawl-mitm-ca \ + -file trawl-ca.crt \ + -keystore "/lib/security/cacerts" \ + -storepass changeit +``` + +JDownloader bundles a JRE. Locate its active Java path in JDownloader's advanced settings, import +the certificate into that JRE's `lib/security/cacerts`, and restart JDownloader. + +Prowlarr, Sonarr, and Radarr are .NET applications. Their containers normally use the Linux system +trust store, not Java `cacerts`. + +## Docker clients + +Mount the certificate into the client container and install it during container initialization. +For Debian-based images: + +```bash +cp /config/trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt +update-ca-certificates +``` + +The exact startup-hook directory depends on the image. LinuxServer images support +`/custom-cont-init.d/`; other images may require a derived Dockerfile. + +## Rotation and recovery + +Do not delete or replace `ca.crt` or `ca.key` during normal upgrades. If either is lost, TRAWL +generates a new root on the next startup and every client must install the new certificate. + +To intentionally rotate the CA: + +1. stop TRAWL; +2. back up and remove both CA files; +3. start TRAWL and download the new `ca.crt`; +4. remove the old root from every client; +5. install the new root. diff --git a/apps/docs/proxy/client-setup.md b/apps/docs/proxy/client-setup.md new file mode 100644 index 0000000..50b1292 --- /dev/null +++ b/apps/docs/proxy/client-setup.md @@ -0,0 +1,106 @@ +--- +title: Proxy Client Setup +description: Configure curl, browsers, Prowlarr, JDownloader, and system-wide proxy settings. +--- + +# Client setup + +Install the [TRAWL root CA](/proxy/ca-installation) before enabling HTTPS proxying. Configure both +the HTTP and HTTPS proxy address as `http://:8192`; the proxy endpoint itself uses +plain HTTP, including for HTTPS `CONNECT`. + +## curl + +```bash +curl --proxy http://127.0.0.1:8192 https://example.com/ +``` + +Before installing the CA system-wide, point curl at it explicitly: + +```bash +curl --proxy http://127.0.0.1:8192 \ + --cacert ./trawl-ca.crt \ + https://example.com/ +``` + +Use verbose output to see `CONNECT`, certificate validation, response headers, and redirects: + +```bash +curl -v --proxy http://127.0.0.1:8192 https://example.com/ +``` + +## Browser or operating-system proxy + +Set: + +```text +HTTP proxy: :8192 +HTTPS proxy: :8192 +``` + +Do not configure TRAWL as a SOCKS proxy. Install the CA in every trust store used by the browser. +Some Firefox profiles use their own NSS store even when the operating-system store is configured. + +Using TRAWL system-wide sends sensitive application traffic through a TLS-terminating service. +Prefer per-application or per-domain proxy rules when possible. + +## Prowlarr + +For indexers that support FlareSolverr, the normal `/v1` integration remains the simplest option. +Use the forward proxy when the indexer performs its own follow-up fetch and the solved cookie is not +portable. + +1. Open **Settings → Indexer Proxies**. +2. Add an **HTTP** proxy. +3. Set the host to the TRAWL hostname and port to `8192`. +4. Assign a tag if only selected indexers should use the proxy. +5. Add that tag to the intended indexers. + +The Prowlarr host or container must trust the TRAWL CA. Prowlarr is a .NET application; install the +CA into its operating-system or container trust store. + +## Jackett + +Jackett normally uses TRAWL through its FlareSolverr-compatible API. If a tracker plugin permits a +general HTTP proxy and needs connection-bound clearance, configure `:8192` there and +install the CA in Jackett's host/container trust store. + +## JDownloader + +1. Import the CA into JDownloader's bundled Java trust store. +2. Open **Settings → Connection Manager**. +3. Add an HTTP proxy using the TRAWL host and port `8192`. +4. Restart JDownloader after changing its Java trust store. + +TRAWL forwards normal downloads, Range requests, redirects, and binary responses. Challenge +escalation is designed primarily for navigation/document requests. On a challenged request with a +binary upload body, the browser fallback is not byte-transparent because `/scrape` accepts a text +body. + +## changedetection.io and other services + +Use `http://:8192` as the service's HTTP and HTTPS proxy and add the CA to its container +trust store. Environment variable conventions differ, but many command-line applications support: + +```ini +HTTP_PROXY=http://trawl:8192 +HTTPS_PROXY=http://trawl:8192 +NO_PROXY=localhost,127.0.0.1,trawl +``` + +Avoid setting these globally inside the TRAWL container itself; that can create a proxy loop. + +## Troubleshooting + +| Symptom | Likely cause | +| -------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| `CERTIFICATE_VERIFY_FAILED`, `PKIX path building failed` | The application does not trust TRAWL's root CA | +| `400 Bad Request` | Invalid proxy request framing, URL, method, or `Content-Length` | +| Challenge HTML is returned | Challenge was not recognized, solving failed, or the response was intentionally streamed | +| A small website downloads slowly | Host is cached as challenged and is entering the browser tiers | +| Range request returns `200` | The destination ignored `Range`; TRAWL does not synthesize partial responses | +| WebSocket handshake is rejected | The direct handshake lacked required cookies or authorization | +| Proxy works in curl but not the application | The application uses a separate trust store or ignores system proxy settings | + +Enable `MITM_PROXY_DEBUG=true`, reproduce one request, and inspect the TRAWL logs to see whether it +used Tier 0, streaming, or the scrape fallback. diff --git a/apps/docs/proxy/configuration.md b/apps/docs/proxy/configuration.md new file mode 100644 index 0000000..fa9f6fa --- /dev/null +++ b/apps/docs/proxy/configuration.md @@ -0,0 +1,107 @@ +--- +title: Proxy Configuration +description: Enable and configure TRAWL's challenge-bypassing HTTP/HTTPS proxy. +--- + +# Proxy configuration + +## Environment variables + +| Variable | Default | Purpose | +| --------------------- | ---------------- | -------------------------------------------------------- | +| `MITM_PROXY_ENABLED` | `false` | Starts the forward-proxy listener | +| `MITM_PROXY_HOST` | `0.0.0.0` | Listener bind address | +| `MITM_PROXY_PORT` | `8192` | Listener port | +| `MITM_PROXY_CA_DIR` | `/data/proxy-ca` | Persistent root CA certificate and private-key directory | +| `MITM_PROXY_MAX_TIER` | `4` | Highest solver tier available to proxy escalation | +| `MITM_PROXY_DEBUG` | `false` | Logs proxied requests and tier attempts | + +Example: + +```ini +MITM_PROXY_ENABLED=true +MITM_PROXY_HOST=127.0.0.1 +MITM_PROXY_PORT=8192 +MITM_PROXY_CA_DIR=/data/proxy-ca +MITM_PROXY_MAX_TIER=4 +MITM_PROXY_DEBUG=false +``` + +Use `127.0.0.1` for a local installation. Docker clients on a bridge network normally require +`0.0.0.0`; restrict access with container networking or a host firewall. + +`MITM_PROXY_MAX_TIER=3` prevents proxy requests from consuming a configured residential Tier 4 +proxy. An empty or invalid value uses the normal maximum of Tier 4. + +## Docker Compose + +The supplied Compose files publish the API and proxy ports and persist the root CA: + +```yaml +services: + trawl: + ports: + - "8191:8191" + - "8192:8192" + environment: + MITM_PROXY_ENABLED: "true" + MITM_PROXY_HOST: 0.0.0.0 + MITM_PROXY_PORT: 8192 + MITM_PROXY_CA_DIR: /data/proxy-ca + volumes: + - trawl_proxy_ca:/data/proxy-ca + +volumes: + trawl_proxy_ca: +``` + +Start or recreate the service after changing proxy variables: + +```bash +docker compose up -d --force-recreate trawl +``` + +## Upstream proxy interaction + +Tier 0 direct traffic leaves from the TRAWL host directly. `PROXY_URL` and +`RESIDENTIAL_PROXY_URL` apply when a challenged request escalates into the scrape tiers; they do +not turn the entire forward proxy into a chain through another proxy. + +The normal sticky-per-domain rotation and failure cooldown rules apply during escalation. Use +`MITM_PROXY_MAX_TIER` to cap which tiers the forward proxy may reach. + +## Verify the listener + +Download the CA and test an HTTPS request: + +```bash +curl http://127.0.0.1:8191/proxy-ca.crt -o trawl-ca.crt +curl --proxy http://127.0.0.1:8192 \ + --cacert ./trawl-ca.crt \ + https://example.com/ +``` + +Test plain HTTP: + +```bash +curl --proxy http://127.0.0.1:8192 http://neverssl.com/ +``` + +Test Range forwarding: + +```bash +curl --proxy http://127.0.0.1:8192 \ + --cacert ./trawl-ca.crt \ + -H 'Range: bytes=0-99' \ + -D - https://httpbin.org/range/1024 +``` + +The Range request should return `206` and a 100-byte body when the upstream supports it. + +## Debug logging + +Set `MITM_PROXY_DEBUG=true` to log direct forwarding, streaming decisions, challenge escalation, +winning scrape tiers, statuses, content types, and payload sizes. Disable it after troubleshooting; +general proxy clients can generate a large volume of requests. + +The proxy has no authentication layer. Never publish port `8192` directly to the internet. diff --git a/apps/docs/proxy/overview.md b/apps/docs/proxy/overview.md new file mode 100644 index 0000000..4aa2bcf --- /dev/null +++ b/apps/docs/proxy/overview.md @@ -0,0 +1,125 @@ +--- +title: Proxy Overview +description: How TRAWL's general HTTP/HTTPS proxy forwards traffic and escalates detected challenges. +--- + +# Proxy overview + +TRAWL can expose a general HTTP/HTTPS forward proxy, normally on port `8192`. It behaves like a +direct proxy for ordinary traffic and invokes the existing `/scrape` tier pipeline only when a +small buffered response looks like a supported challenge. + +The proxy is disabled by default. It is intended for trusted clients on localhost, a private LAN, +or a private container network. + +## Traffic flow + +```text +Client + │ + ├─ HTTP request ─────────────────────────────┐ + │ │ + └─ HTTPS CONNECT → local TLS termination ────┤ + ▼ + Tier 0 direct forward + │ + ┌────────────────────────┼────────────────────────┐ + │ │ │ + normal small response challenge detected large/media response + │ │ │ + buffer + return run scrape tiers stream + │ + Tier 1 → Tier 2 → Tier 3 → Tier 4 + │ + return solved HTML or raw bytes +``` + +HTTPS requires local TLS termination so TRAWL can inspect the response body. TRAWL generates a +per-host certificate signed by its own root CA. The client must trust that root certificate. + +## Direct forwarding + +Tier 0 opens a normal TCP or TLS connection to the destination and forwards the request method, +body, and end-to-end headers. It preserves headers such as: + +- `Authorization`, `Cookie`, `User-Agent`, `Referer`, and `Origin` +- custom API headers +- `Range`, `If-Range`, cache validators, and content metadata + +Hop-by-hop headers are rebuilt or removed at the proxy boundary. + +Responses not selected by the streaming policy—including small HTML, JSON, XML, text, and unknown +small payloads—are buffered. Compressed `gzip`, `deflate`, and Brotli responses are decoded only +for challenge inspection; the original bytes remain the response body. Chunked buffered responses +are de-chunked before being returned. + +## Challenge escalation + +When Tier 0 detects a supported challenge wall, the request is passed to the same `scrape()` +orchestrator used by `POST /scrape`: + +1. plain HTTP fetch; +2. cached browser session; +3. fresh browser challenge solve; +4. residential proxy solve, when configured. + +For browser-tier HTML responses, the proxy returns the rendered solved DOM rather than the original +challenge response. Binary responses use the raw response bytes when the winning tier exposes them. + +A hostname that produced a Cloudflare challenge is cached for five minutes. During that window, +later requests for the hostname skip Tier 0 and go directly to the tiered solver. + +Challenge and CAPTCHA solving is best effort. A site can still reject the browser, require user +interaction, bind clearance to an unsupported signal, or change its challenge implementation. + +## Buffering and streaming + +The proxy buffers ordinary responses by default so challenge detection sees the complete body. +It streams when at least one of these applies: + +- `Content-Length` is at least 8 MiB; +- the content type is `video/*` or `audio/*`; +- an unknown-length response has a known binary content type; +- the URL has a known media, archive, installer, disk-image, PDF, or font extension. + +Streaming keeps large files out of TRAWL's memory. Streamed responses bypass body-based challenge +detection, so the policy intentionally targets content that is unlikely to be an HTML challenge. + +`Range` requests are forwarded unchanged. A compliant upstream `206 Partial Content` response, +including `Content-Range` and `Accept-Ranges`, is passed back to the client. If the Range response +is instead a detectable challenge page, it can escalate through the normal solver pipeline. + +## WebSockets + +HTTP and HTTPS WebSocket upgrade requests use a bidirectional byte relay after the upstream +`101 Switching Protocols` response. WebSocket frames are not buffered or interpreted. + +The WebSocket handshake does not escalate through the browser solver. If a WebSocket endpoint +requires a clearance cookie, the client must already have suitable credentials or obtain them +through a preceding solved HTTP flow. + +## Current protocol boundaries + +The current listener is an HTTP/1.1 proxy: + +- one request is served per proxied TLS connection; +- HTTP keep-alive and pipelining are not reused; +- request bodies use `Content-Length`; chunked uploads are not decoded; +- duplicate upstream response headers are represented by the first observed value; +- proxy authentication is not implemented; +- HTTP/2 and HTTP/3 are not terminated between the client and TRAWL. + +On a challenged request, the browser fallback converts a request body to UTF-8 because the public +scrape contract currently accepts a string body. Ordinary direct forwarding preserves binary +request bodies exactly. + +## Proxy versus API endpoints + +| Interface | Best for | Response handling | +| ------------------ | -------------------------------------------- | ------------------------------------------------------- | +| `POST /v1` | FlareSolverr-compatible integrations | Returns HTML, cookies, and user agent | +| `POST /scrape` | Native programmatic scraping | Returns tier metadata and solved content | +| HTTP proxy `:8192` | Applications that perform their own requests | Forwards HTTP traffic and escalates detected challenges | + +Use `/v1` or `/scrape` when the application explicitly supports a solver API. Use the proxy when the +application only knows how to make ordinary HTTP requests or when cookie handoff is not sufficient.