mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
docs: update API, architecture, proxy, deployment, and integration guides
This commit is contained in:
germondai
@@ -5,7 +5,8 @@ description: Pass custom HTTP headers through TRAWL to the target URL across all
|
||||
|
||||
# Custom Headers
|
||||
|
||||
Both `/v1` and `/scrape` accept an optional `headers` object. Headers are forwarded to the target URL across all four execution tiers.
|
||||
Both `/v1` and `/scrape` accept an optional `headers` object. Allowed headers are forwarded to the
|
||||
target URL across all four execution tiers.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -33,7 +34,21 @@ Both `/v1` and `/scrape` accept an optional `headers` object. Headers are forwar
|
||||
}
|
||||
```
|
||||
|
||||
Custom headers are merged **after** browser defaults, so they take precedence over anything like `User-Agent` or `Cache-Control` that TRAWL sets internally.
|
||||
Custom headers are merged after browser defaults. Security-sensitive routing and fingerprint headers
|
||||
are removed at the public API boundary rather than overridden.
|
||||
|
||||
## API header policy
|
||||
|
||||
The JSON APIs accept application headers such as `Accept`, `Cache-Control`, `Content-Type`,
|
||||
`Origin`, `Referer`, `Range`, validators, and custom API-key headers. They intentionally discard:
|
||||
|
||||
- `Authorization`, `Cookie`, `Proxy-Authorization`, and `User-Agent`;
|
||||
- `Host`, `Content-Length`, connection/framing headers, and forwarding headers;
|
||||
- browser-controlled `Sec-Fetch-*`, `Sec-CH-UA-*`, and Cloudflare routing headers.
|
||||
|
||||
If an integration needs transparent forwarding of authentication cookies, authorization, or its own
|
||||
user agent, use the [HTTP/HTTPS proxy](/proxy/overview). Its trusted proxy path preserves end-to-end
|
||||
headers and strips only hop-by-hop connection headers.
|
||||
|
||||
## How headers are applied per tier
|
||||
|
||||
@@ -41,20 +56,22 @@ Custom headers are merged **after** browser defaults, so they take precedence ov
|
||||
| ----------------------------------------- | ----------------------------------- | -------------------------------- |
|
||||
| **Tier 1** — plain HTTP fetch | Spread into `fetch()` headers | All requests (there is only one) |
|
||||
| **Tier 2** — cached browser session | `page.route(url, ...)` interception | Main document request only |
|
||||
| **Tier 3** — fresh CF challenge solve | `page.route(url, ...)` interception | Main document request only |
|
||||
| **Tier 3** — fresh challenge solve | `page.route(url, ...)` interception | Main document request only |
|
||||
| **Tier 4** — residential proxy escalation | `page.route(url, ...)` interception | Main document request only |
|
||||
|
||||
For browser tiers, route interception is scoped to the **exact target URL**. Subresources (JS, CSS, images, fonts, third-party CDNs) and Cloudflare challenge endpoints (`cdn-cgi/*`) are never intercepted — your `Authorization` header does not leak to third parties, and CF challenge solving is unaffected.
|
||||
For browser tiers, route interception is scoped to the **exact target URL**. Subresources (JS, CSS,
|
||||
images, fonts, third-party CDNs) and provider challenge endpoints are not given the caller's custom
|
||||
headers.
|
||||
|
||||
## CF challenge + custom headers flow
|
||||
## Challenge + custom headers flow
|
||||
|
||||
When a page requires both challenge bypass and custom headers, the sequence is:
|
||||
|
||||
```
|
||||
1. page.goto(url) — route fires, custom headers added to initial request
|
||||
2. CF intercepts → serves JS challenge interstitial
|
||||
3. Challenge scripts run on cdn-cgi/* paths → route never fires, CF sees clean requests
|
||||
4. cf_clearance cookie issued → browser redirects back to original url
|
||||
2. The WAF serves a challenge interstitial
|
||||
3. Provider scripts run on their own endpoints without caller headers
|
||||
4. The browser completes the supported challenge flow and returns to the target
|
||||
5. route fires again → custom headers applied to the real page load ✓
|
||||
```
|
||||
|
||||
@@ -62,16 +79,12 @@ When a page requires both challenge bypass and custom headers, the sequence is:
|
||||
|
||||
| Use case | Header |
|
||||
| -------------------------------------- | ---------------------------------- |
|
||||
| Authenticated APIs and portals | `Authorization: Bearer <token>` |
|
||||
| Embed-only / iframe-restricted content | `Referer: https://parent-site.com` |
|
||||
| CORS-restricted endpoints | `Origin: https://allowed-site.com` |
|
||||
| Custom API keys | `X-API-Key: <key>` |
|
||||
| Additional session tokens | `Cookie: session=<value>` |
|
||||
| Conditional or partial requests | `If-None-Match`, `Range` |
|
||||
|
||||
::: tip Cookie behaviour
|
||||
Passing a `Cookie` header appends to any cookies the browser already holds (CF clearance, cached session cookies). It does not replace them.
|
||||
:::
|
||||
|
||||
::: warning Headers and CF-protected pages
|
||||
Pages that require custom auth headers are rarely also behind CF JS challenges — CF challenges are for public sites needing bot/DDoS protection, while auth headers imply a private/restricted resource. If you hit both, TRAWL handles it correctly as described above.
|
||||
::: warning Authentication headers
|
||||
`Authorization`, `Cookie`, and caller-controlled `User-Agent` values are not accepted by `/v1` or
|
||||
`/scrape`. Configure TRAWL as a forward proxy when those headers must pass through unchanged.
|
||||
:::
|
||||
|
||||
@@ -18,7 +18,7 @@ FlareSolverr-style readiness message — confirms the API process is up (does no
|
||||
```json
|
||||
{
|
||||
"msg": "TRAWL is ready!",
|
||||
"version": "1.2.0",
|
||||
"version": "1.3.0",
|
||||
"uptime": 42
|
||||
}
|
||||
```
|
||||
|
||||
@@ -34,6 +34,17 @@ Content-Type: application/json
|
||||
| `POST` | `/v1` | FlareSolverr v2 compatible |
|
||||
| `POST` | `/scrape` | Native TRAWL API |
|
||||
|
||||
## Forward proxy
|
||||
|
||||
When `MITM_PROXY_ENABLED=true`, TRAWL also listens as an HTTP/HTTPS forward proxy on
|
||||
`MITM_PROXY_PORT` (default `8192`). This is a socket-level proxy interface rather than a JSON API
|
||||
endpoint. It forwards normal traffic directly and escalates recognized challenge walls through the
|
||||
same tier engine as `/scrape`.
|
||||
|
||||
HTTPS clients must trust the generated TRAWL CA. Start with the
|
||||
[proxy overview](/proxy/overview), then follow [client setup](/proxy/client-setup) and
|
||||
[CA installation](/proxy/ca-installation).
|
||||
|
||||
## Error responses
|
||||
|
||||
Most error responses follow this shape:
|
||||
|
||||
@@ -1,21 +1,21 @@
|
||||
---
|
||||
title: Architecture Overview
|
||||
description: How TRAWL's components fit together — embedded browser pool, session cache, and tiered execution.
|
||||
description: How TRAWL's API, forward proxy, browser pool, session cache, and tiered challenge engine fit together.
|
||||
---
|
||||
|
||||
# Architecture Overview
|
||||
|
||||
```
|
||||
Client (Prowlarr, curl, your code)
|
||||
Client (Prowlarr, curl, browser, your code)
|
||||
│
|
||||
▼
|
||||
Elysia API (apps/api)
|
||||
┌──────────────────────────┐
|
||||
│ POST /v1 POST /scrape │
|
||||
│ GET /health GET /stats │
|
||||
└────────────┬─────────────┘
|
||||
│ direct call
|
||||
▼
|
||||
┌────┴───────────────┐
|
||||
▼ ▼
|
||||
Elysia API HTTP/HTTPS proxy :8192
|
||||
POST /v1 Tier 0 direct forwarding
|
||||
POST /scrape + challenge detection
|
||||
GET /health │ detected wall
|
||||
└──────────────┬──────┘
|
||||
▼
|
||||
Orchestrator (packages/tiers)
|
||||
┌──────────────────────────────────────┐
|
||||
│ ├── Tier 1: plain Bun fetch │
|
||||
@@ -39,6 +39,12 @@ Client (Prowlarr, curl, your code)
|
||||
|
||||
An Elysia HTTP server. Accepts scrape requests and calls the orchestrator inline — all browser work happens in the same process. Exposes `/` for a FlareSolverr-style readiness message, plus `/health` and `/stats` for monitoring. Routes live under `apps/api/src/routes/`, with shared config/pool state in `config.ts`/`deps.ts`.
|
||||
|
||||
### Forward Proxy (`apps/api/src/proxy`)
|
||||
|
||||
An optional general HTTP/HTTPS proxy on port `8192`. Ordinary requests use direct TCP/TLS forwarding, preserving methods, bodies, authentication headers, redirects, WebSocket upgrades, binary content, and HTTP Range semantics. Small textual responses are buffered for challenge detection; media and large files can stream directly. When a response is identified as a challenge wall, the proxy sends the request through the same four-tier orchestrator used by `/scrape`.
|
||||
|
||||
HTTPS inspection requires clients to trust TRAWL's generated root CA. See [Proxy overview](/proxy/overview) and [CA installation](/proxy/ca-installation).
|
||||
|
||||
### Browser Pool (`packages/browser/src/pool.ts`)
|
||||
|
||||
Maintains a fixed set of `{ browser, context }` pairs using [Camoufox](https://github.com/daijro/camoufox) (Firefox with fingerprint patching at the C++/Juggler level). Acquisition is sticky — if a browser last served `example.com`, it is preferred for the next request to `example.com`. Browsers accumulate domain cookies across requests, which makes subsequent challenges faster.
|
||||
@@ -51,4 +57,4 @@ Redis is optional — if `REDIS_URL` is not set, the session cache is disabled a
|
||||
|
||||
### Tiers (`packages/tiers`)
|
||||
|
||||
The escalation logic. See [Tiered Execution](/architecture/tiered-execution) for the full breakdown.
|
||||
The escalation logic and challenge-specific browser waits. Cloudflare, Akamai Bot Manager, and Imperva/Incapsula walls are detected separately so Tier 3 and Tier 4 can run the appropriate resolution flow. Embedded Turnstile, reCAPTCHA v2, hCaptcha, and GeeTest slide widgets are attempted after the protected page loads. See [Tiered Execution](/architecture/tiered-execution) for the full breakdown.
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
---
|
||||
title: Session Cache
|
||||
description: How TRAWL caches Cloudflare cookies in Redis to make repeat requests fast.
|
||||
description: How TRAWL caches solved browser sessions in Redis to avoid unnecessary challenge work.
|
||||
---
|
||||
|
||||
# Session Cache
|
||||
|
||||
The session cache is what makes Tier 2 possible. After every successful Tier 3 solve, the extracted Cloudflare cookies are saved to Redis. The next request to the same domain injects those cookies into a browser context, skipping the challenge entirely.
|
||||
The session cache is what makes Tier 2 possible. After a successful Tier 3 or Tier 4 solve, TRAWL
|
||||
saves the extracted cookies and browser user agent in Redis. The next request to the same hostname
|
||||
injects that state into a browser context and attempts to reuse the solved session.
|
||||
|
||||
## Storage format
|
||||
|
||||
@@ -32,7 +34,7 @@ https://example.com/page → session:example.com (same key)
|
||||
https://sub.example.com/ → session:sub.example.com (different key)
|
||||
```
|
||||
|
||||
Subdomains have separate sessions because Cloudflare can issue different challenge cookies per subdomain.
|
||||
Subdomains have separate sessions because WAF and application cookies can differ per subdomain.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
@@ -44,18 +46,18 @@ Tier 3 succeeds
|
||||
│
|
||||
└── next request to same domain:
|
||||
REDIS GET session:hostname
|
||||
├── hit → Tier 2: inject cookies, navigate (500ms)
|
||||
├── hit → Tier 2: inject cookies and navigate
|
||||
└── miss → Tier 3: fresh solve, save to cache
|
||||
```
|
||||
|
||||
## Invalidation
|
||||
|
||||
If Tier 2 navigates with the cached cookies and the result is still a Cloudflare interstitial (the session expired before Redis's TTL), the orchestrator:
|
||||
If Tier 2 navigates with cached state and still receives a recognized challenge wall, the orchestrator:
|
||||
|
||||
1. Calls `sessionCache.invalidate(domain)` — deletes the Redis key
|
||||
2. Escalates to Tier 3 to get a fresh session
|
||||
|
||||
This handles the case where Cloudflare's `cf_clearance` cookie (30-minute expiry) expires before the Redis TTL does.
|
||||
This handles provider cookies expiring or being rejected before the Redis TTL ends.
|
||||
|
||||
## Redis
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ Tier 1: Plain HTTP Fetch ─── success ──→ return (< 100ms)
|
||||
Tier 2: Cached Session ────── success ──→ return (~500ms)
|
||||
│ blocked / cache miss
|
||||
▼
|
||||
Tier 3: Fresh Challenge ───── success ──→ cache cookies, return (4–15s)
|
||||
Tier 3: Fresh Challenge ───── success ──→ cache cookies, return
|
||||
│ IP flagged
|
||||
▼
|
||||
Tier 4: Residential Proxy ─── success ──→ cache cookies, return (15–45s)
|
||||
@@ -37,34 +37,40 @@ Accept-Language: en-US,en;q=0.9
|
||||
Accept-Encoding: gzip, deflate, br
|
||||
```
|
||||
|
||||
**Succeeds for:** sites without Cloudflare, sites that serve HTML to plain HTTP.
|
||||
**Succeeds for:** sites that serve the requested content without a browser challenge.
|
||||
|
||||
**Fails for:** sites behind Cloudflare JS challenge (returns `Checking your browser...` interstitial or HTTP 403/429). The detector checks for the `cf-mitigated` response header and the challenge page title/body.
|
||||
**Escalates for:** recognized Cloudflare, Akamai, or Imperva challenge responses and blocked status codes such as 403 or 429. Detection uses provider-specific headers and HTML markers.
|
||||
|
||||
**Skip with:** `skipHttp: true` in the request body, or `maxTier: 1` to cap at Tier 1.
|
||||
|
||||
## Tier 2 — Cached Browser Session
|
||||
|
||||
Reads `session:{hostname}` from Redis. If found, injects the saved cookies into a pooled Firefox context and navigates. Because the Cloudflare `cf_clearance` cookie is present, the challenge page is skipped and the site loads directly.
|
||||
Reads `session:{hostname}` from Redis. If found, injects the saved cookies into a pooled Firefox context and navigates. When the target accepts the cached protection cookies and browser identity, the site loads without a fresh challenge solve.
|
||||
|
||||
**Succeeds for:** any domain that was previously solved by Tier 3 and whose session hasn't expired.
|
||||
**Succeeds for:** previously solved domains whose cached session is still accepted.
|
||||
|
||||
**Fails for:** expired sessions (Cloudflare `cf_clearance` has a finite lifetime). On failure, TRAWL invalidates the cached session and escalates to Tier 3.
|
||||
**Fails for:** expired or rejected sessions. On failure, TRAWL invalidates the cached session and escalates to Tier 3.
|
||||
|
||||
## Tier 3 — Fresh Cloudflare Challenge Solve
|
||||
## Tier 3 — Fresh Challenge Solve
|
||||
|
||||
Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS` — default 15s — for one to become available). Navigates to the URL with no pre-loaded cookies. Waits for the Cloudflare challenge to resolve by polling `page.content()` every 500ms until the interstitial HTML is gone or `maxTimeout` elapses.
|
||||
Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS` — default 15s — for one to become available), creates a fresh Camoufox context, and navigates without preloaded cookies. TRAWL identifies the wall and runs the matching Cloudflare, Akamai, or Imperva wait flow until the protected page replaces the interstitial or `maxTimeout` elapses.
|
||||
|
||||
On success:
|
||||
- Extracts all cookies from the page context
|
||||
- Writes `session:{hostname} → { cookies, userAgent, savedAt }` to Redis (TTL = `SESSION_TTL_SECONDS`)
|
||||
- Returns the HTML and cookies to the caller
|
||||
|
||||
Uses [Camoufox](https://github.com/daijro/camoufox) — Firefox with fingerprint patching at the C++/Juggler level. CF's detection scripts cannot distinguish it from a real Firefox profile.
|
||||
Uses [Camoufox](https://github.com/daijro/camoufox) — Firefox with fingerprint patching at the C++/Juggler level to reduce common automation signals. Success still depends on the target's challenge variant, IP reputation, and upstream network conditions.
|
||||
|
||||
### Akamai Bot Manager challenges
|
||||
|
||||
Tier 3 and Tier 4 detect Akamai's `sec-cpt` / SBSD behavioral interstitials. The Akamai flow generates human-like pointer movement, handles supported press-and-hold widgets, waits for a valid `_abck` sensor cookie, and revisits the original URL when the interstitial does not reload automatically.
|
||||
|
||||
Akamai configurations vary between properties and change over time. TRAWL treats a persistent interstitial as blocked and can escalate to Tier 4 when a residential proxy is configured.
|
||||
|
||||
### Imperva/Incapsula challenges
|
||||
|
||||
Tier 3 and Tier 4 also detect and solve Imperva/Incapsula WAF challenges, not just Cloudflare. Imperva's `reese84` (current) / `___utmvc` (legacy) sensor cookies are produced by an obfuscated in-page JS challenge — same principle as Cloudflare's `cf_clearance`: a real browser executing real JS produces the cookie without needing to understand the obfuscation. TRAWL detects the challenge page (`packages/tiers/src/detect.ts`'s `hasImpervaChallenge`) and polls for the sensor cookie (`packages/tiers/src/impervaWait.ts`) instead of the Cloudflare-specific wait loop.
|
||||
Tier 3 and Tier 4 also detect and resolve supported Imperva/Incapsula WAF challenges. Imperva's `reese84` (current) / `___utmvc` (legacy) sensor cookies are produced by an obfuscated in-page JS challenge. TRAWL detects the response with `packages/tiers/src/utils/detect.ts` and waits for the sensor cookie through `packages/tiers/src/utils/impervaWait.ts`.
|
||||
|
||||
**Caveat:** unlike Turnstile, Imperva's script sometimes layers in TLS/JA3 and behavioral checks beyond plain cookie generation, and its obfuscation changes periodically — success isn't guaranteed at the same rate as Cloudflare. Some Imperva deployments also show a visible interactive CAPTCHA widget (distinct from hCaptcha/reCAPTCHA) instead of the passive sensor-only path; that variant isn't solved yet.
|
||||
|
||||
@@ -93,5 +99,5 @@ This runs Tier 1, then Tier 2, then returns an error if both fail — never laun
|
||||
| ---- | ------------ | ------------------------- |
|
||||
| 1 | 50–150ms | No |
|
||||
| 2 | 400–700ms | Yes (warm) |
|
||||
| 3 | 4–15s | Yes (fresh solve) |
|
||||
| 3 | Challenge-dependent | Yes (fresh solve) |
|
||||
| 4 | 15–45s | Yes (fresh solve + proxy) |
|
||||
|
||||
@@ -19,7 +19,8 @@ docker compose -f docker-compose.minimal.yml up -d
|
||||
|
||||
### Cached (default)
|
||||
|
||||
`docker-compose.yml` — scraper + Redis session cache. Repeat requests to the same domain return in ~500ms.
|
||||
`docker-compose.yml` — scraper + Redis session cache, with API port `8191` and optional proxy port
|
||||
`8192`. Repeat requests can reuse accepted sessions instead of solving the challenge again.
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
@@ -38,9 +39,9 @@ trawl:
|
||||
restart: always
|
||||
mem_limit: 3g
|
||||
environment:
|
||||
BROWSER_POOL_SIZE: 5
|
||||
BROWSER_POOL_SIZE: 3
|
||||
healthcheck:
|
||||
test: wget -qO- http://localhost:8191/health
|
||||
test: ["CMD", "curl", "-sf", "http://localhost:8191/health"]
|
||||
interval: 30s
|
||||
```
|
||||
|
||||
@@ -69,7 +70,7 @@ See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone
|
||||
|
||||
| Variable | Default | Description |
|
||||
| -------------------------------- | -------------------- | ----------------------------------------------------------------------- |
|
||||
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
|
||||
| `BROWSER_POOL_SIZE` | `3` | Warm browsers; supplied minimal/cached Compose files override this to `1` |
|
||||
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
|
||||
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart after this many blocked/needs-js outcomes; set `0` to disable |
|
||||
| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) |
|
||||
|
||||
@@ -101,7 +101,8 @@ These bounds keep an unresponsive Firefox process from permanently consuming a p
|
||||
|
||||
**Default:** `3600` (1 hour)
|
||||
|
||||
How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
|
||||
How long solved browser cookies and user-agent state are cached in Redis per domain. After this TTL
|
||||
the next protected request triggers a fresh challenge solve (Tier 3) and refreshes the cache.
|
||||
|
||||
Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3).
|
||||
|
||||
|
||||
@@ -28,15 +28,14 @@ Everything has a working default. See [Configuration](/getting-started/configura
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This starts three containers:
|
||||
This starts two containers:
|
||||
|
||||
| Container | Purpose | Port |
|
||||
| --------- | ----------------------- | -------- |
|
||||
| `redis` | Session cache backend | internal |
|
||||
| `api` | Browser pool + HTTP API | 8191 |
|
||||
| `web` | Landing page & docs UI | 3000 |
|
||||
| `trawl` | Browser pool, API, and optional forward proxy | 8191, 8192 |
|
||||
|
||||
The API takes **15–30 seconds** on first boot while it launches the browser pool and downloads Camoufox (only on first build). Watch progress:
|
||||
The API takes **15–30 seconds** to launch and warm the browser pool. Watch progress:
|
||||
|
||||
```bash
|
||||
docker compose logs -f api
|
||||
@@ -69,7 +68,9 @@ Expected:
|
||||
"busy": 0,
|
||||
"available": 3,
|
||||
"restarts": 0,
|
||||
"avgRestarts": 0
|
||||
"avgRestarts": 0,
|
||||
"stalled": 0,
|
||||
"live": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -86,10 +87,13 @@ curl -s -X POST http://localhost:8191/v1 \
|
||||
}' | jq '{status, url: .solution.url}'
|
||||
```
|
||||
|
||||
A `status: "ok"` response confirms TRAWL is working. The **second request to the same domain** returns in ~500ms from the session cache.
|
||||
A `status: "ok"` response confirms TRAWL is working. Repeat requests can reuse the cached browser
|
||||
session when the target still accepts it.
|
||||
|
||||
::: tip First request is slow — that's expected
|
||||
The first time TRAWL sees a domain it solves the Cloudflare challenge fresh (4–15s). Every subsequent request uses the cached session and returns in under 500ms. This is the whole point.
|
||||
::: tip Challenge requests take longer
|
||||
An unprotected target may finish in Tier 1 without opening a browser. A recognized Cloudflare,
|
||||
Akamai, or Imperva wall escalates to a fresh browser flow. Later requests can use Tier 2 while the
|
||||
saved session remains valid.
|
||||
:::
|
||||
|
||||
## 6. Use the native API (optional)
|
||||
@@ -118,8 +122,7 @@ Done. No other configuration changes are needed. TRAWL implements the FlareSolve
|
||||
|
||||
```
|
||||
localhost:8191 → TRAWL API (FlareSolverr-compatible + native endpoint)
|
||||
localhost:3000 → Web UI (landing page, visible in your browser)
|
||||
localhost:3001 → Docs (this documentation, served locally)
|
||||
localhost:8192 → HTTP/HTTPS forward proxy (when MITM_PROXY_ENABLED=true)
|
||||
```
|
||||
|
||||
To stop everything:
|
||||
|
||||
@@ -5,7 +5,9 @@ description: Connect Sonarr, Radarr, Lidarr, Readarr and other *arr apps via Pro
|
||||
|
||||
# *arr Apps
|
||||
|
||||
Sonarr, Radarr, Lidarr, and Readarr do not talk to FlareSolverr directly. They go through **Prowlarr** (recommended) or **Jackett** as an indexer proxy. You only need to configure TRAWL once in the indexer manager — all *arr apps that use that manager get Cloudflare bypass automatically.
|
||||
Sonarr, Radarr, Lidarr, and Readarr do not talk to FlareSolverr directly. They go through
|
||||
**Prowlarr** (recommended) or **Jackett** as an indexer proxy. Configure TRAWL once in the indexer
|
||||
manager and the connected *arr apps can use its challenge handling.
|
||||
|
||||
## Recommended setup
|
||||
|
||||
@@ -47,11 +49,12 @@ Bazarr uses subtitle providers, not torrent indexers, so it does not use FlareSo
|
||||
|
||||
## Performance expectations
|
||||
|
||||
| Request type | Expected time |
|
||||
| ------------------------------------------------ | ------------------------------ |
|
||||
| First request to a domain | 4–15s (fresh Cloudflare solve) |
|
||||
| Repeat request (same domain, session cached) | ~500ms |
|
||||
| Plain site (no Cloudflare) | < 100ms |
|
||||
| IP flagged by Cloudflare (Tier 4, if configured) | 15–45s |
|
||||
| Request type | Expected path |
|
||||
| -------------------------------------------- | ------------------------------------------ |
|
||||
| Unprotected response | Tier 1, no browser |
|
||||
| Recognized WAF challenge | Tier 3 fresh browser solve |
|
||||
| Repeat request with an accepted session | Tier 2 cached browser session |
|
||||
| Direct/datacenter path rejected | Tier 4, when a residential proxy is set |
|
||||
|
||||
The session cache TTL is configurable via `SESSION_TTL_SECONDS` (default 1 hour). Most Cloudflare-protected indexers re-challenge after 30–60 minutes, so daily search schedules in *arr apps almost always hit the cache.
|
||||
The session cache TTL is configurable via `SESSION_TTL_SECONDS` (default 1 hour). Actual latency
|
||||
depends on the indexer, challenge variant, IP reputation, and whether its saved session remains valid.
|
||||
|
||||
@@ -33,7 +33,9 @@ Restart Jackett after editing the file.
|
||||
|
||||
## Verify
|
||||
|
||||
Navigate to any Cloudflare-protected indexer in Jackett and click **Test**. The first test triggers a full challenge solve (~10–30s). Subsequent tests on the same domain return in ~500ms from the TRAWL session cache.
|
||||
Navigate to a protected indexer in Jackett and click **Test**. A first request that encounters a
|
||||
recognized wall triggers the browser challenge flow. Subsequent tests can reuse the cached session
|
||||
while it remains valid.
|
||||
|
||||
## Docker
|
||||
|
||||
|
||||
@@ -28,7 +28,9 @@ TRAWL implements the FlareSolverr v2 API exactly, including the `version: "2.0.0
|
||||
|
||||
## Verification
|
||||
|
||||
After saving, trigger a search on any Cloudflare-protected indexer. TRAWL will handle the challenge transparently. Subsequent searches to the same indexer domain return from cache in ~500ms.
|
||||
After saving, trigger a search on a protected indexer. TRAWL handles recognized Cloudflare, Akamai,
|
||||
and Imperva walls through its tier engine. Later searches can reuse the saved session while the
|
||||
target continues to accept it.
|
||||
|
||||
## Prowlarr inside Docker
|
||||
|
||||
|
||||
@@ -66,7 +66,7 @@ orchestrator used by `POST /scrape`:
|
||||
For browser-tier HTML responses, the proxy returns the rendered solved DOM rather than the original
|
||||
challenge response. Binary responses use the raw response bytes when the winning tier exposes them.
|
||||
|
||||
A hostname that produced a Cloudflare challenge is cached for five minutes. During that window,
|
||||
A hostname that produced a supported challenge wall is cached for five minutes. During that window,
|
||||
later requests for the hostname skip Tier 0 and go directly to the tiered solver.
|
||||
|
||||
Challenge and CAPTCHA solving is best effort. A site can still reject the browser, require user
|
||||
|
||||
Reference in New Issue
Block a user