diff --git a/apps/docs/api-reference/custom-headers.md b/apps/docs/api-reference/custom-headers.md index ba81e92..6ef3d1b 100644 --- a/apps/docs/api-reference/custom-headers.md +++ b/apps/docs/api-reference/custom-headers.md @@ -5,7 +5,8 @@ description: Pass custom HTTP headers through TRAWL to the target URL across all # Custom Headers -Both `/v1` and `/scrape` accept an optional `headers` object. Headers are forwarded to the target URL across all four execution tiers. +Both `/v1` and `/scrape` accept an optional `headers` object. Allowed headers are forwarded to the +target URL across all four execution tiers. ## Usage @@ -33,7 +34,21 @@ Both `/v1` and `/scrape` accept an optional `headers` object. Headers are forwar } ``` -Custom headers are merged **after** browser defaults, so they take precedence over anything like `User-Agent` or `Cache-Control` that TRAWL sets internally. +Custom headers are merged after browser defaults. Security-sensitive routing and fingerprint headers +are removed at the public API boundary rather than overridden. + +## API header policy + +The JSON APIs accept application headers such as `Accept`, `Cache-Control`, `Content-Type`, +`Origin`, `Referer`, `Range`, validators, and custom API-key headers. They intentionally discard: + +- `Authorization`, `Cookie`, `Proxy-Authorization`, and `User-Agent`; +- `Host`, `Content-Length`, connection/framing headers, and forwarding headers; +- browser-controlled `Sec-Fetch-*`, `Sec-CH-UA-*`, and Cloudflare routing headers. + +If an integration needs transparent forwarding of authentication cookies, authorization, or its own +user agent, use the [HTTP/HTTPS proxy](/proxy/overview). Its trusted proxy path preserves end-to-end +headers and strips only hop-by-hop connection headers. ## How headers are applied per tier @@ -41,20 +56,22 @@ Custom headers are merged **after** browser defaults, so they take precedence ov | ----------------------------------------- | ----------------------------------- | -------------------------------- | | **Tier 1** — plain HTTP fetch | Spread into `fetch()` headers | All requests (there is only one) | | **Tier 2** — cached browser session | `page.route(url, ...)` interception | Main document request only | -| **Tier 3** — fresh CF challenge solve | `page.route(url, ...)` interception | Main document request only | +| **Tier 3** — fresh challenge solve | `page.route(url, ...)` interception | Main document request only | | **Tier 4** — residential proxy escalation | `page.route(url, ...)` interception | Main document request only | -For browser tiers, route interception is scoped to the **exact target URL**. Subresources (JS, CSS, images, fonts, third-party CDNs) and Cloudflare challenge endpoints (`cdn-cgi/*`) are never intercepted — your `Authorization` header does not leak to third parties, and CF challenge solving is unaffected. +For browser tiers, route interception is scoped to the **exact target URL**. Subresources (JS, CSS, +images, fonts, third-party CDNs) and provider challenge endpoints are not given the caller's custom +headers. -## CF challenge + custom headers flow +## Challenge + custom headers flow When a page requires both challenge bypass and custom headers, the sequence is: ``` 1. page.goto(url) — route fires, custom headers added to initial request -2. CF intercepts → serves JS challenge interstitial -3. Challenge scripts run on cdn-cgi/* paths → route never fires, CF sees clean requests -4. cf_clearance cookie issued → browser redirects back to original url +2. The WAF serves a challenge interstitial +3. Provider scripts run on their own endpoints without caller headers +4. The browser completes the supported challenge flow and returns to the target 5. route fires again → custom headers applied to the real page load ✓ ``` @@ -62,16 +79,12 @@ When a page requires both challenge bypass and custom headers, the sequence is: | Use case | Header | | -------------------------------------- | ---------------------------------- | -| Authenticated APIs and portals | `Authorization: Bearer ` | | Embed-only / iframe-restricted content | `Referer: https://parent-site.com` | | CORS-restricted endpoints | `Origin: https://allowed-site.com` | | Custom API keys | `X-API-Key: ` | -| Additional session tokens | `Cookie: session=` | +| Conditional or partial requests | `If-None-Match`, `Range` | -::: tip Cookie behaviour -Passing a `Cookie` header appends to any cookies the browser already holds (CF clearance, cached session cookies). It does not replace them. -::: - -::: warning Headers and CF-protected pages -Pages that require custom auth headers are rarely also behind CF JS challenges — CF challenges are for public sites needing bot/DDoS protection, while auth headers imply a private/restricted resource. If you hit both, TRAWL handles it correctly as described above. +::: warning Authentication headers +`Authorization`, `Cookie`, and caller-controlled `User-Agent` values are not accepted by `/v1` or +`/scrape`. Configure TRAWL as a forward proxy when those headers must pass through unchanged. ::: diff --git a/apps/docs/api-reference/health-stats.md b/apps/docs/api-reference/health-stats.md index 34f2586..f82b209 100644 --- a/apps/docs/api-reference/health-stats.md +++ b/apps/docs/api-reference/health-stats.md @@ -18,7 +18,7 @@ FlareSolverr-style readiness message — confirms the API process is up (does no ```json { "msg": "TRAWL is ready!", - "version": "1.2.0", + "version": "1.3.0", "uptime": 42 } ``` diff --git a/apps/docs/api-reference/overview.md b/apps/docs/api-reference/overview.md index e55e1bc..3864d83 100644 --- a/apps/docs/api-reference/overview.md +++ b/apps/docs/api-reference/overview.md @@ -34,6 +34,17 @@ Content-Type: application/json | `POST` | `/v1` | FlareSolverr v2 compatible | | `POST` | `/scrape` | Native TRAWL API | +## Forward proxy + +When `MITM_PROXY_ENABLED=true`, TRAWL also listens as an HTTP/HTTPS forward proxy on +`MITM_PROXY_PORT` (default `8192`). This is a socket-level proxy interface rather than a JSON API +endpoint. It forwards normal traffic directly and escalates recognized challenge walls through the +same tier engine as `/scrape`. + +HTTPS clients must trust the generated TRAWL CA. Start with the +[proxy overview](/proxy/overview), then follow [client setup](/proxy/client-setup) and +[CA installation](/proxy/ca-installation). + ## Error responses Most error responses follow this shape: diff --git a/apps/docs/architecture/overview.md b/apps/docs/architecture/overview.md index 247b985..fb1184c 100644 --- a/apps/docs/architecture/overview.md +++ b/apps/docs/architecture/overview.md @@ -1,21 +1,21 @@ --- title: Architecture Overview -description: How TRAWL's components fit together — embedded browser pool, session cache, and tiered execution. +description: How TRAWL's API, forward proxy, browser pool, session cache, and tiered challenge engine fit together. --- # Architecture Overview ``` -Client (Prowlarr, curl, your code) +Client (Prowlarr, curl, browser, your code) │ - ▼ - Elysia API (apps/api) - ┌──────────────────────────┐ - │ POST /v1 POST /scrape │ - │ GET /health GET /stats │ - └────────────┬─────────────┘ - │ direct call - ▼ + ┌────┴───────────────┐ + ▼ ▼ + Elysia API HTTP/HTTPS proxy :8192 + POST /v1 Tier 0 direct forwarding + POST /scrape + challenge detection + GET /health │ detected wall + └──────────────┬──────┘ + ▼ Orchestrator (packages/tiers) ┌──────────────────────────────────────┐ │ ├── Tier 1: plain Bun fetch │ @@ -39,6 +39,12 @@ Client (Prowlarr, curl, your code) An Elysia HTTP server. Accepts scrape requests and calls the orchestrator inline — all browser work happens in the same process. Exposes `/` for a FlareSolverr-style readiness message, plus `/health` and `/stats` for monitoring. Routes live under `apps/api/src/routes/`, with shared config/pool state in `config.ts`/`deps.ts`. +### Forward Proxy (`apps/api/src/proxy`) + +An optional general HTTP/HTTPS proxy on port `8192`. Ordinary requests use direct TCP/TLS forwarding, preserving methods, bodies, authentication headers, redirects, WebSocket upgrades, binary content, and HTTP Range semantics. Small textual responses are buffered for challenge detection; media and large files can stream directly. When a response is identified as a challenge wall, the proxy sends the request through the same four-tier orchestrator used by `/scrape`. + +HTTPS inspection requires clients to trust TRAWL's generated root CA. See [Proxy overview](/proxy/overview) and [CA installation](/proxy/ca-installation). + ### Browser Pool (`packages/browser/src/pool.ts`) Maintains a fixed set of `{ browser, context }` pairs using [Camoufox](https://github.com/daijro/camoufox) (Firefox with fingerprint patching at the C++/Juggler level). Acquisition is sticky — if a browser last served `example.com`, it is preferred for the next request to `example.com`. Browsers accumulate domain cookies across requests, which makes subsequent challenges faster. @@ -51,4 +57,4 @@ Redis is optional — if `REDIS_URL` is not set, the session cache is disabled a ### Tiers (`packages/tiers`) -The escalation logic. See [Tiered Execution](/architecture/tiered-execution) for the full breakdown. +The escalation logic and challenge-specific browser waits. Cloudflare, Akamai Bot Manager, and Imperva/Incapsula walls are detected separately so Tier 3 and Tier 4 can run the appropriate resolution flow. Embedded Turnstile, reCAPTCHA v2, hCaptcha, and GeeTest slide widgets are attempted after the protected page loads. See [Tiered Execution](/architecture/tiered-execution) for the full breakdown. diff --git a/apps/docs/architecture/session-cache.md b/apps/docs/architecture/session-cache.md index c839838..e851d43 100644 --- a/apps/docs/architecture/session-cache.md +++ b/apps/docs/architecture/session-cache.md @@ -1,11 +1,13 @@ --- title: Session Cache -description: How TRAWL caches Cloudflare cookies in Redis to make repeat requests fast. +description: How TRAWL caches solved browser sessions in Redis to avoid unnecessary challenge work. --- # Session Cache -The session cache is what makes Tier 2 possible. After every successful Tier 3 solve, the extracted Cloudflare cookies are saved to Redis. The next request to the same domain injects those cookies into a browser context, skipping the challenge entirely. +The session cache is what makes Tier 2 possible. After a successful Tier 3 or Tier 4 solve, TRAWL +saves the extracted cookies and browser user agent in Redis. The next request to the same hostname +injects that state into a browser context and attempts to reuse the solved session. ## Storage format @@ -32,7 +34,7 @@ https://example.com/page → session:example.com (same key) https://sub.example.com/ → session:sub.example.com (different key) ``` -Subdomains have separate sessions because Cloudflare can issue different challenge cookies per subdomain. +Subdomains have separate sessions because WAF and application cookies can differ per subdomain. ## Lifecycle @@ -44,18 +46,18 @@ Tier 3 succeeds │ └── next request to same domain: REDIS GET session:hostname - ├── hit → Tier 2: inject cookies, navigate (500ms) + ├── hit → Tier 2: inject cookies and navigate └── miss → Tier 3: fresh solve, save to cache ``` ## Invalidation -If Tier 2 navigates with the cached cookies and the result is still a Cloudflare interstitial (the session expired before Redis's TTL), the orchestrator: +If Tier 2 navigates with cached state and still receives a recognized challenge wall, the orchestrator: 1. Calls `sessionCache.invalidate(domain)` — deletes the Redis key 2. Escalates to Tier 3 to get a fresh session -This handles the case where Cloudflare's `cf_clearance` cookie (30-minute expiry) expires before the Redis TTL does. +This handles provider cookies expiring or being rejected before the Redis TTL ends. ## Redis diff --git a/apps/docs/architecture/tiered-execution.md b/apps/docs/architecture/tiered-execution.md index d4f1155..aa89511 100644 --- a/apps/docs/architecture/tiered-execution.md +++ b/apps/docs/architecture/tiered-execution.md @@ -17,7 +17,7 @@ Tier 1: Plain HTTP Fetch ─── success ──→ return (< 100ms) Tier 2: Cached Session ────── success ──→ return (~500ms) │ blocked / cache miss ▼ -Tier 3: Fresh Challenge ───── success ──→ cache cookies, return (4–15s) +Tier 3: Fresh Challenge ───── success ──→ cache cookies, return │ IP flagged ▼ Tier 4: Residential Proxy ─── success ──→ cache cookies, return (15–45s) @@ -37,34 +37,40 @@ Accept-Language: en-US,en;q=0.9 Accept-Encoding: gzip, deflate, br ``` -**Succeeds for:** sites without Cloudflare, sites that serve HTML to plain HTTP. +**Succeeds for:** sites that serve the requested content without a browser challenge. -**Fails for:** sites behind Cloudflare JS challenge (returns `Checking your browser...` interstitial or HTTP 403/429). The detector checks for the `cf-mitigated` response header and the challenge page title/body. +**Escalates for:** recognized Cloudflare, Akamai, or Imperva challenge responses and blocked status codes such as 403 or 429. Detection uses provider-specific headers and HTML markers. **Skip with:** `skipHttp: true` in the request body, or `maxTier: 1` to cap at Tier 1. ## Tier 2 — Cached Browser Session -Reads `session:{hostname}` from Redis. If found, injects the saved cookies into a pooled Firefox context and navigates. Because the Cloudflare `cf_clearance` cookie is present, the challenge page is skipped and the site loads directly. +Reads `session:{hostname}` from Redis. If found, injects the saved cookies into a pooled Firefox context and navigates. When the target accepts the cached protection cookies and browser identity, the site loads without a fresh challenge solve. -**Succeeds for:** any domain that was previously solved by Tier 3 and whose session hasn't expired. +**Succeeds for:** previously solved domains whose cached session is still accepted. -**Fails for:** expired sessions (Cloudflare `cf_clearance` has a finite lifetime). On failure, TRAWL invalidates the cached session and escalates to Tier 3. +**Fails for:** expired or rejected sessions. On failure, TRAWL invalidates the cached session and escalates to Tier 3. -## Tier 3 — Fresh Cloudflare Challenge Solve +## Tier 3 — Fresh Challenge Solve -Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS` — default 15s — for one to become available). Navigates to the URL with no pre-loaded cookies. Waits for the Cloudflare challenge to resolve by polling `page.content()` every 500ms until the interstitial HTML is gone or `maxTimeout` elapses. +Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS` — default 15s — for one to become available), creates a fresh Camoufox context, and navigates without preloaded cookies. TRAWL identifies the wall and runs the matching Cloudflare, Akamai, or Imperva wait flow until the protected page replaces the interstitial or `maxTimeout` elapses. On success: - Extracts all cookies from the page context - Writes `session:{hostname} → { cookies, userAgent, savedAt }` to Redis (TTL = `SESSION_TTL_SECONDS`) - Returns the HTML and cookies to the caller -Uses [Camoufox](https://github.com/daijro/camoufox) — Firefox with fingerprint patching at the C++/Juggler level. CF's detection scripts cannot distinguish it from a real Firefox profile. +Uses [Camoufox](https://github.com/daijro/camoufox) — Firefox with fingerprint patching at the C++/Juggler level to reduce common automation signals. Success still depends on the target's challenge variant, IP reputation, and upstream network conditions. + +### Akamai Bot Manager challenges + +Tier 3 and Tier 4 detect Akamai's `sec-cpt` / SBSD behavioral interstitials. The Akamai flow generates human-like pointer movement, handles supported press-and-hold widgets, waits for a valid `_abck` sensor cookie, and revisits the original URL when the interstitial does not reload automatically. + +Akamai configurations vary between properties and change over time. TRAWL treats a persistent interstitial as blocked and can escalate to Tier 4 when a residential proxy is configured. ### Imperva/Incapsula challenges -Tier 3 and Tier 4 also detect and solve Imperva/Incapsula WAF challenges, not just Cloudflare. Imperva's `reese84` (current) / `___utmvc` (legacy) sensor cookies are produced by an obfuscated in-page JS challenge — same principle as Cloudflare's `cf_clearance`: a real browser executing real JS produces the cookie without needing to understand the obfuscation. TRAWL detects the challenge page (`packages/tiers/src/detect.ts`'s `hasImpervaChallenge`) and polls for the sensor cookie (`packages/tiers/src/impervaWait.ts`) instead of the Cloudflare-specific wait loop. +Tier 3 and Tier 4 also detect and resolve supported Imperva/Incapsula WAF challenges. Imperva's `reese84` (current) / `___utmvc` (legacy) sensor cookies are produced by an obfuscated in-page JS challenge. TRAWL detects the response with `packages/tiers/src/utils/detect.ts` and waits for the sensor cookie through `packages/tiers/src/utils/impervaWait.ts`. **Caveat:** unlike Turnstile, Imperva's script sometimes layers in TLS/JA3 and behavioral checks beyond plain cookie generation, and its obfuscation changes periodically — success isn't guaranteed at the same rate as Cloudflare. Some Imperva deployments also show a visible interactive CAPTCHA widget (distinct from hCaptcha/reCAPTCHA) instead of the passive sensor-only path; that variant isn't solved yet. @@ -93,5 +99,5 @@ This runs Tier 1, then Tier 2, then returns an error if both fail — never laun | ---- | ------------ | ------------------------- | | 1 | 50–150ms | No | | 2 | 400–700ms | Yes (warm) | -| 3 | 4–15s | Yes (fresh solve) | +| 3 | Challenge-dependent | Yes (fresh solve) | | 4 | 15–45s | Yes (fresh solve + proxy) | diff --git a/apps/docs/deployment/docker-compose.md b/apps/docs/deployment/docker-compose.md index 4dda9fa..7da8e17 100644 --- a/apps/docs/deployment/docker-compose.md +++ b/apps/docs/deployment/docker-compose.md @@ -19,7 +19,8 @@ docker compose -f docker-compose.minimal.yml up -d ### Cached (default) -`docker-compose.yml` — scraper + Redis session cache. Repeat requests to the same domain return in ~500ms. +`docker-compose.yml` — scraper + Redis session cache, with API port `8191` and optional proxy port +`8192`. Repeat requests can reuse accepted sessions instead of solving the challenge again. ```bash docker compose up -d @@ -38,9 +39,9 @@ trawl: restart: always mem_limit: 3g environment: - BROWSER_POOL_SIZE: 5 + BROWSER_POOL_SIZE: 3 healthcheck: - test: wget -qO- http://localhost:8191/health + test: ["CMD", "curl", "-sf", "http://localhost:8191/health"] interval: 30s ``` @@ -69,7 +70,7 @@ See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone | Variable | Default | Description | | -------------------------------- | -------------------- | ----------------------------------------------------------------------- | -| `BROWSER_POOL_SIZE` | `3` | Warm browser instances | +| `BROWSER_POOL_SIZE` | `3` | Warm browsers; supplied minimal/cached Compose files override this to `1` | | `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 | | `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart after this many blocked/needs-js outcomes; set `0` to disable | | `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) | diff --git a/apps/docs/getting-started/configuration.md b/apps/docs/getting-started/configuration.md index e95c6c8..aa39d2c 100644 --- a/apps/docs/getting-started/configuration.md +++ b/apps/docs/getting-started/configuration.md @@ -101,7 +101,8 @@ These bounds keep an unresponsive Firefox process from permanently consuming a p **Default:** `3600` (1 hour) -How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache. +How long solved browser cookies and user-agent state are cached in Redis per domain. After this TTL +the next protected request triggers a fresh challenge solve (Tier 3) and refreshes the cache. Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3). diff --git a/apps/docs/getting-started/quick-start.md b/apps/docs/getting-started/quick-start.md index 9b1cf2e..6c54909 100644 --- a/apps/docs/getting-started/quick-start.md +++ b/apps/docs/getting-started/quick-start.md @@ -28,15 +28,14 @@ Everything has a working default. See [Configuration](/getting-started/configura docker compose up -d ``` -This starts three containers: +This starts two containers: | Container | Purpose | Port | | --------- | ----------------------- | -------- | | `redis` | Session cache backend | internal | -| `api` | Browser pool + HTTP API | 8191 | -| `web` | Landing page & docs UI | 3000 | +| `trawl` | Browser pool, API, and optional forward proxy | 8191, 8192 | -The API takes **15–30 seconds** on first boot while it launches the browser pool and downloads Camoufox (only on first build). Watch progress: +The API takes **15–30 seconds** to launch and warm the browser pool. Watch progress: ```bash docker compose logs -f api @@ -69,7 +68,9 @@ Expected: "busy": 0, "available": 3, "restarts": 0, - "avgRestarts": 0 + "avgRestarts": 0, + "stalled": 0, + "live": 3 } } ``` @@ -86,10 +87,13 @@ curl -s -X POST http://localhost:8191/v1 \ }' | jq '{status, url: .solution.url}' ``` -A `status: "ok"` response confirms TRAWL is working. The **second request to the same domain** returns in ~500ms from the session cache. +A `status: "ok"` response confirms TRAWL is working. Repeat requests can reuse the cached browser +session when the target still accepts it. -::: tip First request is slow — that's expected -The first time TRAWL sees a domain it solves the Cloudflare challenge fresh (4–15s). Every subsequent request uses the cached session and returns in under 500ms. This is the whole point. +::: tip Challenge requests take longer +An unprotected target may finish in Tier 1 without opening a browser. A recognized Cloudflare, +Akamai, or Imperva wall escalates to a fresh browser flow. Later requests can use Tier 2 while the +saved session remains valid. ::: ## 6. Use the native API (optional) @@ -118,8 +122,7 @@ Done. No other configuration changes are needed. TRAWL implements the FlareSolve ``` localhost:8191 → TRAWL API (FlareSolverr-compatible + native endpoint) -localhost:3000 → Web UI (landing page, visible in your browser) -localhost:3001 → Docs (this documentation, served locally) +localhost:8192 → HTTP/HTTPS forward proxy (when MITM_PROXY_ENABLED=true) ``` To stop everything: diff --git a/apps/docs/integrations/arr-apps.md b/apps/docs/integrations/arr-apps.md index 2d054cb..1ba6967 100644 --- a/apps/docs/integrations/arr-apps.md +++ b/apps/docs/integrations/arr-apps.md @@ -5,7 +5,9 @@ description: Connect Sonarr, Radarr, Lidarr, Readarr and other *arr apps via Pro # *arr Apps -Sonarr, Radarr, Lidarr, and Readarr do not talk to FlareSolverr directly. They go through **Prowlarr** (recommended) or **Jackett** as an indexer proxy. You only need to configure TRAWL once in the indexer manager — all *arr apps that use that manager get Cloudflare bypass automatically. +Sonarr, Radarr, Lidarr, and Readarr do not talk to FlareSolverr directly. They go through +**Prowlarr** (recommended) or **Jackett** as an indexer proxy. Configure TRAWL once in the indexer +manager and the connected *arr apps can use its challenge handling. ## Recommended setup @@ -47,11 +49,12 @@ Bazarr uses subtitle providers, not torrent indexers, so it does not use FlareSo ## Performance expectations -| Request type | Expected time | -| ------------------------------------------------ | ------------------------------ | -| First request to a domain | 4–15s (fresh Cloudflare solve) | -| Repeat request (same domain, session cached) | ~500ms | -| Plain site (no Cloudflare) | < 100ms | -| IP flagged by Cloudflare (Tier 4, if configured) | 15–45s | +| Request type | Expected path | +| -------------------------------------------- | ------------------------------------------ | +| Unprotected response | Tier 1, no browser | +| Recognized WAF challenge | Tier 3 fresh browser solve | +| Repeat request with an accepted session | Tier 2 cached browser session | +| Direct/datacenter path rejected | Tier 4, when a residential proxy is set | -The session cache TTL is configurable via `SESSION_TTL_SECONDS` (default 1 hour). Most Cloudflare-protected indexers re-challenge after 30–60 minutes, so daily search schedules in *arr apps almost always hit the cache. +The session cache TTL is configurable via `SESSION_TTL_SECONDS` (default 1 hour). Actual latency +depends on the indexer, challenge variant, IP reputation, and whether its saved session remains valid. diff --git a/apps/docs/integrations/jackett.md b/apps/docs/integrations/jackett.md index 1282da8..bcbd87a 100644 --- a/apps/docs/integrations/jackett.md +++ b/apps/docs/integrations/jackett.md @@ -33,7 +33,9 @@ Restart Jackett after editing the file. ## Verify -Navigate to any Cloudflare-protected indexer in Jackett and click **Test**. The first test triggers a full challenge solve (~10–30s). Subsequent tests on the same domain return in ~500ms from the TRAWL session cache. +Navigate to a protected indexer in Jackett and click **Test**. A first request that encounters a +recognized wall triggers the browser challenge flow. Subsequent tests can reuse the cached session +while it remains valid. ## Docker diff --git a/apps/docs/integrations/prowlarr.md b/apps/docs/integrations/prowlarr.md index 58e97c8..c37bcae 100644 --- a/apps/docs/integrations/prowlarr.md +++ b/apps/docs/integrations/prowlarr.md @@ -28,7 +28,9 @@ TRAWL implements the FlareSolverr v2 API exactly, including the `version: "2.0.0 ## Verification -After saving, trigger a search on any Cloudflare-protected indexer. TRAWL will handle the challenge transparently. Subsequent searches to the same indexer domain return from cache in ~500ms. +After saving, trigger a search on a protected indexer. TRAWL handles recognized Cloudflare, Akamai, +and Imperva walls through its tier engine. Later searches can reuse the saved session while the +target continues to accept it. ## Prowlarr inside Docker diff --git a/apps/docs/proxy/overview.md b/apps/docs/proxy/overview.md index 4aa2bcf..ced86bd 100644 --- a/apps/docs/proxy/overview.md +++ b/apps/docs/proxy/overview.md @@ -66,7 +66,7 @@ orchestrator used by `POST /scrape`: For browser-tier HTML responses, the proxy returns the rendered solved DOM rather than the original challenge response. Binary responses use the raw response bytes when the winning tier exposes them. -A hostname that produced a Cloudflare challenge is cached for five minutes. During that window, +A hostname that produced a supported challenge wall is cached for five minutes. During that window, later requests for the hostname skip Tier 0 and go directly to the tiered solver. Challenge and CAPTCHA solving is best effort. A site can still reject the browser, require user