External/trawl
1
0
Fork 0
mirror of https://github.com/germondai/trawl.git synced 2026-08-17 12:11:23 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: replace dragonflydb references with redis 8.8 and reformat markdown

Browse Source
This commit is contained in:
germondai
2026-07-11 17:00:29 +02:00
parent f746206de9
commit f8c90f0c49
22 changed files with 159 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/docs/api-reference/custom-headers.md
+12 -12
View File
@@ -37,12 +37,12 @@ Custom headers are merged **after** browser defaults, so they take precedence ov
## How headers are applied per tier
| Tier | Mechanism | Scope |
|------|-----------|-------|
| **Tier 1** — plain HTTP fetch | Spread into `fetch()` headers | All requests (there is only one) |
| **Tier 2** — cached browser session | `page.route(url, ...)` interception | Main document request only |
| **Tier 3** — fresh CF challenge solve | `page.route(url, ...)` interception | Main document request only |
| **Tier 4** — residential proxy escalation | `page.route(url, ...)` interception | Main document request only |
| Tier | Mechanism | Scope |
| ----------------------------------------- | ----------------------------------- | -------------------------------- |
| **Tier 1** — plain HTTP fetch | Spread into `fetch()` headers | All requests (there is only one) |
| **Tier 2** — cached browser session | `page.route(url, ...)` interception | Main document request only |
| **Tier 3** — fresh CF challenge solve | `page.route(url, ...)` interception | Main document request only |
| **Tier 4** — residential proxy escalation | `page.route(url, ...)` interception | Main document request only |
For browser tiers, route interception is scoped to the **exact target URL**. Subresources (JS, CSS, images, fonts, third-party CDNs) and Cloudflare challenge endpoints (`cdn-cgi/*`) are never intercepted — your `Authorization` header does not leak to third parties, and CF challenge solving is unaffected.
@@ -60,13 +60,13 @@ When a page requires both challenge bypass and custom headers, the sequence is:
## Common use cases
| Use case | Header |
|----------|--------|
| Authenticated APIs and portals | `Authorization: Bearer <token>` |
| Use case | Header |
| -------------------------------------- | ---------------------------------- |
| Authenticated APIs and portals | `Authorization: Bearer <token>` |
| Embed-only / iframe-restricted content | `Referer: https://parent-site.com` |
| CORS-restricted endpoints | `Origin: https://allowed-site.com` |
| Custom API keys | `X-API-Key: <key>` |
| Additional session tokens | `Cookie: session=<value>` |
| CORS-restricted endpoints | `Origin: https://allowed-site.com` |
| Custom API keys | `X-API-Key: <key>` |
| Additional session tokens | `Cookie: session=<value>` |
::: tip Cookie behaviour
Passing a `Cookie` header appends to any cookies the browser already holds (CF clearance, cached session cookies). It does not replace them.
apps/docs/api-reference/flaresolvr-compat.md
+14 -14
View File
@@ -24,14 +24,14 @@ interface FlareSolverrRequest {
### Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `cmd` | string | No | `"request.get"` or `"request.post"` (default `"request.get"`) |
| `url` | string | Yes | The URL to scrape |
| `maxTimeout` | number | No | Max wait in ms (default 60000) |
| `postData` | string | No | POST body (only for `request.post`). On TRAWL's native `/scrape` endpoint this field is named `body`; the `/v1` adapter maps `postData` → `body` internally so the FlareSolverr wire contract stays unchanged for existing callers. |
| `headers` | object | No | Custom headers forwarded to the target across all tiers — see [Custom Headers](/api-reference/custom-headers) |
| `proxy` | string | No | **TRAWL-specific extension** (not in the real FlareSolverr v2 contract) — per-request proxy override for Tier 3/4, see [Configuration § Proxies](/getting-started/configuration#proxies) |
| Field | Type | Required | Description |
| ------------ | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cmd` | string | No | `"request.get"` or `"request.post"` (default `"request.get"`) |
| `url` | string | Yes | The URL to scrape |
| `maxTimeout` | number | No | Max wait in ms (default 60000) |
| `postData` | string | No | POST body (only for `request.post`). On TRAWL's native `/scrape` endpoint this field is named `body`; the `/v1` adapter maps `postData` → `body` internally so the FlareSolverr wire contract stays unchanged for existing callers. |
| `headers` | object | No | Custom headers forwarded to the target across all tiers — see [Custom Headers](/api-reference/custom-headers) |
| `proxy` | string | No | **TRAWL-specific extension** (not in the real FlareSolverr v2 contract) — per-request proxy override for Tier 3/4, see [Configuration § Proxies](/getting-started/configuration#proxies) |
## Response
@@ -133,12 +133,12 @@ curl -s -X POST http://localhost:8191/v1 \
When the request fails, the response is still a FlareSolverr v2 envelope with `status: "error"` and an empty `solution`. The HTTP status code carries the failure class:
| Code | Meaning |
|------|---------|
| 200 | `status: "ok"` (request succeeded) |
| 400 | Malformed request body |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 500 | Internal error |
| Code | Meaning |
| ---- | -------------------------------------------------------------------- |
| 200 | `status: "ok"` (request succeeded) |
| 400 | Malformed request body |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 500 | Internal error |
Example — pool exhausted (HTTP 429):
apps/docs/api-reference/health-stats.md
+16 -18
View File
@@ -41,7 +41,6 @@ Full system health check. Used by Docker Compose health checks and monitoring sy
{
"status": "ok",
"uptime": 3842,
"dragonfly": "ok",
"pool": {
"total": 5,
"busy": 1,
@@ -52,16 +51,15 @@ Full system health check. Used by Docker Compose health checks and monitoring sy
}
```
| Field | Type | Description |
|-------|------|-------------|
| `status` | `"ok"` | Always `"ok"` when the API is reachable |
| `uptime` | number | Seconds since the API process started |
| `dragonfly` | `"ok"` or `"error"` | Whether the Dragonfly PING succeeded |
| `pool.total` | number | Total browser instances in the pool |
| `pool.busy` | number | Browsers currently processing a request |
| `pool.available` | number | Browsers ready to accept a request |
| `pool.restarts` | number | Total browser restarts since worker boot |
| `pool.avgRestarts` | number | Average restarts per browser |
| Field | Type | Description |
| ------------------ | ------ | ---------------------------------------- |
| `status` | `"ok"` | Always `"ok"` when the API is reachable |
| `uptime` | number | Seconds since the API process started |
| `pool.total` | number | Total browser instances in the pool |
| `pool.busy` | number | Browsers currently processing a request |
| `pool.available` | number | Browsers ready to accept a request |
| `pool.restarts` | number | Total browser restarts since worker boot |
| `pool.avgRestarts` | number | Average restarts per browser |
Pool stats are read directly from the browser pool. If the pool hasn't initialised yet, pool values will be zero.
@@ -88,12 +86,12 @@ Lightweight public stats for dashboards and landing pages.
}
```
| Field | Type | Description |
|-------|------|-------------|
| `browsers` | number | Total browser pool size |
| `available` | number | Idle browsers |
| `busy` | number | Browsers in use |
| `restarts` | number | Total browser restarts since startup |
| Field | Type | Description |
| ----------- | ------ | ------------------------------------ |
| `browsers` | number | Total browser pool size |
| `available` | number | Idle browsers |
| `busy` | number | Browsers in use |
| `restarts` | number | Total browser restarts since startup |
### Curl
@@ -103,6 +101,6 @@ curl -s http://localhost:8191/stats | jq
### Prometheus / uptime monitoring
Point an uptime monitor (e.g. UptimeRobot, Uptime Kuma) at `/health`. A 200 response with `"status": "ok"` and `"dragonfly": "ok"` confirms full operation.
Point an uptime monitor (e.g. UptimeRobot, Uptime Kuma) at `/health`. A 200 response with `"status": "ok"` confirms full operation.
For Prometheus, scrape `/stats` and parse the JSON — or add a `/metrics` endpoint as a future extension.
apps/docs/api-reference/native-api.md
+16 -16
View File
@@ -23,15 +23,15 @@ interface ScrapeRequest {
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `url` | string | — | The URL to scrape |
| `maxTimeout` | number | 60000 | Max total time in milliseconds |
| `skipHttp` | boolean | false | Skip Tier 1 (go straight to browser) |
| `maxTier` | 1–4 | 4 | Never escalate beyond this tier |
| `sessionId` | string | hostname | Override the Dragonfly session key |
| `headers` | object | — | Custom headers forwarded to the target across all tiers — see [Custom Headers](/api-reference/custom-headers) |
| `proxy` | string | — | Proxy URL used for this request's Tier 3/4 attempts instead of the configured `PROXY_URL`/`RESIDENTIAL_PROXY_URL` pool — see [Configuration § Proxies](/getting-started/configuration#proxies) |
| Field | Type | Default | Description |
| ------------ | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | string | — | The URL to scrape |
| `maxTimeout` | number | 60000 | Max total time in milliseconds |
| `skipHttp` | boolean | false | Skip Tier 1 (go straight to browser) |
| `maxTier` | 1–4 | 4 | Never escalate beyond this tier |
| `sessionId` | string | hostname | Override the Redis session key |
| `headers` | object | — | Custom headers forwarded to the target across all tiers — see [Custom Headers](/api-reference/custom-headers) |
| `proxy` | string | — | Proxy URL used for this request's Tier 3/4 attempts instead of the configured `PROXY_URL`/`RESIDENTIAL_PROXY_URL` pool — see [Configuration § Proxies](/getting-started/configuration#proxies) |
## Response
@@ -125,13 +125,13 @@ for (const t of result.timings) {
HTTP status codes:
| Code | Meaning |
|------|---------|
| 200 | `tier` succeeded |
| 400 | Malformed request body |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 503 | Browser pool initializing |
| 500 | Internal error |
| Code | Meaning |
| ---- | -------------------------------------------------------------------- |
| 200 | `tier` succeeded |
| 400 | Malformed request body |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 503 | Browser pool initializing |
| 500 | Internal error |
For 429 pool-exhaustion errors, the body is a **FlareSolverr v2 envelope** (same shape `/v1` uses) so clients can parse both endpoints uniformly:
apps/docs/api-reference/overview.md
+13 -13
View File
@@ -27,12 +27,12 @@ Content-Type: application/json
## Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Pool status and uptime |
| `GET` | `/stats` | Public numbers for dashboards |
| `POST` | `/v1` | FlareSolverr v2 compatible |
| `POST` | `/scrape` | Native TRAWL API |
| Method | Path | Description |
| ------ | --------- | ----------------------------- |
| `GET` | `/health` | Pool status and uptime |
| `GET` | `/stats` | Public numbers for dashboards |
| `POST` | `/v1` | FlareSolverr v2 compatible |
| `POST` | `/scrape` | Native TRAWL API |
## Error responses
@@ -46,13 +46,13 @@ Pool-exhaustion errors are an exception — they return a FlareSolverr v2 envelo
HTTP status codes:
| Code | Meaning |
|------|---------|
| 200 | Success |
| 400 | Bad request (missing/invalid fields) |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 503 | Browser pool initializing |
| 500 | Internal error |
| Code | Meaning |
| ---- | -------------------------------------------------------------------- |
| 200 | Success |
| 400 | Bad request (missing/invalid fields) |
| 429 | Pool exhausted — all browsers busy past `BROWSER_ACQUIRE_TIMEOUT_MS` |
| 503 | Browser pool initializing |
| 500 | Internal error |
::: info CORS
The API does **not** emit `Access-Control-Allow-Origin` headers. TRAWL is designed for direct, same-network access (e.g. Prowlarr/Jackett, internal services, your reverse proxy). If you need browser-based cross-origin access, terminate at a proxy that adds the CORS headers you need.
apps/docs/architecture/browser-pool.md
+5 -5
View File
@@ -97,10 +97,10 @@ const browser = await Camoufox({
Each Camoufox instance uses ~350–500 MB. With the default pool of 3:
| Pool size | RAM usage (browser only) |
|-----------|--------------------------|
| 1 | ~400 MB |
| 3 | ~1.2 GB |
| 5 | ~2 GB |
| 8 | ~3.2 GB |
| --------- | ------------------------ |
| 1 | ~400 MB |
| 3 | ~1.2 GB |
| 5 | ~2 GB |
| 8 | ~3.2 GB |
The API service sets `shm_size: 1gb` in Docker Compose. Firefox uses `/dev/shm` heavily; without enough shared memory, tabs crash silently.
apps/docs/architecture/overview.md
+3 -3
View File
@@ -29,7 +29,7 @@ Client (Prowlarr, curl, your code)
(packages/browser) (packages/browser)
│ │
▼ ▼
Camoufox Dragonfly (cache)
Camoufox Redis (cache)
Firefox N×
```
@@ -45,9 +45,9 @@ Maintains a fixed set of `{ browser, context }` pairs using [Camoufox](https://g
### Session Cache (`packages/browser/src/session.ts`)
Stores `{ cookies, userAgent, savedAt }` in Dragonfly, keyed by hostname (`session:example.com`). The TTL is configurable (default 1 hour). Tier 3 writes to it on every successful challenge solve. Tier 2 reads from it at the start of every request.
Stores `{ cookies, userAgent, savedAt }` in Redis, keyed by hostname (`session:example.com`). The TTL is configurable (default 1 hour). Tier 3 writes to it on every successful challenge solve. Tier 2 reads from it at the start of every request.
Dragonfly is optional — if `REDIS_URL` is not set, the session cache is disabled and every request escalates to Tier 3.
Redis is optional — if `REDIS_URL` is not set, the session cache is disabled and every request escalates to Tier 3.
### Tiers (`packages/tiers`)
apps/docs/architecture/session-cache.md
+9 -11
View File
@@ -1,11 +1,11 @@
---
title: Session Cache
description: How TRAWL caches Cloudflare cookies in Dragonfly to make repeat requests fast.
description: How TRAWL caches Cloudflare cookies in Redis to make repeat requests fast.
---
# Session Cache
The session cache is what makes Tier 2 possible. After every successful Tier 3 solve, the extracted Cloudflare cookies are saved to Dragonfly. The next request to the same domain injects those cookies into a browser context, skipping the challenge entirely.
The session cache is what makes Tier 2 possible. After every successful Tier 3 solve, the extracted Cloudflare cookies are saved to Redis. The next request to the same domain injects those cookies into a browser context, skipping the challenge entirely.
## Storage format
@@ -40,28 +40,26 @@ Subdomains have separate sessions because Cloudflare can issue different challen
Tier 3 succeeds
│
├── extract cookies from browser context
├── DRAGONFLY SET session:hostname → JSON EX SESSION_TTL_SECONDS
├── REDIS SET session:hostname → JSON EX SESSION_TTL_SECONDS
│
└── next request to same domain:
DRAGONFLY GET session:hostname
REDIS GET session:hostname
├── hit → Tier 2: inject cookies, navigate (500ms)
└── miss → Tier 3: fresh solve, save to cache
```
## Invalidation
If Tier 2 navigates with the cached cookies and the result is still a Cloudflare interstitial (the session expired before Dragonfly's TTL), the orchestrator:
If Tier 2 navigates with the cached cookies and the result is still a Cloudflare interstitial (the session expired before Redis's TTL), the orchestrator:
1. Calls `sessionCache.invalidate(domain)` — deletes the Dragonfly key
1. Calls `sessionCache.invalidate(domain)` — deletes the Redis key
2. Escalates to Tier 3 to get a fresh session
This handles the case where Cloudflare's `cf_clearance` cookie (30-minute expiry) expires before the Dragonfly TTL does.
This handles the case where Cloudflare's `cf_clearance` cookie (30-minute expiry) expires before the Redis TTL does.
## Dragonfly
## Redis
TRAWL's default cache backend is [Dragonfly](https://www.dragonflydb.io/) — a multi-threaded, shared-nothing in-memory datastore that's wire-compatible with the Redis protocol, so no query/command changes were needed to adopt it. It's a drop-in replacement for Redis at the connection-string level (`REDIS_URL` still points at it) while scaling across CPU cores instead of Redis's single-threaded event loop.
TRAWL talks to it with `new RedisClient(REDIS_URL)` from Bun's native Redis client (not ioredis) — Bun's client speaks the same RESP protocol Dragonfly serves, so this needed no code changes either.
TRAWL's cache backend is Redis 8.8. TRAWL talks to it with `new RedisClient(REDIS_URL)` from Bun's native Redis client (not ioredis).
```typescript
import { RedisClient } from 'bun'
apps/docs/architecture/tiered-execution.md
+8 -8
View File
@@ -45,7 +45,7 @@ Accept-Encoding: gzip, deflate, br
## Tier 2 — Cached Browser Session
Reads `session:{hostname}` from Dragonfly. If found, injects the saved cookies into a pooled Firefox context and navigates. Because the Cloudflare `cf_clearance` cookie is present, the challenge page is skipped and the site loads directly.
Reads `session:{hostname}` from Redis. If found, injects the saved cookies into a pooled Firefox context and navigates. Because the Cloudflare `cf_clearance` cookie is present, the challenge page is skipped and the site loads directly.
**Succeeds for:** any domain that was previously solved by Tier 3 and whose session hasn't expired.
@@ -57,7 +57,7 @@ Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS`
On success:
- Extracts all cookies from the page context
- Writes `session:{hostname} → { cookies, userAgent, savedAt }` to Dragonfly (TTL = `SESSION_TTL_SECONDS`)
- Writes `session:{hostname} → { cookies, userAgent, savedAt }` to Redis (TTL = `SESSION_TTL_SECONDS`)
- Returns the HTML and cookies to the caller
Uses [Camoufox](https://github.com/daijro/camoufox) — Firefox with fingerprint patching at the C++/Juggler level. CF's detection scripts cannot distinguish it from a real Firefox profile.
@@ -89,9 +89,9 @@ This runs Tier 1, then Tier 2, then returns an error if both fail — never laun
## Timing reference
| Tier | Typical time | Browser used |
|------|-------------|--------------|
| 1 | 50–150ms | No |
| 2 | 400–700ms | Yes (warm) |
| 3 | 4–15s | Yes (fresh solve) |
| 4 | 15–45s | Yes (fresh solve + proxy) |
| Tier | Typical time | Browser used |
| ---- | ------------ | ------------------------- |
| 1 | 50–150ms | No |
| 2 | 400–700ms | Yes (warm) |
| 3 | 4–15s | Yes (fresh solve) |
| 4 | 15–45s | Yes (fresh solve + proxy) |
apps/docs/deployment/docker-compose.md
+20 -20
View File
@@ -11,7 +11,7 @@ Four compose files live in the repo root, matching the setups shown on the landi
### Minimal
`docker-compose.minimal.yml` — single service, no Dragonfly. Fastest to get started, no session caching.
`docker-compose.minimal.yml` — single service, no Redis. Fastest to get started, no session caching.
```bash
docker compose -f docker-compose.minimal.yml up -d
@@ -19,7 +19,7 @@ docker compose -f docker-compose.minimal.yml up -d
### Cached (default)
`docker-compose.yml` — scraper + Dragonfly session cache. Repeat requests to the same domain return in ~500ms.
`docker-compose.yml` — scraper + Redis session cache. Repeat requests to the same domain return in ~500ms.
```bash
docker compose up -d
@@ -73,40 +73,40 @@ See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone
docker compose -f docker-compose.full.yml up -d
```
| Service | URL | Description |
|---------|-----|-------------|
| `trawl` | `localhost:8191` | Scraper API |
| `web` | `localhost:3000` | Landing page |
| Service | URL | Description |
| ------- | ---------------- | ------------- |
| `trawl` | `localhost:8191` | Scraper API |
| `web` | `localhost:3000` | Landing page |
| `docs` | `localhost:3001` | Documentation |
| `dragonfly` | internal | Session cache |
| `redis` | internal | Session cache |
First run builds the web and docs images locally — takes a couple of minutes. Subsequent runs are fast (layers cached).
## Environment variables
| Variable | Default | Description |
|----------|---------|-------------|
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable |
| `REDIS_URL` | `redis://dragonfly:6379` | Dragonfly connection (set automatically in compose) |
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
| Variable | Default | Description |
| -------------------------------- | -------------------- | -------------------------------------------------------------------------- |
| `BROWSER_POOL_SIZE` | `3` | Warm browser instances |
| `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 |
| `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable |
| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) |
| `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation |
## Logs
```bash
docker compose logs -f trawl
docker compose logs -f dragonfly
docker compose logs -f redis
```
## Memory guide
| `BROWSER_POOL_SIZE` | Approx. RAM | Recommended host RAM |
|---------------------|-------------|----------------------|
| 1 | ~500 MB | 1 GB |
| 3 | ~1.2 GB | 2 GB |
| 5 | ~2 GB | 3 GB |
| 10 | ~4 GB | 6 GB |
| ------------------- | ----------- | -------------------- |
| 1 | ~500 MB | 1 GB |
| 3 | ~1.2 GB | 2 GB |
| 5 | ~2 GB | 3 GB |
| 10 | ~4 GB | 6 GB |
Each Camoufox Firefox instance uses ~350–500 MB under load.
apps/docs/deployment/standalone.md
+3 -3
View File
@@ -15,19 +15,19 @@ The API image is published to GHCR on every push to `main`. Pull it directly —
# Pull
docker pull ghcr.io/germondai/trawl:latest
# Run (no Dragonfly — session caching disabled, scraping still works)
# Run (no Redis — session caching disabled, scraping still works)
docker run -d \
--name trawl \
-p 8191:8191 \
--shm-size=1gb \
ghcr.io/germondai/trawl:latest
# Run (with external Dragonfly, Redis-protocol compatible)
# Run (with external Redis)
docker run -d \
--name trawl \
-p 8191:8191 \
--shm-size=1gb \
-e REDIS_URL=redis://your-dragonfly-host:6379 \
-e REDIS_URL=redis://your-redis-host:6379 \
-e BROWSER_POOL_SIZE=3 \
ghcr.io/germondai/trawl:latest
```
apps/docs/deployment/troubleshooting.md
+4 -4
View File
@@ -11,7 +11,7 @@ description: Common issues and how to fix them.
**Causes:**
1. **Dragonfly not reachable** — Check `REDIS_URL`. From inside Docker, use `redis://dragonfly:6379` not `redis://localhost:6379`.
1. **Redis not reachable** — Check `REDIS_URL`. From inside Docker, use `redis://redis:6379` not `redis://localhost:6379`.
2. **Camoufox binary not installed** — The API Dockerfile runs `bunx camoufox-js fetch`. If this step was skipped (e.g. build cache reuse), rebuild: `docker compose build --no-cache api`.
3. **shm_size too small** — Ensure `shm_size: 1gb` is set on the API service.
@@ -46,7 +46,7 @@ docker compose up -d --force-recreate
**Causes:**
1. **Dragonfly session data is not persisting** — Run `docker exec trawl-dragonfly redis-cli keys "session:*"` after a successful scrape. If empty, the session cache write is failing. Check API logs for Dragonfly connection errors.
1. **Redis session data is not persisting** — Run `docker compose exec redis redis-cli keys "session:*"` after a successful scrape. If empty, the session cache write is failing. Check API logs for Redis connection errors.
2. **`SESSION_TTL_SECONDS` set too low** — If it's shorter than Cloudflare's challenge interval, the cache expires before the next request.
3. **Domain key mismatch** — The key is the hostname only. `sub.example.com` and `www.example.com` are separate sessions.
@@ -95,8 +95,8 @@ Each Camoufox instance uses 350–500 MB. With 3 browsers, expect ~1.5 GB total.
# Live API logs
docker compose logs -f api
# Check Dragonfly keys
docker exec trawl-dragonfly redis-cli keys "*"
# Check Redis keys
docker compose exec redis redis-cli keys "*"
# Test scrape endpoint
curl -s -X POST http://localhost:8191/scrape \
apps/docs/getting-started/configuration.md
+7 -7
View File
@@ -7,28 +7,28 @@ description: All environment variables for TRAWL, with defaults and examples.
All configuration is via environment variables. Copy `.env.example` to `.env` and edit before starting.
## Dragonfly
## Redis
### `REDIS_URL`
**Default:** `redis://localhost:6379`
Standard Redis-protocol connection URL — TRAWL's default cache backend is [Dragonfly](https://www.dragonflydb.io/), which is wire-compatible with Redis, so the URL scheme and env var name are unchanged. When running inside Docker Compose use the service name:
Standard Redis connection URL — TRAWL's cache backend is Redis 8.8. When running inside Docker Compose use the service name:
```ini
REDIS_URL=redis://dragonfly:6379
REDIS_URL=redis://redis:6379
```
With authentication:
```ini
REDIS_URL=redis://:yourpassword@dragonfly:6379
REDIS_URL=redis://:yourpassword@redis:6379
```
With a specific database index:
```ini
REDIS_URL=redis://dragonfly:6379/1
REDIS_URL=redis://redis:6379/1
```
## Browser Pool
@@ -91,7 +91,7 @@ BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall
**Default:** `3600` (1 hour)
How long Cloudflare cookies are cached in Dragonfly per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3).
@@ -176,7 +176,7 @@ Port the Nuxt landing page listens on.
## Full `.env.example`
```ini
# ── Dragonfly ─────────────────────────────────
# ── Redis ─────────────────────────────────────
REDIS_URL=redis://localhost:6379
# ── Browser pool ──────────────────────────────
apps/docs/getting-started/local-development.md
+9 -9
View File
@@ -9,10 +9,10 @@ TRAWL is a Bun workspace monorepo. You can run each service locally with hot-rel
## Prerequisites
| Tool | Version | Install |
|------|---------|---------|
| Bun | ≥ 1.2 | `curl -fsSL https://bun.sh/install \| bash` |
| Dragonfly | 1.39+ | Docker (see below) — Redis-protocol compatible |
| Tool | Version | Install |
| ----- | ------- | ------------------------------------------- |
| Bun | ≥ 1.2 | `curl -fsSL https://bun.sh/install \| bash` |
| Redis | 8.8+ | Docker (see below) |
## 1. Install dependencies
@@ -33,13 +33,13 @@ bun x camoufox-js fetch
This downloads the browser into the local cache. It only needs to run once per machine.
## 3. Start Dragonfly
## 3. Start Redis
```bash
docker run -d --name trawl-dragonfly -p 6379:6379 docker.dragonflydb.io/dragonflydb/dragonfly:v1.39.0
docker run -d --name trawl-redis -p 6379:6379 redis:8.8-alpine
```
Or use any Redis-protocol-compatible server you already have — Dragonfly speaks the same wire protocol.
Or use any Redis-protocol-compatible server you already have.
## 4. Create your `.env`
@@ -92,8 +92,8 @@ trawl/
│ ├── types/ Shared TypeScript interfaces — no runtime logic
│ ├── browser/ BrowserPool + SessionCache
│ └── tiers/ Tier 1–4 executors + orchestrator
├── docker-compose.yml scraper + Dragonfly (default)
├── docker-compose.minimal.yml scraper only, no Dragonfly
├── docker-compose.yml scraper + Redis (default)
├── docker-compose.minimal.yml scraper only, no Redis
├── docker-compose.prod.yml production with restart + healthcheck
└── docker-compose.full.yml full stack including web + docs
```
apps/docs/getting-started/quick-start.md
+1 -1
View File
@@ -32,7 +32,7 @@ This starts three containers:
| Container | Purpose | Port |
| --------- | ----------------------- | -------- |
| `dragonfly` | Session cache backend | internal |
| `redis` | Session cache backend | internal |
| `api` | Browser pool + HTTP API | 8191 |
| `web` | Landing page & docs UI | 3000 |
apps/docs/integrations/arr-apps.md
+6 -6
View File
@@ -47,11 +47,11 @@ Bazarr uses subtitle providers, not torrent indexers, so it does not use FlareSo
## Performance expectations
| Request type | Expected time |
|---|---|
| First request to a domain | 4–15s (fresh Cloudflare solve) |
| Repeat request (same domain, session cached) | ~500ms |
| Plain site (no Cloudflare) | < 100ms |
| IP flagged by Cloudflare (Tier 4, if configured) | 15–45s |
| Request type | Expected time |
| ------------------------------------------------ | ------------------------------ |
| First request to a domain | 4–15s (fresh Cloudflare solve) |
| Repeat request (same domain, session cached) | ~500ms |
| Plain site (no Cloudflare) | < 100ms |
| IP flagged by Cloudflare (Tier 4, if configured) | 15–45s |
The session cache TTL is configurable via `SESSION_TTL_SECONDS` (default 1 hour). Most Cloudflare-protected indexers re-challenge after 30–60 minutes, so daily search schedules in *arr apps almost always hit the cache.