docs: update docs and README for Dragonfly and new structure

This commit is contained in:
germondai
2026-07-10 08:31:54 +02:00
parent e28463b214
commit 09bdd18b30
14 changed files with 86 additions and 58 deletions
+7 -7
View File
@@ -7,28 +7,28 @@ description: All environment variables for TRAWL, with defaults and examples.
All configuration is via environment variables. Copy `.env.example` to `.env` and edit before starting.
## Redis
## Dragonfly
### `REDIS_URL`
**Default:** `redis://localhost:6379`
Standard Redis connection URL. When running inside Docker Compose use the service name:
Standard Redis-protocol connection URL — TRAWL's default cache backend is [Dragonfly](https://www.dragonflydb.io/), which is wire-compatible with Redis, so the URL scheme and env var name are unchanged. When running inside Docker Compose use the service name:
```ini
REDIS_URL=redis://redis:6379
REDIS_URL=redis://dragonfly:6379
```
With authentication:
```ini
REDIS_URL=redis://:yourpassword@redis:6379
REDIS_URL=redis://:yourpassword@dragonfly:6379
```
With a specific database index:
```ini
REDIS_URL=redis://redis:6379/1
REDIS_URL=redis://dragonfly:6379/1
```
## Browser Pool
@@ -91,7 +91,7 @@ BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall
**Default:** `3600` (1 hour)
How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
How long Cloudflare cookies are cached in Dragonfly per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3).
@@ -176,7 +176,7 @@ Port the Nuxt landing page listens on.
## Full `.env.example`
```ini
# ── Redis ─────────────────────────────────────
# ── Dragonfly ─────────────────────────────────
REDIS_URL=redis://localhost:6379
# ── Browser pool ──────────────────────────────
@@ -12,7 +12,7 @@ TRAWL is a Bun workspace monorepo. You can run each service locally with hot-rel
| Tool | Version | Install |
|------|---------|---------|
| Bun | ≥ 1.2 | `curl -fsSL https://bun.sh/install \| bash` |
| Redis | 7+ | Docker (see below) or system package |
| Dragonfly | 1.39+ | Docker (see below) — Redis-protocol compatible |
## 1. Install dependencies
@@ -33,13 +33,13 @@ bun x camoufox-js fetch
This downloads the browser into the local cache. It only needs to run once per machine.
## 3. Start Redis
## 3. Start Dragonfly
```bash
docker run -d --name trawl-redis -p 6379:6379 redis:7-alpine
docker run -d --name trawl-dragonfly -p 6379:6379 docker.dragonflydb.io/dragonflydb/dragonfly:v1.39.0
```
Or use a local Redis installation if you have one.
Or use any Redis-protocol-compatible server you already have — Dragonfly speaks the same wire protocol.
## 4. Create your `.env`
@@ -92,8 +92,8 @@ trawl/
│ ├── types/ Shared TypeScript interfaces — no runtime logic
│ ├── browser/ BrowserPool + SessionCache
│ └── tiers/ Tier 14 executors + orchestrator
├── docker-compose.yml scraper + Redis (default)
├── docker-compose.minimal.yml scraper only, no Redis
├── docker-compose.yml scraper + Dragonfly (default)
├── docker-compose.minimal.yml scraper only, no Dragonfly
├── docker-compose.prod.yml production with restart + healthcheck
└── docker-compose.full.yml full stack including web + docs
```
+1 -1
View File
@@ -32,7 +32,7 @@ This starts three containers:
| Container | Purpose | Port |
| --------- | ----------------------- | -------- |
| `redis` | Session cache backend | internal |
| `dragonfly` | Session cache backend | internal |
| `api` | Browser pool + HTTP API | 8191 |
| `web` | Landing page & docs UI | 3000 |