mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
docs: update docs and README for Dragonfly and new structure
This commit is contained in:
@@ -7,28 +7,28 @@ description: All environment variables for TRAWL, with defaults and examples.
|
||||
|
||||
All configuration is via environment variables. Copy `.env.example` to `.env` and edit before starting.
|
||||
|
||||
## Redis
|
||||
## Dragonfly
|
||||
|
||||
### `REDIS_URL`
|
||||
|
||||
**Default:** `redis://localhost:6379`
|
||||
|
||||
Standard Redis connection URL. When running inside Docker Compose use the service name:
|
||||
Standard Redis-protocol connection URL — TRAWL's default cache backend is [Dragonfly](https://www.dragonflydb.io/), which is wire-compatible with Redis, so the URL scheme and env var name are unchanged. When running inside Docker Compose use the service name:
|
||||
|
||||
```ini
|
||||
REDIS_URL=redis://redis:6379
|
||||
REDIS_URL=redis://dragonfly:6379
|
||||
```
|
||||
|
||||
With authentication:
|
||||
|
||||
```ini
|
||||
REDIS_URL=redis://:yourpassword@redis:6379
|
||||
REDIS_URL=redis://:yourpassword@dragonfly:6379
|
||||
```
|
||||
|
||||
With a specific database index:
|
||||
|
||||
```ini
|
||||
REDIS_URL=redis://redis:6379/1
|
||||
REDIS_URL=redis://dragonfly:6379/1
|
||||
```
|
||||
|
||||
## Browser Pool
|
||||
@@ -91,7 +91,7 @@ BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall
|
||||
|
||||
**Default:** `3600` (1 hour)
|
||||
|
||||
How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
|
||||
How long Cloudflare cookies are cached in Dragonfly per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
|
||||
|
||||
Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3).
|
||||
|
||||
@@ -176,7 +176,7 @@ Port the Nuxt landing page listens on.
|
||||
## Full `.env.example`
|
||||
|
||||
```ini
|
||||
# ── Redis ─────────────────────────────────────
|
||||
# ── Dragonfly ─────────────────────────────────
|
||||
REDIS_URL=redis://localhost:6379
|
||||
|
||||
# ── Browser pool ──────────────────────────────
|
||||
|
||||
@@ -12,7 +12,7 @@ TRAWL is a Bun workspace monorepo. You can run each service locally with hot-rel
|
||||
| Tool | Version | Install |
|
||||
|------|---------|---------|
|
||||
| Bun | ≥ 1.2 | `curl -fsSL https://bun.sh/install \| bash` |
|
||||
| Redis | 7+ | Docker (see below) or system package |
|
||||
| Dragonfly | 1.39+ | Docker (see below) — Redis-protocol compatible |
|
||||
|
||||
## 1. Install dependencies
|
||||
|
||||
@@ -33,13 +33,13 @@ bun x camoufox-js fetch
|
||||
|
||||
This downloads the browser into the local cache. It only needs to run once per machine.
|
||||
|
||||
## 3. Start Redis
|
||||
## 3. Start Dragonfly
|
||||
|
||||
```bash
|
||||
docker run -d --name trawl-redis -p 6379:6379 redis:7-alpine
|
||||
docker run -d --name trawl-dragonfly -p 6379:6379 docker.dragonflydb.io/dragonflydb/dragonfly:v1.39.0
|
||||
```
|
||||
|
||||
Or use a local Redis installation if you have one.
|
||||
Or use any Redis-protocol-compatible server you already have — Dragonfly speaks the same wire protocol.
|
||||
|
||||
## 4. Create your `.env`
|
||||
|
||||
@@ -92,8 +92,8 @@ trawl/
|
||||
│ ├── types/ Shared TypeScript interfaces — no runtime logic
|
||||
│ ├── browser/ BrowserPool + SessionCache
|
||||
│ └── tiers/ Tier 1–4 executors + orchestrator
|
||||
├── docker-compose.yml scraper + Redis (default)
|
||||
├── docker-compose.minimal.yml scraper only, no Redis
|
||||
├── docker-compose.yml scraper + Dragonfly (default)
|
||||
├── docker-compose.minimal.yml scraper only, no Dragonfly
|
||||
├── docker-compose.prod.yml production with restart + healthcheck
|
||||
└── docker-compose.full.yml full stack including web + docs
|
||||
```
|
||||
|
||||
@@ -32,7 +32,7 @@ This starts three containers:
|
||||
|
||||
| Container | Purpose | Port |
|
||||
| --------- | ----------------------- | -------- |
|
||||
| `redis` | Session cache backend | internal |
|
||||
| `dragonfly` | Session cache backend | internal |
|
||||
| `api` | Browser pool + HTTP API | 8191 |
|
||||
| `web` | Landing page & docs UI | 3000 |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user