From 09bdd18b3002e4fd64278e6563a76a39ad7888ee Mon Sep 17 00:00:00 2001 From: germondai Date: Fri, 10 Jul 2026 08:31:54 +0200 Subject: [PATCH] docs: update docs and README for Dragonfly and new structure --- CONTRIBUTING.md | 4 +-- README.md | 20 +++++++----- SECURITY.md | 2 +- apps/docs/api-reference/health-stats.md | 32 ++++++++++++++++--- apps/docs/api-reference/native-api.md | 2 +- apps/docs/architecture/overview.md | 8 ++--- apps/docs/architecture/session-cache.md | 20 ++++++------ apps/docs/architecture/tiered-execution.md | 4 +-- apps/docs/deployment/docker-compose.md | 10 +++--- apps/docs/deployment/standalone.md | 6 ++-- apps/docs/deployment/troubleshooting.md | 8 ++--- apps/docs/getting-started/configuration.md | 14 ++++---- .../docs/getting-started/local-development.md | 12 +++---- apps/docs/getting-started/quick-start.md | 2 +- 14 files changed, 86 insertions(+), 58 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 73290a6..ab76627 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,7 +17,7 @@ For security issues, **do not open a public issue** — see [SECURITY.md](SECURI ## Development setup -Requirements: **Bun ≥ 1.1** and **Docker** (for the Redis service used in tests). +Requirements: **Bun ≥ 1.1** and **Docker** (for the Dragonfly service used in tests). ```bash git clone https://github.com/germondai/trawl.git @@ -34,7 +34,7 @@ bun run dev:web # Nuxt 4 landing page bun run dev:docs # VitePress docs site ``` -The API requires Redis. The fastest way is `docker compose up -d redis`. +The API requires Dragonfly (Redis-protocol compatible). The fastest way is `docker compose up -d dragonfly`. ### Linting & formatting diff --git a/README.md b/README.md index 95a23c3..8986787 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@

-

+

TRAWL

@@ -14,7 +14,7 @@ Self-hosted web scraping engine with adaptive tier execution. Solves Cloudflare - **4-tier execution** — plain HTTP → cached session → fresh CF solve → residential proxy - **Native captcha solving** — CF Turnstile, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide - **Camoufox Firefox** — fingerprint-patched at the C++/Juggler level; indistinguishable from a real browser -- **Session cache** — `cf_clearance` cookies stored in Redis; repeat requests to the same domain return in ~500ms +- **Session cache** — `cf_clearance` cookies stored in Dragonfly; repeat requests to the same domain return in ~500ms - **FlareSolverr v2 compatible** — works with Prowlarr, Jackett, Sonarr, Radarr, and the full \*arr ecosystem out of the box - **No external APIs** — reCAPTCHA audio transcription uses Google's free STT endpoint by default; everything else is local @@ -26,7 +26,7 @@ git clone https://github.com/germondai/trawl.git cd trawl cp .env.example .env -# Start scraper + Redis +# Start scraper + Dragonfly docker compose up -d # Verify @@ -89,8 +89,8 @@ Tier 4: Residential proxy ──── success ──→ cache + return (15–45 | File | Description | | ---------------------------- | -------------------------------------------------------- | -| `docker-compose.yml` | Scraper + Redis (default) | -| `docker-compose.minimal.yml` | Scraper only, no Redis | +| `docker-compose.yml` | Scraper + Dragonfly (default) | +| `docker-compose.minimal.yml` | Scraper only, no Dragonfly | | `docker-compose.prod.yml` | Production: `restart: always`, memory limit, healthcheck | | `docker-compose.full.yml` | Full stack: scraper + web + docs | @@ -121,20 +121,24 @@ Synology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship | `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before HTTP 429 is returned | | `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Recycle a browser after this many `blocked`/`needs-js` outcomes; set `0` to disable | | `BROWSER_CONTENT_PROCESSES` | `2` | Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU | -| `SESSION_TTL_SECONDS` | `3600` | Redis session cache TTL (seconds) | -| `REDIS_URL` | `redis://localhost:6379` | Redis connection string | +| `SESSION_TTL_SECONDS` | `3600` | Dragonfly session cache TTL (seconds) | +| `REDIS_URL` | `redis://localhost:6379` | Dragonfly connection string (Redis-protocol compatible) | | `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation | | `STT_URL` | — | Local Whisper endpoint for reCAPTCHA (optional) | | `PORT` | `8191` | API listen port | ## Stack +Built on a modern, fast-by-default stack: Bun + Elysia for the API, Dragonfly for caching, +Camoufox (hardened Firefox) for browser automation, and Nuxt for the web UI — no legacy +Node/Express/Redis baggage. + | Layer | Technology | | ------------- | ---------------------------------- | | Runtime | Bun | | API | Elysia | | Browser | Camoufox Firefox (via camoufox-js) | -| Session cache | Redis | +| Session cache | Dragonfly | | Landing page | Nuxt 4 | | Documentation | VitePress | diff --git a/SECURITY.md b/SECURITY.md index 349e8b3..a90884d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -39,7 +39,7 @@ In scope: - Anything in `apps/api/` that lets an unauthenticated remote actor read or modify data they shouldn't. - Anything in `packages/browser/` that escapes the browser sandbox or exposes host state. -- Anything in the Redis-backed session cache that leaks another operator's session. +- Anything in the Dragonfly-backed session cache that leaks another operator's session. - Supply-chain issues in pinned dependencies (compromised lockfile entries, malicious transitive packages). Out of scope: diff --git a/apps/docs/api-reference/health-stats.md b/apps/docs/api-reference/health-stats.md index c60b3fd..c6add02 100644 --- a/apps/docs/api-reference/health-stats.md +++ b/apps/docs/api-reference/health-stats.md @@ -1,11 +1,33 @@ --- title: Health & Stats -description: GET /health and GET /stats — monitoring endpoints. +description: GET /, GET /health and GET /stats — status and monitoring endpoints. --- # Health & Stats -Both endpoints require no authentication and are safe to expose to monitoring tools. +These endpoints require no authentication and are safe to expose to monitoring tools. + +--- + +## `GET /` + +FlareSolverr-style readiness message — confirms the API process is up (does not wait on the browser pool). + +### Response + +```json +{ + "msg": "TRAWL is ready!", + "version": "0.1.0", + "uptime": 42 +} +``` + +### Curl + +```bash +curl -s http://localhost:8191/ +``` --- @@ -19,7 +41,7 @@ Full system health check. Used by Docker Compose health checks and monitoring sy { "status": "ok", "uptime": 3842, - "redis": "ok", + "dragonfly": "ok", "pool": { "total": 5, "busy": 1, @@ -34,7 +56,7 @@ Full system health check. Used by Docker Compose health checks and monitoring sy |-------|------|-------------| | `status` | `"ok"` | Always `"ok"` when the API is reachable | | `uptime` | number | Seconds since the API process started | -| `redis` | `"ok"` or `"error"` | Whether the Redis PING succeeded | +| `dragonfly` | `"ok"` or `"error"` | Whether the Dragonfly PING succeeded | | `pool.total` | number | Total browser instances in the pool | | `pool.busy` | number | Browsers currently processing a request | | `pool.available` | number | Browsers ready to accept a request | @@ -81,6 +103,6 @@ curl -s http://localhost:8191/stats | jq ### Prometheus / uptime monitoring -Point an uptime monitor (e.g. UptimeRobot, Uptime Kuma) at `/health`. A 200 response with `"status": "ok"` and `"redis": "ok"` confirms full operation. +Point an uptime monitor (e.g. UptimeRobot, Uptime Kuma) at `/health`. A 200 response with `"status": "ok"` and `"dragonfly": "ok"` confirms full operation. For Prometheus, scrape `/stats` and parse the JSON — or add a `/metrics` endpoint as a future extension. diff --git a/apps/docs/api-reference/native-api.md b/apps/docs/api-reference/native-api.md index 3fd9dfd..1d9008b 100644 --- a/apps/docs/api-reference/native-api.md +++ b/apps/docs/api-reference/native-api.md @@ -29,7 +29,7 @@ interface ScrapeRequest { | `maxTimeout` | number | 60000 | Max total time in milliseconds | | `skipHttp` | boolean | false | Skip Tier 1 (go straight to browser) | | `maxTier` | 1–4 | 4 | Never escalate beyond this tier | -| `sessionId` | string | hostname | Override the Redis session key | +| `sessionId` | string | hostname | Override the Dragonfly session key | | `headers` | object | — | Custom headers forwarded to the target across all tiers — see [Custom Headers](/api-reference/custom-headers) | | `proxy` | string | — | Proxy URL used for this request's Tier 3/4 attempts instead of the configured `PROXY_URL`/`RESIDENTIAL_PROXY_URL` pool — see [Configuration § Proxies](/getting-started/configuration#proxies) | diff --git a/apps/docs/architecture/overview.md b/apps/docs/architecture/overview.md index 308e6aa..1e9cad6 100644 --- a/apps/docs/architecture/overview.md +++ b/apps/docs/architecture/overview.md @@ -29,7 +29,7 @@ Client (Prowlarr, curl, your code) (packages/browser) (packages/browser) │ │ ▼ ▼ - Camoufox Redis (cache) + Camoufox Dragonfly (cache) Firefox N× ``` @@ -37,7 +37,7 @@ Client (Prowlarr, curl, your code) ### API (`apps/api`) -An Elysia HTTP server. Accepts scrape requests and calls the orchestrator inline — all browser work happens in the same process. Exposes `/health` and `/stats` for monitoring. +An Elysia HTTP server. Accepts scrape requests and calls the orchestrator inline — all browser work happens in the same process. Exposes `/` for a FlareSolverr-style readiness message, plus `/health` and `/stats` for monitoring. Routes live under `apps/api/src/routes/`, with shared config/pool state in `config.ts`/`deps.ts`. ### Browser Pool (`packages/browser/src/pool.ts`) @@ -45,9 +45,9 @@ Maintains a fixed set of `{ browser, context }` pairs using [Camoufox](https://g ### Session Cache (`packages/browser/src/session.ts`) -Stores `{ cookies, userAgent, savedAt }` in Redis, keyed by hostname (`session:example.com`). The TTL is configurable (default 1 hour). Tier 3 writes to it on every successful challenge solve. Tier 2 reads from it at the start of every request. +Stores `{ cookies, userAgent, savedAt }` in Dragonfly, keyed by hostname (`session:example.com`). The TTL is configurable (default 1 hour). Tier 3 writes to it on every successful challenge solve. Tier 2 reads from it at the start of every request. -Redis is optional — if `REDIS_URL` is not set, the session cache is disabled and every request escalates to Tier 3. +Dragonfly is optional — if `REDIS_URL` is not set, the session cache is disabled and every request escalates to Tier 3. ### Tiers (`packages/tiers`) diff --git a/apps/docs/architecture/session-cache.md b/apps/docs/architecture/session-cache.md index 90f184b..8f38326 100644 --- a/apps/docs/architecture/session-cache.md +++ b/apps/docs/architecture/session-cache.md @@ -1,11 +1,11 @@ --- title: Session Cache -description: How TRAWL caches Cloudflare cookies in Redis to make repeat requests fast. +description: How TRAWL caches Cloudflare cookies in Dragonfly to make repeat requests fast. --- # Session Cache -The session cache is what makes Tier 2 possible. After every successful Tier 3 solve, the extracted Cloudflare cookies are saved to Redis. The next request to the same domain injects those cookies into a browser context, skipping the challenge entirely. +The session cache is what makes Tier 2 possible. After every successful Tier 3 solve, the extracted Cloudflare cookies are saved to Dragonfly. The next request to the same domain injects those cookies into a browser context, skipping the challenge entirely. ## Storage format @@ -40,26 +40,28 @@ Subdomains have separate sessions because Cloudflare can issue different challen Tier 3 succeeds │ ├── extract cookies from browser context - ├── REDIS SET session:hostname → JSON EX SESSION_TTL_SECONDS + ├── DRAGONFLY SET session:hostname → JSON EX SESSION_TTL_SECONDS │ └── next request to same domain: - REDIS GET session:hostname + DRAGONFLY GET session:hostname ├── hit → Tier 2: inject cookies, navigate (500ms) └── miss → Tier 3: fresh solve, save to cache ``` ## Invalidation -If Tier 2 navigates with the cached cookies and the result is still a Cloudflare interstitial (the session expired before Redis TTL), the orchestrator: +If Tier 2 navigates with the cached cookies and the result is still a Cloudflare interstitial (the session expired before Dragonfly's TTL), the orchestrator: -1. Calls `sessionCache.invalidate(domain)` — deletes the Redis key +1. Calls `sessionCache.invalidate(domain)` — deletes the Dragonfly key 2. Escalates to Tier 3 to get a fresh session -This handles the case where Cloudflare's `cf_clearance` cookie (30-minute expiry) expires before the Redis TTL does. +This handles the case where Cloudflare's `cf_clearance` cookie (30-minute expiry) expires before the Dragonfly TTL does. -## Redis client +## Dragonfly -TRAWL uses `new RedisClient(REDIS_URL)` from Bun's native Redis client (not ioredis). Bun's Redis client is ~7.9× faster than ioredis for simple GET/SET operations, which matters since every request does at least one Redis read. +TRAWL's default cache backend is [Dragonfly](https://www.dragonflydb.io/) — a multi-threaded, shared-nothing in-memory datastore that's wire-compatible with the Redis protocol, so no query/command changes were needed to adopt it. It's a drop-in replacement for Redis at the connection-string level (`REDIS_URL` still points at it) while scaling across CPU cores instead of Redis's single-threaded event loop. + +TRAWL talks to it with `new RedisClient(REDIS_URL)` from Bun's native Redis client (not ioredis) — Bun's client speaks the same RESP protocol Dragonfly serves, so this needed no code changes either. ```typescript import { RedisClient } from 'bun' diff --git a/apps/docs/architecture/tiered-execution.md b/apps/docs/architecture/tiered-execution.md index dc7fe80..9f4bb3d 100644 --- a/apps/docs/architecture/tiered-execution.md +++ b/apps/docs/architecture/tiered-execution.md @@ -45,7 +45,7 @@ Accept-Encoding: gzip, deflate, br ## Tier 2 — Cached Browser Session -Reads `session:{hostname}` from Redis. If found, injects the saved cookies into a pooled Firefox context and navigates. Because the Cloudflare `cf_clearance` cookie is present, the challenge page is skipped and the site loads directly. +Reads `session:{hostname}` from Dragonfly. If found, injects the saved cookies into a pooled Firefox context and navigates. Because the Cloudflare `cf_clearance` cookie is present, the challenge page is skipped and the site loads directly. **Succeeds for:** any domain that was previously solved by Tier 3 and whose session hasn't expired. @@ -57,7 +57,7 @@ Acquires a browser from the pool (or waits up to `BROWSER_ACQUIRE_TIMEOUT_MS` On success: - Extracts all cookies from the page context -- Writes `session:{hostname} → { cookies, userAgent, savedAt }` to Redis (TTL = `SESSION_TTL_SECONDS`) +- Writes `session:{hostname} → { cookies, userAgent, savedAt }` to Dragonfly (TTL = `SESSION_TTL_SECONDS`) - Returns the HTML and cookies to the caller Uses [Camoufox](https://github.com/daijro/camoufox) — Firefox with fingerprint patching at the C++/Juggler level. CF's detection scripts cannot distinguish it from a real Firefox profile. diff --git a/apps/docs/deployment/docker-compose.md b/apps/docs/deployment/docker-compose.md index d6be3db..4b182e4 100644 --- a/apps/docs/deployment/docker-compose.md +++ b/apps/docs/deployment/docker-compose.md @@ -11,7 +11,7 @@ Four compose files live in the repo root, matching the setups shown on the landi ### Minimal -`docker-compose.minimal.yml` — single service, no Redis. Fastest to get started, no session caching. +`docker-compose.minimal.yml` — single service, no Dragonfly. Fastest to get started, no session caching. ```bash docker compose -f docker-compose.minimal.yml up -d @@ -19,7 +19,7 @@ docker compose -f docker-compose.minimal.yml up -d ### Cached (default) -`docker-compose.yml` — scraper + Redis session cache. Repeat requests to the same domain return in ~500ms. +`docker-compose.yml` — scraper + Dragonfly session cache. Repeat requests to the same domain return in ~500ms. ```bash docker compose up -d @@ -78,7 +78,7 @@ docker compose -f docker-compose.full.yml up -d | `trawl` | `localhost:8191` | Scraper API | | `web` | `localhost:3000` | Landing page | | `docs` | `localhost:3001` | Documentation | -| `redis` | internal | Session cache | +| `dragonfly` | internal | Session cache | First run builds the web and docs images locally — takes a couple of minutes. Subsequent runs are fast (layers cached). @@ -89,14 +89,14 @@ First run builds the web and docs images locally — takes a couple of minutes. | `BROWSER_POOL_SIZE` | `3` | Warm browser instances | | `BROWSER_ACQUIRE_TIMEOUT_MS` | `15000` | How long `acquire()` polls for a free browser before returning HTTP 429 | | `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8` | Restart a browser after this many fresh/proxy contexts; set `0` to disable | -| `REDIS_URL` | `redis://redis:6379` | Redis connection (set automatically in compose) | +| `REDIS_URL` | `redis://dragonfly:6379` | Dragonfly connection (set automatically in compose) | | `RESIDENTIAL_PROXY_URL` | — | Enables Tier 4 proxy escalation | ## Logs ```bash docker compose logs -f trawl -docker compose logs -f redis +docker compose logs -f dragonfly ``` ## Memory guide diff --git a/apps/docs/deployment/standalone.md b/apps/docs/deployment/standalone.md index bbeb9ca..a562544 100644 --- a/apps/docs/deployment/standalone.md +++ b/apps/docs/deployment/standalone.md @@ -15,19 +15,19 @@ The API image is published to GHCR on every push to `main`. Pull it directly — # Pull docker pull ghcr.io/germondai/trawl:latest -# Run (no Redis — session caching disabled, scraping still works) +# Run (no Dragonfly — session caching disabled, scraping still works) docker run -d \ --name trawl \ -p 8191:8191 \ --shm-size=1gb \ ghcr.io/germondai/trawl:latest -# Run (with external Redis) +# Run (with external Dragonfly, Redis-protocol compatible) docker run -d \ --name trawl \ -p 8191:8191 \ --shm-size=1gb \ - -e REDIS_URL=redis://your-redis-host:6379 \ + -e REDIS_URL=redis://your-dragonfly-host:6379 \ -e BROWSER_POOL_SIZE=3 \ ghcr.io/germondai/trawl:latest ``` diff --git a/apps/docs/deployment/troubleshooting.md b/apps/docs/deployment/troubleshooting.md index 0c19b25..74f181b 100644 --- a/apps/docs/deployment/troubleshooting.md +++ b/apps/docs/deployment/troubleshooting.md @@ -11,7 +11,7 @@ description: Common issues and how to fix them. **Causes:** -1. **Redis not reachable** — Check `REDIS_URL`. From inside Docker, use `redis://redis:6379` not `redis://localhost:6379`. +1. **Dragonfly not reachable** — Check `REDIS_URL`. From inside Docker, use `redis://dragonfly:6379` not `redis://localhost:6379`. 2. **Camoufox binary not installed** — The API Dockerfile runs `bunx camoufox-js fetch`. If this step was skipped (e.g. build cache reuse), rebuild: `docker compose build --no-cache api`. 3. **shm_size too small** — Ensure `shm_size: 1gb` is set on the API service. @@ -46,7 +46,7 @@ docker compose up -d --force-recreate **Causes:** -1. **Redis session data is not persisting** — Run `docker exec trawl-redis redis-cli keys "session:*"` after a successful scrape. If empty, the session cache write is failing. Check API logs for Redis errors. +1. **Dragonfly session data is not persisting** — Run `docker exec trawl-dragonfly redis-cli keys "session:*"` after a successful scrape. If empty, the session cache write is failing. Check API logs for Dragonfly connection errors. 2. **`SESSION_TTL_SECONDS` set too low** — If it's shorter than Cloudflare's challenge interval, the cache expires before the next request. 3. **Domain key mismatch** — The key is the hostname only. `sub.example.com` and `www.example.com` are separate sessions. @@ -95,8 +95,8 @@ Each Camoufox instance uses 350–500 MB. With 3 browsers, expect ~1.5 GB total. # Live API logs docker compose logs -f api -# Check Redis keys -docker exec trawl-redis redis-cli keys "*" +# Check Dragonfly keys +docker exec trawl-dragonfly redis-cli keys "*" # Test scrape endpoint curl -s -X POST http://localhost:8191/scrape \ diff --git a/apps/docs/getting-started/configuration.md b/apps/docs/getting-started/configuration.md index e2e4e06..53a1623 100644 --- a/apps/docs/getting-started/configuration.md +++ b/apps/docs/getting-started/configuration.md @@ -7,28 +7,28 @@ description: All environment variables for TRAWL, with defaults and examples. All configuration is via environment variables. Copy `.env.example` to `.env` and edit before starting. -## Redis +## Dragonfly ### `REDIS_URL` **Default:** `redis://localhost:6379` -Standard Redis connection URL. When running inside Docker Compose use the service name: +Standard Redis-protocol connection URL — TRAWL's default cache backend is [Dragonfly](https://www.dragonflydb.io/), which is wire-compatible with Redis, so the URL scheme and env var name are unchanged. When running inside Docker Compose use the service name: ```ini -REDIS_URL=redis://redis:6379 +REDIS_URL=redis://dragonfly:6379 ``` With authentication: ```ini -REDIS_URL=redis://:yourpassword@redis:6379 +REDIS_URL=redis://:yourpassword@dragonfly:6379 ``` With a specific database index: ```ini -REDIS_URL=redis://redis:6379/1 +REDIS_URL=redis://dragonfly:6379/1 ``` ## Browser Pool @@ -91,7 +91,7 @@ BROWSER_CONTENT_PROCESSES=4 # raise if CF/Imperva challenges stall **Default:** `3600` (1 hour) -How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache. +How long Cloudflare cookies are cached in Dragonfly per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache. Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3). @@ -176,7 +176,7 @@ Port the Nuxt landing page listens on. ## Full `.env.example` ```ini -# ── Redis ───────────────────────────────────── +# ── Dragonfly ───────────────────────────────── REDIS_URL=redis://localhost:6379 # ── Browser pool ────────────────────────────── diff --git a/apps/docs/getting-started/local-development.md b/apps/docs/getting-started/local-development.md index c96d4d2..45c4d3e 100644 --- a/apps/docs/getting-started/local-development.md +++ b/apps/docs/getting-started/local-development.md @@ -12,7 +12,7 @@ TRAWL is a Bun workspace monorepo. You can run each service locally with hot-rel | Tool | Version | Install | |------|---------|---------| | Bun | ≥ 1.2 | `curl -fsSL https://bun.sh/install \| bash` | -| Redis | 7+ | Docker (see below) or system package | +| Dragonfly | 1.39+ | Docker (see below) — Redis-protocol compatible | ## 1. Install dependencies @@ -33,13 +33,13 @@ bun x camoufox-js fetch This downloads the browser into the local cache. It only needs to run once per machine. -## 3. Start Redis +## 3. Start Dragonfly ```bash -docker run -d --name trawl-redis -p 6379:6379 redis:7-alpine +docker run -d --name trawl-dragonfly -p 6379:6379 docker.dragonflydb.io/dragonflydb/dragonfly:v1.39.0 ``` -Or use a local Redis installation if you have one. +Or use any Redis-protocol-compatible server you already have — Dragonfly speaks the same wire protocol. ## 4. Create your `.env` @@ -92,8 +92,8 @@ trawl/ │ ├── types/ Shared TypeScript interfaces — no runtime logic │ ├── browser/ BrowserPool + SessionCache │ └── tiers/ Tier 1–4 executors + orchestrator -├── docker-compose.yml scraper + Redis (default) -├── docker-compose.minimal.yml scraper only, no Redis +├── docker-compose.yml scraper + Dragonfly (default) +├── docker-compose.minimal.yml scraper only, no Dragonfly ├── docker-compose.prod.yml production with restart + healthcheck └── docker-compose.full.yml full stack including web + docs ``` diff --git a/apps/docs/getting-started/quick-start.md b/apps/docs/getting-started/quick-start.md index 9b1cf2e..a458d00 100644 --- a/apps/docs/getting-started/quick-start.md +++ b/apps/docs/getting-started/quick-start.md @@ -32,7 +32,7 @@ This starts three containers: | Container | Purpose | Port | | --------- | ----------------------- | -------- | -| `redis` | Session cache backend | internal | +| `dragonfly` | Session cache backend | internal | | `api` | Browser pool + HTTP API | 8191 | | `web` | Landing page & docs UI | 3000 |