External/trawl
1
0
Fork 0
mirror of https://github.com/germondai/trawl.git synced 2026-08-17 12:11:23 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: update API, architecture, proxy, deployment, and integration guides

Browse Source
This commit is contained in:
germondai
2026-08-02 02:35:35 +02:00
parent 357767ce3c
commit 2fc1031874
13 changed files with 121 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/docs/getting-started/configuration.md
+2 -1
View File
@@ -101,7 +101,8 @@ These bounds keep an unresponsive Firefox process from permanently consuming a p
**Default:** `3600` (1 hour)
How long Cloudflare cookies are cached in Redis per domain. After this TTL the next request to the domain triggers a fresh challenge solve (Tier 3) and refreshes the cache.
How long solved browser cookies and user-agent state are cached in Redis per domain. After this TTL
the next protected request triggers a fresh challenge solve (Tier 3) and refreshes the cache.
Cloudflare's `cf_clearance` cookie typically has a 30-minute expiry. Setting `SESSION_TTL_SECONDS` below 1800 wastes cache hits; setting it above 7200 risks replaying expired cookies (TRAWL handles this gracefully by invalidating the cache and falling back to Tier 3).
apps/docs/getting-started/quick-start.md
+13 -10
View File
@@ -28,15 +28,14 @@ Everything has a working default. See [Configuration](/getting-started/configura
docker compose up -d
```
This starts three containers:
This starts two containers:
| Container | Purpose | Port |
| --------- | ----------------------- | -------- |
| `redis` | Session cache backend | internal |
| `api` | Browser pool + HTTP API | 8191 |
| `web` | Landing page & docs UI | 3000 |
| `trawl` | Browser pool, API, and optional forward proxy | 8191, 8192 |
The API takes **15–30 seconds** on first boot while it launches the browser pool and downloads Camoufox (only on first build). Watch progress:
The API takes **15–30 seconds** to launch and warm the browser pool. Watch progress:
```bash
docker compose logs -f api
@@ -69,7 +68,9 @@ Expected:
"busy": 0,
"available": 3,
"restarts": 0,
"avgRestarts": 0
"avgRestarts": 0,
"stalled": 0,
"live": 3
}
}
```
@@ -86,10 +87,13 @@ curl -s -X POST http://localhost:8191/v1 \
}' | jq '{status, url: .solution.url}'
```
A `status: "ok"` response confirms TRAWL is working. The **second request to the same domain** returns in ~500ms from the session cache.
A `status: "ok"` response confirms TRAWL is working. Repeat requests can reuse the cached browser
session when the target still accepts it.
::: tip First request is slow — that's expected
The first time TRAWL sees a domain it solves the Cloudflare challenge fresh (4–15s). Every subsequent request uses the cached session and returns in under 500ms. This is the whole point.
::: tip Challenge requests take longer
An unprotected target may finish in Tier 1 without opening a browser. A recognized Cloudflare,
Akamai, or Imperva wall escalates to a fresh browser flow. Later requests can use Tier 2 while the
saved session remains valid.
:::
## 6. Use the native API (optional)
@@ -118,8 +122,7 @@ Done. No other configuration changes are needed. TRAWL implements the FlareSolve
```
localhost:8191 → TRAWL API (FlareSolverr-compatible + native endpoint)
localhost:3000 → Web UI (landing page, visible in your browser)
localhost:3001 → Docs (this documentation, served locally)
localhost:8192 → HTTP/HTTPS forward proxy (when MITM_PROXY_ENABLED=true)
```
To stop everything: