mirror of
https://github.com/germondai/trawl.git
synced 2026-08-17 12:11:23 +02:00
71 lines
2.3 KiB
Markdown
71 lines
2.3 KiB
Markdown
---
|
||
title: Session Cache
|
||
description: How TRAWL caches Cloudflare cookies in Redis to make repeat requests fast.
|
||
---
|
||
|
||
# Session Cache
|
||
|
||
The session cache is what makes Tier 2 possible. After every successful Tier 3 solve, the extracted Cloudflare cookies are saved to Redis. The next request to the same domain injects those cookies into a browser context, skipping the challenge entirely.
|
||
|
||
## Storage format
|
||
|
||
Key: `session:{hostname}` (e.g. `session:nowsecure.nl`)
|
||
|
||
Value (JSON):
|
||
```typescript
|
||
interface SessionData {
|
||
cookies: Cookie[]
|
||
userAgent: string
|
||
savedAt: number // unix timestamp ms
|
||
}
|
||
```
|
||
|
||
TTL: `SESSION_TTL_SECONDS` (default 3600 seconds / 1 hour).
|
||
|
||
## Session key
|
||
|
||
The key is the **hostname only** — no path, no port, no protocol. This means all pages on a domain share one session:
|
||
|
||
```
|
||
https://example.com/ → session:example.com
|
||
https://example.com/page → session:example.com (same key)
|
||
https://sub.example.com/ → session:sub.example.com (different key)
|
||
```
|
||
|
||
Subdomains have separate sessions because Cloudflare can issue different challenge cookies per subdomain.
|
||
|
||
## Lifecycle
|
||
|
||
```
|
||
Tier 3 succeeds
|
||
│
|
||
├── extract cookies from browser context
|
||
├── REDIS SET session:hostname → JSON EX SESSION_TTL_SECONDS
|
||
│
|
||
└── next request to same domain:
|
||
REDIS GET session:hostname
|
||
├── hit → Tier 2: inject cookies, navigate (500ms)
|
||
└── miss → Tier 3: fresh solve, save to cache
|
||
```
|
||
|
||
## Invalidation
|
||
|
||
If Tier 2 navigates with the cached cookies and the result is still a Cloudflare interstitial (the session expired before Redis TTL), the orchestrator:
|
||
|
||
1. Calls `sessionCache.invalidate(domain)` — deletes the Redis key
|
||
2. Escalates to Tier 3 to get a fresh session
|
||
|
||
This handles the case where Cloudflare's `cf_clearance` cookie (30-minute expiry) expires before the Redis TTL does.
|
||
|
||
## Redis client
|
||
|
||
TRAWL uses `new RedisClient(REDIS_URL)` from Bun's native Redis client (not ioredis). Bun's Redis client is ~7.9× faster than ioredis for simple GET/SET operations, which matters since every request does at least one Redis read.
|
||
|
||
```typescript
|
||
import { RedisClient } from 'bun'
|
||
|
||
const redis = new RedisClient('redis://localhost:6379')
|
||
await redis.set('session:example.com', JSON.stringify(data), 'EX', 3600)
|
||
const raw = await redis.get('session:example.com')
|
||
```
|