trawl/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Fixed

• Translate Prowlarr's serialized headers.contentType metadata at the FlareSolverr /v1 compatibility boundary and discard contentLength, allowing form POST requests to enter the scraper pipeline (#50).
• Bound Camoufox memory growth by counting every Tier 3/4 temporary context and rolling-replacing browsers at BROWSER_RECYCLE_AFTER_CONTEXTS, while keeping existing capacity available during warm-up. Replacement launches are serialized, cleanup is timeout-bounded, and failed launches retain the usable browser (#52).

[1.3.1] - 2026-08-02

Fixed

• Keep pooled browser contexts under BrowserPool ownership so repeat Tier 2 requests cannot reuse a context closed by a separate cache (#45).

Changed

• Bump all application and internal package versions to 1.3.1.

[1.3.0] - 2026-08-02

Added

• Detect and resolve Akamai Bot Manager behavioral interstitials across scraper tiers and the HTTP/HTTPS proxy (#33).
• Document supported upstream proxy formats, local Compose configuration, and residential proxy pools (#26).

Fixed

• Reject non-object request bodies and missing, non-string, or blank url values with HTTP 400 before scraper-tier execution (#34).
• Recover stalled browser checkouts and bound browser close/launch operations so wedged Firefox processes cannot silently exhaust the pool (#36, #37, #48).
• Report HTTP 503 from /health whenever the browser pool has no live capacity.
• Pass authenticated proxy credentials to Firefox separately from the proxy server URL in Tier 3 and Tier 4 (#40).
• Return complete Tier 1 text responses instead of the 4 KiB challenge-detection preview (#46, #47).
• Record the correct architecture-specific Camoufox release metadata in API images.

Changed

• Bump all application and internal package versions to 1.3.0.
• Update workspace dependencies to their latest compatible releases. TypeScript remains on 5.9 for the Nuxt app until vue-tsc supports TypeScript 7.

[1.2.0] - 2026-07-26

Added

• Tier 0 direct forward in the MITM proxy (apps/api/src/proxy/directForward.ts): the proxy at :8192 now forwards requests directly to upstream via raw TCP/TLS instead of spinning up a browser for every request. Pool is reserved for the requests that actually need CF bypass; Netflix/YouTube/banks/etc. flow at near-direct speed.
• Smart adaptive streaming (apps/api/src/proxy/streaming.ts): small JSON/HTML/text responses are buffered so the challenge detector can inspect them; video/audio/binary files (.mp4, .mkv, .m3u8, .zip, .exe, .dmg, etc.) are streamed straight through to the client. Default threshold: 8 MiB.
• Per-hostname challenge cache (apps/api/src/proxy/challengeCache.ts): the proxy remembers which hostnames recently returned Cloudflare challenges and sends repeat visits directly to the tiered solver for 5 minutes.
• Persistent browser context cache: solved sessions retain their browser fingerprint, cookies, cache, and storage across proxy requests, with bounded per-proxy reuse and cleanup.
• General HTTP/HTTPS proxy support: CONNECT tunneling, plain HTTP absolute-form requests, WebSocket upgrades, request bodies, redirects, authentication headers, cookies, and HTTP Range/206 responses are forwarded with their required semantics.
• proxySanitizeHeaders() (packages/tiers/src/utils/sanitize.ts): permissive header sanitizer for the transparent MITM proxy — passes Authorization, Cookie, Range, User-Agent, Referer, and custom API tokens through, strips only RFC 7230 hop-by-hop headers.
• Raw bytes in @trawl/tiers: ScrapeResult and tier results now expose body?: Uint8Array, responseHeaders?: Record<string,string>, and contentType?: string alongside the existing html field. The proxy uses these fields to preserve binary content without HTML normalization.
• Tier 1 method handling fix: runTier1() now forwards the request body for PUT, PATCH, DELETE, QUERY (was only POST).
• Graceful proxy shutdown (shutdownMitmProxy()): the API captures the proxy handle on startup and lifecycle.ts calls shutdownMitmProxy() on SIGTERM/SIGINT before the browser pool shutdown, so in-flight connections drain.
• MITM proxy port + CA volume in docker-compose: 8192:8192 port mapping, MITM_PROXY_* env vars, and a persistent trawl_proxy_ca volume so the CA survives container restarts.
• Proxy tests: header sanitization, response policy, adaptive streaming, challenge caching, direct HTTP forwarding, Range/206 handling, chunked responses, compressed challenge detection, and explicit media streaming.
• Proxy documentation: architecture, configuration, client setup, and CA installation guides for operating systems, browsers, Java/JDownloader, containers, and application-specific trust stores.

Changed

• packages/types/src/index.ts — ScrapeResult extended additively with optional body, responseHeaders, and contentType; native /scrape consumers should tolerate these additional fields.
• apps/api/src/proxy/server.ts — fetchRaw/reissue replaced by proxyRequest() (Tier 0 with scrape() fallback) for both CONNECT-based HTTPS and plain HTTP traffic.
• lifecycle.ts — registerLifecycleHandlers() accepts an optional { onShutdown } callback.
• apps/api/src/index.ts — captures the proxy handle on startup and wires it into registerLifecycleHandlers.
• All application and internal package versions bumped to 1.2.0.

[1.1.0] - 2026-07-22

Added

• Browser-backed MITM forward-proxy mode (MITM_PROXY_ENABLED, off by default): HTTP(S) forward proxy that re-issues requests through the browser pool so clients like Prowlarr, Jackett, JDownloader, and changedetection.io can hit fingerprint-bound Cloudflare sites that the /v1 cookie handoff cannot. The generated root CA is persisted, per-host certificates are minted in memory, and the root is downloadable at GET /proxy-ca.crt. New env: MITM_PROXY_{ENABLED,PORT,HOST,CA_DIR,MAX_TIER,DEBUG}.

Changed

• fetchRaw rotates proxyPool on Cloudflare challenge (same markBad → next() pattern as Tier 3) instead of retrying on the same IP.
• Main MITM proxy listener supports a configurable bind address through MITM_PROXY_HOST.
• ci.yml runs on PRs targeting dev in addition to main.
• publish.yml inspects the actually-pushed tag from docker/metadata-action instead of re-deriving from github.sha (which previously mismatched the 7-char short SHA).
• node-forge ^1.3.1 runtime dep for CA + per-host leaf cert generation.

[1.0.1] - 2026-07-18

Changed

• packages/browser/src/pool.ts — renamed Firefox prefs key from prefs (silently ignored by camoufox-js@0.11.1) to firefox_user_prefs (which camoufox-js maps to Playwright's firefoxUserPrefs). The prefs are now actually applied.
• packages/browser/src/pool.ts — added the safe-only subset of Firefox prefs: telemetry off (datareporting.*, toolkit.telemetry.*, app.crashreporter, breakpad.*), dead UI features off (extensions.screenshots.*, browser.sessionstore.max_tabs_undo), dead network services off (browser.safebrowsing.*, extensions.update.*, browser.fixup.alternate.*, app.normandy.*, app.shield.*, network.connectivity-service.*, network.captive-portal-service.*, network.prefetch-next, beacon.enabled), security.OCSP.enabled: 0, and tightened network timeouts (tls-handshake-timeout: 30, connection-timeout: 60, response.timeout: 120). None of these touch the JS/CSS fingerprint surface.
• apps/api/Dockerfile — stage-3 prune of apt cache + /usr/share/{locale,doc,man} (image-size win, runtime-neutral).
• apps/api/Dockerfile — added 9 Bun runtime ENV flags (BUN_DISABLE_CJS=1, BUN_DEBUG=0, BUN_DISABLE_SOURCEMAPS=1, BUN_HTTP_KEEPALIVE=0, BUN_AGENT_DISABLE=1, BUN_INSPECT=0, BUN_LOCKFILE_MIGRATION=false, MIMALLOC_PURGE_DELAY=0, NODE_NO_WARNINGS=1). All verified runtime-neutral in smoke tests.
• packages/browser/package.json — moved patchright + playwright-core from dependencies to devDependencies (build hygiene; camoufox-js bundles both transitively at runtime).
• All packages bumped to 1.0.1.

Added

• scripts/bench-targets.sh, scripts/bench-success-rate.sh, scripts/bench-compare.sh — observability harnesses for measuring CF challenge latency + bypass success rate.

[1.0.0] - 2026-07-10

Changed

• Shared types (BrowserHandle, BrowserFingerprint, SupportedMethod) centralized instead of being duplicated per-package
• packages/tiers split into tiers/ and utils/, deduplicating cookie and network-failure helpers; apps/api's entrypoint split into config, deps, and routes, adding a proper root status route
• Evaluated switching the cache backend to Dragonfly and reverted: benchmarking showed Dragonfly only wins throughput when load is spread across multiple connections, but packages/browser's SessionCache holds a single shared RedisClient connection for the process lifetime — so at TRAWL's actual access pattern, plain Redis is faster at every scale tested, regardless of BROWSER_POOL_SIZE. Docker Compose configs (docker-compose.yml, .full.yml, .prod.yml) now pin redis:8.8-alpine; the redis service name and REDIS_URL env var are unchanged from before the Dragonfly experiment.

Added

• Landing page shows a live GitHub star count

[0.7.0] - 2026-07-08

Added

• Audio STT fallback for the hCaptcha solver

Fixed

• JS-only challenge pages that only look like plain HTML now correctly escalate from Tier 1 to the browser tiers (#22, #23)

[0.6.0] - 2026-07-08

Added

• BROWSER_RECYCLE_AFTER_CONTEXTS env var (default 8, set 0 to disable) bounds long-running browser process growth by recycling the pooled Camoufox/Firefox instance after a configurable number of Tier 3/Tier 4 temporary context creations
• BROWSER_CONTENT_PROCESSES env var (default 2) caps Firefox content processes per pooled browser via the dom.ipc.processCount Firefox pref. Firefox's default of 8 lets thread count climb when Tier 3/Tier 4 churn disposable contexts (see #13). The cap bounds the leak at the source without needing to restart the browser.
• Browser fingerprints now randomize OS/screen/window per instance and match the HTTP User-Agent to the emulated platform

Changed

• BROWSER_RECYCLE_AFTER_CONTEXTS no longer recycles preemptively after every N temporary contexts. The pool now recycles only when Tier 3 or Tier 4 returns a blocked / needs-js outcome, preserving cookies, cf_clearance, and warm fingerprint state across successful solves. This eliminates the HTTP-429 storm observed in single-browser setups where the previous "recycle every N uses" logic left the only browser restarting=true for ~13s during every recycle window (#17, thanks @CoolDotty)
• Tier detection now recognizes more block/error page variants; Tier 4 gains full captcha parity, with proxy/timing info surfaced in responses (#19, thanks @edasque)

Fixed

• Missing curl in the API runtime image broke healthchecks (#20, #21)
• Missing GeoLite2 mmdb caused a GeoIP startup crash on boot; now baked into the image (#20, #21)

[0.5.0] - 2026-07-06

Added

• Native method + body support across all four scraper tiers — the FlareSolverrRequest.cmd=request.post body is now actually delivered upstream instead of being silently dropped (thanks @whoshoe for the original POST support)
• ScrapeRequest.method accepts the full standard verb set: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, TRACE, QUERY (RFC 9341). CONNECT is intentionally excluded (tunneling verb, inappropriate for a proxy)
• POST / * request bodies are forwarded uncapped — operators who want a byte ceiling should impose it at their ingress / fronting proxy
• Body-bearing requests require a Content-Type header; the tier functions no longer auto-inject application/x-www-form-urlencoded, which previously mislabelled JSON / XML bodies
• ScrapeRequest field renamed from postData → body for REST-idiomatic naming. (FlareSolverrRequest.postData is unchanged because it's the upstream wire contract.)
• Native Imperva/Incapsula WAF challenge detection and solving in Tier 3 and Tier 4
• Proxy rotator reworked into a sticky, failure-aware pool with per-request override support
• PORT_API env var renamed to PORT and made properly configurable (#9, #10)

Security

• Reserved-name header denylist prevents callers from spoofing cf_clearance cookies, overriding the per-tier User-Agent, or rewriting routing signals (X-Forwarded-For, Host) during a POST bypass flow

Fixed

• /v1 now accepts Prowlarr's Cardigann FlareSolverrProxy object shape ({url, username, password}) for the per-request proxy field, instead of crashing with proxy.server: expected string, got object when Prowlarr sends it through (#12, #15). The boundary normalises both the object form and a plain URL string into a single URL string before the orchestrator forwards it to Playwright/Camoufox. Credentials are URL-encoded so embedded @/: characters survive the round-trip.

Limitations

• The Playwright page.route(url, …) interceptor only handles the first top-frame GET to that exact URL. Server redirects to a different URL, XHR sub-resources, and chained POST→POST form flows do not have the postData override applied
• No idempotency-key support; transient network failures and pool churn can re-fire a POST (separate ticket)

Tests

• packages/tiers/tests/sanitize.test.ts — header sanitiser, method allowlist, postData size cap, Content-Type enforcement
• packages/tiers/tests/runTier1Post.test.ts — tier1 GET/POST round-trip and User-Agent non-override
• Run via bun --cwd packages/tiers test

[0.4.0] - 2026-07-01

Added

• :baseline Docker image variant for pre-AVX2 CPUs and older kernels, published to its own GHCR tag — confirmed working on a Synology DS920+ (DSM 7.3.2, kernel 4.4.302), see #1

Fixed

• Docker healthchecks failing, root-caused to wget vs. the runtime image; switched to a curl-based healthcheck with a proper timeout and start period (#3, #4)
• Startup crash loop (EISDIR, missing memoirist/camoufox-js) fixed by switching bun install to --linker=hoisted (#1, #6)
• /health now correctly returns 503 while the browser pool is still initializing

[0.3.0] - 2026-06-30

Added

• Configurable browser pool concurrency limiter to guard against OOM under burst load
• BROWSER_ACQUIRE_TIMEOUT_MS env var, default 15s

Changed

• A saturated browser pool now returns HTTP 429 instead of a raw 500
• Default BROWSER_POOL_SIZE raised to 3

[0.2.0] - 2026-06-26

Added

• Custom headers field on ScrapeRequest, forwarded through Tier 1-4 via URL-scoped route interception and exposed on /v1 and /scrape with CORS support
• cmd is now optional on /v1, defaulting to request.get

Changed

• Multi-arch Docker publish now runs on native arm64 runners with a two-phase per-digest build and manifest merge

[0.1.0] - 2026-06-26

Added

• Initial release with 4-tier execution engine
• Native captcha solving for Cloudflare Turnstile, reCAPTCHA v2 (audio STT), hCaptcha (audio bypass), and GeeTest v3 slider
• Persistent browser pool with real Camoufox Firefox
• Session caching via Redis
• FlareSolverr v2-compatible /v1 endpoint
• WebSocket live scrape streaming at /scrape/live
• Self-healing browser pool with automatic restart on crash
• Sticky domain routing to maximize session cache hits
• Nuxt 4 landing page with live stats
• VitePress documentation site
• Docker Compose deployment with amd64/arm64 platform targeting