The FlareSolverr /v1 contract only returns cookies + user-agent. Clients like
Prowlarr take those and re-fetch the target with their own HTTP stack, which is
re-challenged on sites whose Cloudflare clearance is bound to the solving
browser's connection fingerprint (e.g. 1337x) — no cookie is portable to a
plain HTTP client, so those indexers can't be used at all.
Add an optional HTTP(S) forward proxy (MITM_PROXY_ENABLED). Point such a client's
proxy at it (per-indexer HTTP proxy in Prowlarr) and every request — search and
the .torrent/magnet grab — is transparently re-issued through the browser pool,
returning the RAW response bytes so binary downloads pass through intact.
- ca.ts: self-generated CA (persisted) + on-demand per-host leaf certs
- server.ts: per-host loopback-TLS termination (Bun's node:tls can't drive a
handshake via emit("connection") or honor SNICallback, so one listening TLS
server per host is the reliable path); raw-byte capture via page.goto response
body, with the download-event path for binaries; scrape() fallback solves CF
- /proxy-ca.crt route to fetch the CA for the client's trust store
- New env: MITM_PROXY_{ENABLED,PORT,CA_DIR,MAX_TIER,DEBUG}
Off by default; localhost-only by design (a MITM proxy can impersonate any host
to a client that trusts its CA).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Splits the single [Unreleased] CHANGELOG block into dated 0.1.0-1.0.0
sections matching the milestone commits being tagged for issue #24
(numeric release tags), and bumps every package.json to 1.0.0.
Found while running trawl against a large batch of real-world URLs: several
cases where the API returned 200 with content that was actually a blocked
page, an empty challenge stub, or Firefox's own error page. Each was a
detection gap where a tier didn't recognize the failure and reported it as a
successful scrape.
- Recognize Firefox's about:neterror/about:certerror page (browser never
reached a server), Cloudflare's static "you have been blocked" WAF-deny
page, and a lean CF challenge stub (blank title/body, just the bootstrap
script) — the stub check is gated on page size since the same script
snippet also appears on ordinary, fully-loaded CF pages as bot-management
telemetry.
- Wire the existing isBlocked() status-code check (403/429/202) into Tiers 2
and 3 — previously only Tier 1 checked status code, so a generic non-CF WAF
deny that escalated to a browser tier was reported as a success.
- Bring Tier 4 up to parity with Tier 3: captcha solving and the same block
detection. Sites that need Tier 4 for IP reputation can just as easily have
an in-page captcha widget.
- Add proxyUsed: boolean to the response, set from the actual proxy used by
the winning tier — previously the only signal was inferring from tier === 4,
which doesn't distinguish "no proxy" from Tier 3's datacenter proxy.
- Attach the per-tier timings array to thrown errors via a new ScrapeError,
and return it in /scrape's error response. The array was already being
built in memory; it just never survived the throw, so failed requests gave
a flat error string with no way to see which tier failed or why.
- Add process-level uncaughtException/unhandledRejection handlers. One target
site's page threw a JS error that Camoufox/Firefox reports in a shape
playwright-core's dispatcher doesn't expect, which crashed the entire
process and dropped every in-flight request across all clients.
- Update the native API docs for the new response fields and error shape.
All additive — no existing fields changed shape. Full existing test suite
passes (58/58), and this is rebuilt/smoke-tested against latest dev.
germondai