Germond and GitHub
d873e2662c
Merge pull request #23 from germondai/22-bug-js-only-challenge-pages-bypass-tier-1-escalation
...
22 bug js only challenge pages bypass tier 1 escalation
2026-07-08 20:06:24 +02:00
germondai
b72103bba7
chore: add hCaptcha audio solver smoke test script
2026-07-08 19:40:56 +02:00
germondai
4dfe5be680
fix(tiers): escalate JS-shell challenge pages from Tier 1 to browser
2026-07-08 19:40:38 +02:00
germondai
16499ce2fc
feat(tiers): add audio STT fallback to hCaptcha solver
2026-07-08 19:40:24 +02:00
Germond and GitHub
9142838c46
Merge pull request #21 from germondai/20-bug-trawl-fails-to-start---issues-with-healtcheck
...
20 bug trawl fails to start issues with healtcheck
2026-07-08 16:05:58 +02:00
germondai
7f2dc4fb8e
docs(web): bump docker image size to 1.14 GB in comparison table
2026-07-08 15:48:48 +02:00
germondai
9fb92d9809
fix(docker): bake GeoLite2 mmdb into api image to prevent geoip startup crash
2026-07-08 15:48:48 +02:00
germondai
52cd0ce3ac
fix(docker): install curl in api runtime image so healthcheck works
2026-07-08 15:48:48 +02:00
germondai
973505b7f3
chore(browser): suppress noExplicitAny lint in pool.ts
2026-07-08 03:49:12 +02:00
Germond and GitHub
a8eaf8f843
Merge pull request #19 from edasque/fix/tier-detection-and-proxy-wiring
...
Fix silent false-successes in tier detection, add Tier 4 captcha parity, surface proxy/timing info
2026-07-08 03:36:55 +02:00
germondai
1da2a021b4
docs(web): update comparison benchmarks with measured latency + infra specs
2026-07-08 01:25:00 +02:00
Erik Dasque
7d3204351c
fix(tiers): recognize more block/error page variants, add Tier 4 captcha parity, surface proxy/timing info
...
Found while running trawl against a large batch of real-world URLs: several
cases where the API returned 200 with content that was actually a blocked
page, an empty challenge stub, or Firefox's own error page. Each was a
detection gap where a tier didn't recognize the failure and reported it as a
successful scrape.
- Recognize Firefox's about:neterror/about:certerror page (browser never
reached a server), Cloudflare's static "you have been blocked" WAF-deny
page, and a lean CF challenge stub (blank title/body, just the bootstrap
script) — the stub check is gated on page size since the same script
snippet also appears on ordinary, fully-loaded CF pages as bot-management
telemetry.
- Wire the existing isBlocked() status-code check (403/429/202) into Tiers 2
and 3 — previously only Tier 1 checked status code, so a generic non-CF WAF
deny that escalated to a browser tier was reported as a success.
- Bring Tier 4 up to parity with Tier 3: captcha solving and the same block
detection. Sites that need Tier 4 for IP reputation can just as easily have
an in-page captcha widget.
- Add proxyUsed: boolean to the response, set from the actual proxy used by
the winning tier — previously the only signal was inferring from tier === 4,
which doesn't distinguish "no proxy" from Tier 3's datacenter proxy.
- Attach the per-tier timings array to thrown errors via a new ScrapeError,
and return it in /scrape's error response. The array was already being
built in memory; it just never survived the throw, so failed requests gave
a flat error string with no way to see which tier failed or why.
- Add process-level uncaughtException/unhandledRejection handlers. One target
site's page threw a JS error that Camoufox/Firefox reports in a shape
playwright-core's dispatcher doesn't expect, which crashed the entire
process and dropped every in-flight request across all clients.
- Update the native API docs for the new response fields and error shape.
All additive — no existing fields changed shape. Full existing test suite
passes (58/58), and this is rebuilt/smoke-tested against latest dev.
2026-07-07 15:58:54 +00:00
germondai
d7476274a2
feat(browser): randomize OS/screen/window per browser and match HTTP UA to platform
2026-07-07 02:32:12 +02:00
germondai
f44f7511f9
build(api): prune devDeps and impit platform binaries in deps stage
2026-07-06 23:41:53 +02:00
germondai
fb620141f6
build(api): pin Camoufox version, strip symbols, prune deps, switch to debian-slim
2026-07-06 23:10:16 +02:00
germondai
1d818f0498
chore: tighten .dockerignore to drop local agent IDE and unused Dockerfile sources
2026-07-06 22:47:37 +02:00
germondai
e42582e6b6
ci: use native bun test instead of npm-script wrapper in CI
2026-07-06 20:52:59 +02:00
Germond and GitHub
91b49d593f
Merge pull request #18 from germondai/17-improve-browser-pool-avoid-cold-starts-and-pool-exhaustion-429s-from-browser_recycle_after_contexts
...
Improve browser pool: recycle-on-blocked + cap content processes
2026-07-06 20:47:23 +02:00
germondai
444df08d00
docs: update CHANGELOG, README, and docs for recycle-on-blocked and content-processes
2026-07-06 20:11:08 +02:00
germondai
2560f9157a
feat(scripts): add automated soak test for repeatable perf measurement
2026-07-06 20:10:56 +02:00
germondai
4082c0f3c3
ci: run typecheck and tests in CI workflow
2026-07-06 20:10:42 +02:00
germondai
bb062c6bd8
test(browser): cover recycle-on-blocked and contentProcesses options
2026-07-06 20:10:19 +02:00
germondai
7b57cd4f80
refactor(tiers): add normalizeSameSite helper for cookie sameSite
2026-07-06 20:10:13 +02:00
germondai
da8f9261e1
feat(api): add BROWSER_CONTENT_PROCESSES env var
2026-07-06 20:09:30 +02:00
germondai
a307fe835d
feat(browser): cap Firefox content processes via dom.ipc.processCount
2026-07-06 20:08:56 +02:00
germondai
1c90df92d0
feat(tiers): recycle browser only when tier returns blocked
2026-07-06 20:08:32 +02:00
germondai
5d0074c540
docs: add changelog entry for browser recycle env var
2026-07-06 19:43:21 +02:00
Germond and GitHub
47fc0fa647
Merge pull request #14 from CoolDotty/codex/fix-browser-process-leak
...
Recycle browser pool after temporary contexts
2026-07-06 19:41:42 +02:00
Germond and GitHub
d0877c5344
Merge pull request #15 from germondai/12-bug-500-internalservererror-in-prowlarr
...
12 bug 500 internalservererror in prowlarr
2026-07-06 14:59:06 +02:00
germondai
e8129735b0
fix(api): normalize proxy field at /v1 boundary for Prowlarr input
2026-07-06 13:56:09 +02:00
germondai
7ade0665b3
docs: note Prowlarr proxy object form fix in changelog
2026-07-06 13:55:33 +02:00
germondai
1e5acb7975
test(tiers): cover normalizeProxy for string and object inputs
2026-07-06 13:55:25 +02:00
germondai
4adf1e0347
feat(tiers): add normalizeProxy helper for proxy field normalization
2026-07-06 13:55:02 +02:00
germondai
ff1804bcb7
fix(types): accept Prowlarr object form for FlareSolverrRequest.proxy
2026-07-06 13:53:53 +02:00
dot
171f673f28
Table formatting
2026-07-06 01:15:16 -07:00
CoolDotty
68ad2e0f1a
Recycle browsers after temporary contexts
2026-07-06 00:47:32 -07:00
germondai
da8008d200
feat(api): forward per-request proxy override to scrape handler
2026-07-06 02:29:51 +02:00
germondai
2520345c48
feat(tiers): rotate proxy pool on blocked in tier 3 and 4
2026-07-06 02:29:31 +02:00
germondai
4057d7bdac
refactor(tiers): consolidate ProxyPool export and drop dead re-exports
2026-07-06 02:29:09 +02:00
Germond and GitHub
79f56b68fe
Merge pull request #11 from germondai/fix/pr-8-hardening
...
feat: harden POST/method support with full verb set and validation
2026-07-06 01:33:38 +02:00
Germond and GitHub
8d69e660c3
Merge branch 'main' into fix/pr-8-hardening
2026-07-06 01:33:13 +02:00
germondai
59f78f38e7
docs: document method/body support and header sanitisation
2026-07-06 01:12:30 +02:00
germondai
ab8c34c216
test: add e2e probe script for method and validation variants
2026-07-06 01:12:30 +02:00
germondai
ad7764eeb6
test(tiers): cover header sanitiser and tier1 post support
2026-07-06 01:12:30 +02:00
germondai
d8e0745662
feat(api): validate methods and sanitise headers at the boundary
2026-07-06 01:12:11 +02:00
germondai
06783038e4
feat(tiers): thread method and body through all tiers
2026-07-06 01:12:11 +02:00
germondai
abee1e5063
feat(tiers): add header sanitisation and request validation
2026-07-06 01:12:06 +02:00
germondai
7c87dc0b1a
feat(types): support full HTTP method set and rename postData to body
2026-07-06 01:12:06 +02:00
Germond and GitHub
7be13e88e1
Merge pull request #10 from germondai/9-unable-to-use-any-other-port-numbers-other-than-8191
...
9 unable to use any other port numbers other than 8191
2026-07-05 18:15:36 +02:00
germondai
9261d5d92f
docs: update references to renamed PORT env var
2026-07-05 18:06:12 +02:00
