Sideloaded Widevine works on the first launch of a persistent context
instead of needing a manual two-launch hint-file workaround. The wrapper
writes Chromium's CDM hint file into the profile before launch when a
WidevineCdm directory is present next to the binary.
- New cloakbrowser/widevine.py and js/src/widevine.ts: resolve a sideloaded
CDM (CLOAKBROWSER_WIDEVINE_CDM env var, else next to the binary) and seed
the hint file. Linux only; no-op elsewhere. CLOAKBROWSER_WIDEVINE=0 disables.
- Never bundles/downloads/copies the CDM (proprietary); seeds only when the
user-provided CDM is already present.
- Wired into launch_persistent_context[_async] and launchPersistentContext.
- README + js/README: Widevine / DRM section, env vars, FPJS tradeoff note.
- Tests: tests/test_widevine.py, js/tests/widevine.test.ts, persistent-context
integration assertions.
Mirror the JS fix from #303 in the sync and async Python actionability
checks: compute and apply the iframe coordinate offset before
elementFromPoint, and fail open when the check itself cannot run. Add
fail-open regression tests for both Python and JS.
The ElementHandle pointer-events check passed page-space click coordinates to elementFromPoint, which runs inside the target element's frame. For elements in an iframe the coordinate spaces differ, so the check looked at the wrong point and wrongly reported the element as covered. Now the iframe offset is computed and applied. Also fails open when the check itself cannot run.
Thanks @eofreternal for the fix.
Frame-level methods (click, dblclick, hover, dragAndDrop) passed the raw
timeout to each sequential operation independently, causing 3x actual
wait time when elements don't exist. ElementHandle methods had a similar
2x issue between actionability and pointer-events checks.
Port the deadline + remainingMs() pattern already used by page-level
methods. Also fix bot detection test selector after site added a hidden
duplicate submit button.
Bypass Playwright's CDP Fetch.authRequired interceptor for authenticated
HTTP proxies by passing inline credentials via Chrome's --proxy-server
flag. Chrome sends Proxy-Authorization preemptively, avoiding the 407
round-trip that breaks on some proxies and Google domains (#182).
Gated on platform (linux-x64, windows-x64) and binary version >= 146.0.7680.177.5.
Unsupported platforms fall back to Playwright's proxy dict.
Puppeteer falls back to page.authenticate() on unsupported platforms.
On `docker restart`, `/tmp` is preserved across container instances, so the
previous Xvfb's `/tmp/.X99-lock` survives into the new container. The new
Xvfb sees the existing lock and refuses to start, leaving the container
with no X server. Every Chrome launch then dies with "Missing X server or
$DISPLAY", and `cloakserve` returns 502 from `/json/version` forever.
Any orchestrator that restarts on unhealthy (the README-recommended
healthcheck + `restart: always`, autoheal sidecars, etc.) then enters a
permanent restart loop because every restart hits the same broken state.
This is silent for first-time users: the container appears to start
successfully (Xvfb did launch *once*), then degrades only after the first
restart. The fix is one line in the entrypoint: remove the stale lock
before starting Xvfb.
Fixes#283
Expose userDataDir support via launchPersistentContext() for
cloakbrowser/puppeteer, matching the existing Playwright API.
Includes proxy auth, geoip, and humanize support.
Add `extension_paths` parameter to all launch functions (Python + JS) for loading Chrome extensions.
Resolves paths to absolute, injects `--load-extension` and `--disable-extensions-except` flags via `build_args()`.
Note: Extensions require a persistent context (`launch_persistent_context`) to function — this is a Chromium limitation.
Co-authored-by: zackycodes <75211659+zackycodes@users.noreply.github.com>
* feat: add flake.nix
* refactor: improve code style and add more information to flake.nix
* chore(nix): ignore build result symlink
* chore(nix): use unversioned pytest packages
Detected by Aeon + osv-scanner.
Severity: high (runtime tar) / high+moderate (dev deps)
Patches 8 of 14 CVEs flagged by osv-scanner — all that can be fixed
within current semver ranges via `npm audit fix --package-lock-only`.
The remaining 6 are gated on a puppeteer-core/vitest major-version
bump (out of scope for this PR).
Runtime (shipped to users):
- tar 7.5.9 -> 7.5.15
- GHSA-9ppj-qmqm-q256 HIGH: Symlink Path Traversal via Drive-Relative Linkpath
- GHSA-qffp-2rhf-9h96 HIGH: Hardlink Path Traversal via Drive-Relative Linkpath
- Reachable in js/src/download.ts (extractTar) — the existing filter() rejects
absolute paths and "..", but does not inspect linkpath, so a malicious
Chromium tarball could write outside the cache dir on Windows.
Dev (build-time only):
- basic-ftp 5.2.0 -> 5.3.1 (4 HIGH: CRLF injection x2, DoS x2)
- ip-address 10.1.0 -> 10.2.0 (1 MOD: XSS in Address6 HTML methods)
- postcss 8.5.6 -> 8.5.14 (1 MOD: XSS via unescaped </style>)
Lockfile metadata side-effects (npm-regenerated, not editorial):
- name@version block synced from package.json (0.3.23 -> 0.3.28)
- devDependencies + peerDependencies version ranges synced to current
package.json (the lockfile was stale relative to head package.json)
Verification:
- `npm test` -> 320 passed / 11 skipped / 0 failed (9 test files)
- `npm run typecheck` -> clean
- osv-scanner before: 14 CVEs; after: 6 (those 6 need a breaking
major-version bump to land — happy to follow up if you want it)
Co-authored-by: Aeon <aeon@aaronjmars.eth>
Remove dead null checks, document CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS
env var, credit contributor.
Fix review findings:
- Use timeout-bounded resolve_proxy_exit_ip in _resolve_webrtc_args
- Add missing timeout handler on tunneled HTTPS request in JS
- Reject nan/inf in Python timeout parsing (parity with JS)
- Recompute deadline after CONNECT succeeds in JS proxy tunnel
- Extract HumanActionOptions type alias to replace ~40 inline copies
- Frame check/uncheck: let isChecked errors propagate instead of silently clicking non-checkbox elements
- SOCKS5 credential log: console.debug → console.info (parity with Python logger.info)
- Add contributors to README
* fix(proxy): log when SOCKS5 credential auto-encoding rewrites URL (#157)
Auto URL-encoding of SOCKS5 credentials (added in v0.3.26 to fix Chromium's
'=' truncation bug) currently happens silently. Users debugging connectivity
have no way to know the wrapper rewrote their proxy URL — the original #157
thread took 8 round-trips to surface this exact ambiguity.
Emit a log when re-encoding actually changes the URL: INFO on Python's
'cloakbrowser' logger, console.debug in JavaScript. Stays silent on
already-encoded inputs and credential-less URLs to avoid false-positive
noise. Credentials are not included in the log message.
Tests: 3 new cases per language (Python caplog, JS vi.spyOn console.debug)
covering trigger / silent-when-encoded / silent-when-no-creds.
* fix(proxy): gate log on credential change, not full URL diff
Per Copilot review on #209: urlparse cosmetically lowercases scheme and
hostname, so comparing the full reconstructed URL to the input would emit
"Auto URL-encoded SOCKS5..." even for inputs like
`socks5://USER:pass@HOST.com:1080` where no credential encoding happened.
Compare raw vs encoded user/password substrings instead. Mirror the same
condition in JS for parity (JS's manual parser preserves case today, but the
credential-level compare is more robust against future changes).
Adds one regression test per language.
The humanize layer hardcoded timeout=2000ms for element lookups, causing
locator.click() and page.click() to fail instantly instead of retrying
for 30s like standard Playwright. Aligned all defaults to 30000ms across
Python sync/async, JS Playwright, and JS Puppeteer paths. Bumped the
outer retry sleep from 200ms to 500ms for DOM mutation settle time.
* feat(lambda): cold-start hardening + handler-side retry orchestration
Two related improvements based on benchmarking the integration at scale
(3454-site sample, multiple iterations).
Cold-start hardening (lambda-entrypoint.sh + lambda_handler.py):
- Clean stale Xvfb lock file before starting the X server. We observed
that under cold-start storms, a previous Xvfb sometimes died and left
/tmp/.X99-lock + /tmp/.X11-unix/X99 behind, so the next start failed
with "Server is already active for display 99". Removing both files
makes Xvfb start cleanly every time.
- Replace `sleep 0.5` with a poll-for-X11-socket loop (up to 10s) plus
a 200ms post-socket buffer for listen()/accept() to settle. The
fixed sleep lost the race during concurrent cold inits, surfacing as
"Looks like you launched a headed browser without having a XServer
running" failures (~10% rate at 100-concurrent cold-start storm).
- Add _launch_with_retry helper in the handler: 3 attempts with linear
backoff (0.3s, 0.6s) on launch_context_async failures. Belt-and-
suspenders for whatever the entrypoint fix doesn't catch — a retry on
a now-warm container almost always succeeds.
Handler-side retry orchestration (lambda_handler.py):
- Add _classify_error() — maps Playwright errors to retry-strategy
overrides:
ERR_CERT_* -> --ignore-certificate-errors + 60s goto
Timeout exceeded -> 90s goto + 25s smart_wait cap
ERR_CONNECTION_TIMED_OUT -> same as Timeout
Returns None for unrecoverable site issues (DNS, SSL, refused, HTTP
4xx/5xx) — those bail immediately without burning a retry slot.
- Add _attempt_scrape() — extracted scrape body so the retry loop can
call it with overridden event dicts. Each attempt relaunches the
browser; uniform behavior across strategies.
- Rewrite _run() as a retry loop: first attempt uses event verbatim;
on a classifiable failure, merge the strategy's overrides into the
event and retry. Bounded by the new `retries` event field (default 1;
set to 0 to disable retry).
- Add _raise_with_history() — surfaces a final failure with a
retry_history block embedded in the error message so callers see
exactly what was tried before bailing. Successful invocations return
the standard response shape unchanged — no surprise fields.
INSTRUCTIONS.md updates:
- Bump function timeout recommendation from 60-120s to 120-180s. Under
retry, a Timeout-class first failure (30s) plus a longer-budget retry
(90s) plus cleanup can total ~120-130s; 180s leaves headroom.
- Document the new `retries` event field in the schema.
- Add a "Retry orchestration" subsection covering both layers (launch
retries and strategy retries) with the full strategy table.
Bench results on the 3454-site sample (seed=1):
v1 baseline (no fixes, c=100): 13.5% failure rate, $1.07
v2 (entrypoint Xvfb poll only, c=100): 9.9% failure rate, $1.11
v3 (cold-start fix + bench-side retry): 3.3% failure rate, $1.32
This change (handler retry, c=250): 2.1% failure rate, $1.13
The remaining 2.1% are all genuinely unrecoverable: DNS doesn't exist,
broken SSL, connection refused, 4xx/5xx responses, payload >6MB Lambda
limit. No retry logic can fix those.
* fix(lambda): merge extra_args on strategy retry instead of clobbering
A flat dict spread replaced caller-supplied extra_args (e.g.
--proxy-server=...) with the strategy's extra_args on a cert retry.
Append both lists so caller flags survive the merge.
Chromium's --proxy-server parser truncates passwords at '=' and other
special chars, causing SOCKS5 auth to silently fail and fall back to
direct connection. The dict path already encoded creds; now the string
path does too. Idempotent: pre-encoded input stays encoded.
CloakHQ