mirror of
https://github.com/CloakHQ/CloakBrowser.git
synced 2026-06-23 11:41:46 +02:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
db0b5f1946 | ||
|
|
babef04e07 | ||
|
|
f8026a7b39 | ||
|
|
71f57d00d1 | ||
|
|
e9735392e8 | ||
|
|
0d41a4f023 | ||
|
|
80d9f7c14e | ||
|
|
114b3c826b | ||
|
|
c07c2b6b4a | ||
|
|
13b1b98b68 | ||
|
|
0d6ce76b1d | ||
|
|
f01902025a | ||
|
|
2df8c7e2d1 | ||
|
|
661b873dad | ||
|
|
ee346a6a57 | ||
|
|
3e699f554c | ||
|
|
9eb90da012 | ||
|
|
6b8d8b6378 | ||
|
|
252e79b17d | ||
|
|
0ccdc71e47 | ||
|
|
74b1ff64db | ||
|
|
a9a0ba13ba | ||
|
|
b04ad6ec2a | ||
|
|
4459f66593 | ||
|
|
ce8b92ba4f | ||
|
|
4e1027847e | ||
|
|
f164c1c874 | ||
|
|
f5e242a160 | ||
|
|
935beef980 | ||
|
|
c6d3469e4c | ||
|
|
cb0b87873e | ||
|
|
2be8cdcc03 | ||
|
|
be9a98db67 | ||
|
|
9b004bbd85 |
@@ -23,7 +23,7 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: 20
|
||||
- name: Install and build
|
||||
|
||||
@@ -32,7 +32,7 @@ jobs:
|
||||
run: |
|
||||
pip install -e ".[dev]" pytest pytest-asyncio
|
||||
pytest tests/ -v -m "not slow"
|
||||
- uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: 22
|
||||
- name: JavaScript tests
|
||||
@@ -71,7 +71,7 @@ jobs:
|
||||
pip install build
|
||||
python -m build
|
||||
- name: Publish to PyPI
|
||||
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1
|
||||
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1
|
||||
|
||||
publish-npm:
|
||||
needs: [test, validate-version]
|
||||
@@ -81,7 +81,7 @@ jobs:
|
||||
id-token: write # OIDC trusted publishing + provenance — no NPM_TOKEN needed
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: 24 # npm 11.11.0 native — no upgrade needed (Node 22.22.2 has broken npm)
|
||||
registry-url: 'https://registry.npmjs.org'
|
||||
@@ -113,7 +113,7 @@ jobs:
|
||||
password: ${{ secrets.DOCKER_PAT }}
|
||||
- name: Build and push
|
||||
id: build
|
||||
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
with:
|
||||
context: .
|
||||
platforms: linux/amd64,linux/arm64
|
||||
@@ -123,7 +123,7 @@ jobs:
|
||||
cloakhq/cloakbrowser:latest
|
||||
provenance: true
|
||||
sbom: true
|
||||
- uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003 # v4.1.1
|
||||
- uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
|
||||
- name: Sign image
|
||||
run: cosign sign --yes cloakhq/cloakbrowser@${{ steps.build.outputs.digest }}
|
||||
- name: Attest build provenance
|
||||
|
||||
@@ -6,6 +6,51 @@ Changes are tagged: **[wrapper]** for Python/JS wrapper, **[binary]** for Chromi
|
||||
|
||||
---
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.3.28] — 2026-05-11
|
||||
|
||||
- **[wrapper]** **Security**: `cloakserve` — sanitize fingerprint seed to prevent path traversal, bind to `127.0.0.1` on bare metal, detect Podman containers (#217)
|
||||
- **[wrapper]** Fix GeoIP resolution hanging indefinitely — bounded with 10s timeout so `launch()` cannot stall (thanks [@manaskarra](https://github.com/manaskarra), #213)
|
||||
- **[wrapper]** JS: preserve iframe scope in humanized frame actions — `check()`, `uncheck()`, `selectOption()` now execute in the correct frame (thanks [@manaskarra](https://github.com/manaskarra), #201)
|
||||
- **[wrapper]** JS: add TypeScript types to humanized method options — `HumanActionOptions` type for `human_config` and `timeout` overrides (thanks [@eofreternal](https://github.com/eofreternal), #205)
|
||||
- **[wrapper]** Log when SOCKS5 credential auto-encoding rewrites a proxy URL (thanks [@Youhai020616](https://github.com/Youhai020616), #209)
|
||||
- **[wrapper]** JS: bump `playwright-core` peer dependency minimum to >=1.53.0 (#200)
|
||||
- **[meta]** Bump sigstore/cosign-installer in CI (#214)
|
||||
|
||||
## [0.3.27] — 2026-05-06
|
||||
|
||||
- **[wrapper]** Per-call `human_config` override — pass `human_config={...}` to individual humanized methods to override global HumanConfig on a per-action basis (#183)
|
||||
- **[wrapper]** Humanized `scrollIntoViewIfNeeded` — auto-scrolls with human-like behavior when `humanize=True` (#183)
|
||||
- **[wrapper]** Forward `timeout` parameter through humanized Playwright methods (#183)
|
||||
- **[wrapper]** Fix humanize timeout default to align with Playwright's 30s auto-retry instead of custom 2s (#172)
|
||||
|
||||
## [0.3.26] — 2026-04-28
|
||||
|
||||
- **[binary]** Windows x64 upgraded to Chromium 146.0.7680.177.4 — 57 source-level fingerprint patches (up from 33 on 145.0.7632.159.7), now matches Linux. Includes all binary improvements from 0.3.18–0.3.25: native SOCKS5 proxy with UDP ASSOCIATE (QUIC/HTTP3), WebRTC IP spoofing, proxy signal removal, CDP input stealth, storage quota normalization, WebAuthn/AAC/window position patches, WebGL and canvas consistency fixes, expanded GPU model database
|
||||
- **[wrapper]** Auto URL-encode SOCKS5 credentials containing special characters in string URLs (#157)
|
||||
- **[wrapper]** AWS Lambda integration example with cold-start hardening and handler-side retry orchestration (#177, thanks [@AlexTech314](https://github.com/AlexTech314))
|
||||
- **[docker]** Add emoji and extended font packages to resolve Kasada/Akamai canvas fingerprint blocks (#179)
|
||||
- **[docs]** Add Font Setup on Linux section to README (#179)
|
||||
- **[docs]** Add Deployment Integrations section to README (#177)
|
||||
- **[meta]** Bump GitHub Actions dependencies (#178)
|
||||
|
||||
## [0.3.25] — 2026-04-16
|
||||
|
||||
- **[wrapper]** Python: add `launch_context_async()` — async counterpart to `launch_context()`. Returns a BrowserContext with all kwargs forwarded to `browser.new_context()`, enabling `storage_state`, `permissions`, `extra_http_headers`, etc. without a persistent profile folder. Closes #141.
|
||||
- **[wrapper]** JS: `launchContext()` and `launchPersistentContext()` silently dropped unknown options (including `storageState`). New `contextOptions` escape hatch forwards arbitrary options to Playwright's `newContext()`.
|
||||
- **[wrapper]** Fix `humanConfig` TypeScript typing (#151).
|
||||
- **[binary]** New build 146.0.7680.177.3 for Linux x64 + arm64 — 57 source-level fingerprint patches (up from 49): WebAuthn capabilities, AAC audio encoder, and window position spoofing; WebGL and canvas format consistency fixes; SOCKS5 warm connection pool auth fix for credentialed proxies.
|
||||
- **[docs]** Add recommended anti-bot config and SOCKS5 tips to troubleshooting.
|
||||
|
||||
## [0.3.24] — 2026-04-10
|
||||
|
||||
- **[wrapper]** Native SOCKS5 proxy support — pass `proxy="socks5://user:pass@host:port"` directly. Credentials handled natively by Chrome. Works across all launch functions, Python + JS.
|
||||
- **[wrapper]** Add Playwright ElementHandle humanize support — `element_handle.click()`, `.fill()`, `.type()` now use human-like behavior when `humanize=True` (thanks [@evelaa123](https://github.com/evelaa123), #133)
|
||||
- **[binary]** Upgrade Linux arm64 to Chromium 146.0.7680.177.2 (49 patches) — now matches Linux x64
|
||||
- **[binary]** New build 146.0.7680.177.2 for both Linux platforms: native SOCKS5 proxy with UDP ASSOCIATE (QUIC/HTTP3 over SOCKS5)
|
||||
- **[docs]** Clarify humanize requires wrapper import over CDP (#126)
|
||||
|
||||
## [0.3.23] — 2026-04-09
|
||||
|
||||
- **[wrapper]** Add full Puppeteer humanize support — human-like mouse, keyboard, and scroll behavior for `puppeteer-core` users (thanks [@evelaa123](https://github.com/evelaa123), #129)
|
||||
|
||||
@@ -9,6 +9,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
libxcb1 libxext6 libxshmfence1 \
|
||||
libglib2.0-0 libgtk-3-0 libpangocairo-1.0-0 libcairo-gobject2 \
|
||||
libgdk-pixbuf-2.0-0 libxss1 libxtst6 fonts-liberation \
|
||||
fonts-noto-color-emoji fonts-unifont fonts-freefont-ttf \
|
||||
fonts-ipafont-gothic fonts-wqy-zenhei fonts-tlwg-loma-otf \
|
||||
xvfb xdotool \
|
||||
curl ca-certificates \
|
||||
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
|
||||
|
||||
@@ -128,10 +128,13 @@ Open [http://localhost:8080](http://localhost:8080). Create a profile. Click **L
|
||||
|
||||
---
|
||||
|
||||
## Latest: v0.3.22 (Chromium 146.0.7680.177.1)
|
||||
## Latest: v0.3.26 (Chromium 146.0.7680.177.4)
|
||||
|
||||
- **`launch_context_async()`** — async counterpart to `launch_context()`. Forwards kwargs to `browser.new_context()` for `storage_state`, `permissions`, `extra_http_headers` without a persistent profile folder.
|
||||
- **JS `contextOptions` escape hatch** — forward arbitrary options (including `storageState`) to Playwright's `newContext()` from `launchContext()` / `launchPersistentContext()`.
|
||||
- **Native SOCKS5 proxy** — `proxy="socks5://user:pass@host:port"` works directly in all launch functions, Python + JS. QUIC/HTTP3 tunnels through SOCKS5 via UDP ASSOCIATE.
|
||||
- **Chromium 146 upgrade** — rebased all patches from 145.0.7632.x to 146.0.7680.177
|
||||
- **49 fingerprint patches** (Linux x64) — 1 new patch, all existing patches carried forward
|
||||
- **57 fingerprint patches** — additional detection-vector coverage (WebAuthn, AAC audio, window position) and WebGL/canvas consistency fixes
|
||||
- **WebRTC IP spoofing** — `--fingerprint-webrtc-ip=auto` resolves your proxy's exit IP and spoofs WebRTC ICE candidates. Auto-injected when using `geoip=True` (no extra network call)
|
||||
- **Proxy signal removal** — DNS/connect/SSL timing zeroed, proxy cache headers stripped, Proxy-Connection header leak removed
|
||||
- **`cloakserve` CDP multiplexer** — rewritten as a multi-connection CDP proxy with per-connection fingerprint seeds
|
||||
@@ -242,8 +245,9 @@ browser = launch()
|
||||
# Headed mode (see the browser window)
|
||||
browser = launch(headless=False)
|
||||
|
||||
# With proxy
|
||||
# With proxy (HTTP or SOCKS5)
|
||||
browser = launch(proxy="http://user:pass@proxy:8080")
|
||||
browser = launch(proxy="socks5://user:pass@proxy:1080")
|
||||
|
||||
# With proxy dict (bypass, separate auth fields)
|
||||
browser = launch(proxy={"server": "http://proxy:8080", "bypass": ".google.com", "username": "user", "password": "pass"})
|
||||
@@ -314,6 +318,38 @@ page.goto("https://protected-site.com")
|
||||
context.close()
|
||||
```
|
||||
|
||||
Extra kwargs are forwarded to Playwright's `browser.new_context()` — use this for `storage_state`, `permissions`, `extra_http_headers`, etc. without needing a persistent profile folder:
|
||||
|
||||
```python
|
||||
from cloakbrowser import launch_context
|
||||
|
||||
# Restore a saved session (cookies, localStorage) from a JSON file
|
||||
context = launch_context(storage_state="state.json")
|
||||
page = context.new_page()
|
||||
page.goto("https://example.com")
|
||||
# Save state back for next run
|
||||
context.storage_state(path="state.json")
|
||||
context.close()
|
||||
```
|
||||
|
||||
### `launch_context_async()`
|
||||
|
||||
Async counterpart to `launch_context()`. Same signature and kwargs forwarding:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from cloakbrowser import launch_context_async
|
||||
|
||||
async def main():
|
||||
ctx = await launch_context_async(storage_state="state.json")
|
||||
page = await ctx.new_page()
|
||||
await page.goto("https://example.com")
|
||||
await ctx.storage_state(path="state.json")
|
||||
await ctx.close()
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### `launch_persistent_context()`
|
||||
|
||||
Same as `launch_context()`, but with a persistent user profile. Cookies, localStorage, and cache persist across sessions.
|
||||
@@ -370,7 +406,7 @@ from cloakbrowser import binary_info, clear_cache, ensure_binary
|
||||
|
||||
# Check binary installation status
|
||||
print(binary_info())
|
||||
# {'version': '146.0.7680.177.1', 'platform': 'linux-x64', 'installed': True, ...}
|
||||
# {'version': '146.0.7680.177.3', 'platform': 'linux-x64', 'installed': True, ...}
|
||||
|
||||
# Force re-download
|
||||
clear_cache()
|
||||
@@ -534,6 +570,7 @@ Access the original un-patched Playwright page at `page._original` if you need r
|
||||
| `CLOAKBROWSER_DOWNLOAD_URL` | `cloakbrowser.dev` | Custom download URL for binary |
|
||||
| `CLOAKBROWSER_AUTO_UPDATE` | `true` | Set to `false` to disable background update checks |
|
||||
| `CLOAKBROWSER_SKIP_CHECKSUM` | `false` | Set to `true` to skip SHA-256 verification after download |
|
||||
| `CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS` | `5` | Max seconds for GeoIP resolution before continuing without it |
|
||||
|
||||
## Fingerprint Management
|
||||
|
||||
@@ -590,13 +627,39 @@ Supported by the binary but **not set by default** — pass via `args` to custom
|
||||
| `--fingerprint-locale` | Locale (e.g. `en-US`) |
|
||||
| `--fingerprint-storage-quota` | Override storage quota in MB — affects `storage.estimate()`, `storageBuckets`, and legacy webkit APIs. Auto-normalized when `--fingerprint` is set |
|
||||
| `--fingerprint-taskbar-height` | Override taskbar height (binary defaults: Win=48, Mac=95, Linux=0) |
|
||||
| `--fingerprint-fonts-dir` | Path to cross-platform font directory |
|
||||
| `--fingerprint-fonts-dir` | Path to directory containing target-platform fonts (see [Font Setup on Linux](#font-setup-on-linux)) |
|
||||
| `--fingerprint-webrtc-ip` | WebRTC ICE candidate IP replacement. Use `auto` to resolve from proxy exit IP (makes an HTTP call through the proxy), or pass an explicit IP. Auto-injected when `geoip=True` |
|
||||
| `--fingerprint-noise=false` | Disable noise injection (canvas, WebGL, audio, client rects) while keeping the deterministic fingerprint seed active |
|
||||
| `--enable-blink-features=FakeShadowRoot` | Access closed shadow DOM elements |
|
||||
|
||||
> **Note:** All stealth tests were verified with the default fingerprint config above. Changing these flags may affect detection results — test your configuration before using in production.
|
||||
|
||||
### Font Setup on Linux
|
||||
|
||||
**Required for aggressive anti-bot sites (Kasada, Akamai).** These systems render emoji on a hidden canvas and hash the pixel output. Minimal Linux environments (Docker, cloud VMs) often lack emoji and extended fonts, producing hashes that don't match any real browser. Install standard font packages to fix this:
|
||||
|
||||
```bash
|
||||
sudo apt install -y fonts-noto-color-emoji fonts-freefont-ttf fonts-unifont \
|
||||
fonts-ipafont-gothic fonts-wqy-zenhei fonts-tlwg-loma-otf
|
||||
```
|
||||
|
||||
The Docker image (`cloakhq/cloakbrowser`) ships with these pre-installed. If you run the binary directly on a Linux server or in a custom Docker image, install them manually.
|
||||
|
||||
**Optional: Windows fonts for CreepJS font enumeration.** The packages above fix anti-bot canvas checks but won't improve your CreepJS font score. For that, you need actual Windows fonts (Segoe UI, Calibri, Bahnschrift, etc.) from a Windows machine's `C:\Windows\Fonts\` directory — `ttf-mscorefonts-installer` only has old XP-era fonts and isn't enough.
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.local/share/fonts/windows
|
||||
cp /path/to/windows/fonts/*.ttf ~/.local/share/fonts/windows/
|
||||
cp /path/to/windows/fonts/*.TTF ~/.local/share/fonts/windows/
|
||||
fc-cache -f # mandatory for manually copied fonts
|
||||
```
|
||||
|
||||
```python
|
||||
browser = launch(
|
||||
args=["--fingerprint-fonts-dir=/home/user/.local/share/fonts/windows"],
|
||||
)
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
```python
|
||||
@@ -645,8 +708,16 @@ stealth_args = get_default_stealth_args() # all fingerprint flags
|
||||
from cloakbrowser import launch_async
|
||||
browser = await launch_async(args=["--remote-debugging-port=9242"])
|
||||
# Connect your framework to http://127.0.0.1:9242 — all stealth flags are set
|
||||
# Note: humanize requires the wrapper (see below)
|
||||
```
|
||||
|
||||
> **Humanize over CDP**: Stealth fingerprint patches work automatically over CDP, but `humanize=True` is a wrapper-level feature. If you connect to CloakBrowser via CDP from a separate script, import the patching functions to add humanization:
|
||||
>
|
||||
> ```js
|
||||
> import { patchBrowser, resolveConfig } from 'cloakbrowser/human';
|
||||
> patchBrowser(browser, resolveConfig('default'));
|
||||
> ```
|
||||
|
||||
| Framework | Stars | Language | Example |
|
||||
|-----------|-------|----------|---------|
|
||||
| [browser-use](https://github.com/browser-use/browser-use) | 70K | Python | [`browser_use_example.py`](examples/integrations/browser_use_example.py) |
|
||||
@@ -659,15 +730,21 @@ browser = await launch_async(args=["--remote-debugging-port=9242"])
|
||||
| [undetected-chromedriver](https://github.com/ultrafunkamsterdam/undetected-chromedriver) | 12K | Python | [`undetected_chromedriver.py`](examples/integrations/undetected_chromedriver.py) |
|
||||
| [agent-browser](https://github.com/nichochar/agent-browser) | — | Shell | [`agent_browser.sh`](examples/integrations/agent_browser.sh) |
|
||||
|
||||
### Deployment Integrations
|
||||
|
||||
| Platform | Example |
|
||||
|----------|---------|
|
||||
| [AWS Lambda](https://aws.amazon.com/lambda/) | [`aws_lambda/`](examples/integrations/aws_lambda/) — One-shot scrapes in Lambda (container image) |
|
||||
|
||||
## Platforms
|
||||
|
||||
| Platform | Chromium | Patches | Status |
|
||||
|---|---|---|---|
|
||||
| Linux x86_64 | 146 | 49 | ✅ Latest |
|
||||
| Linux arm64 (RPi, Graviton) | 145 | 48 | ✅ |
|
||||
| Linux x86_64 | 146 | 57 | ✅ Latest |
|
||||
| Linux arm64 (RPi, Graviton) | 146 | 57 | ✅ Latest |
|
||||
| macOS arm64 (Apple Silicon) | 145 | 26 | ✅ |
|
||||
| macOS x86_64 (Intel) | 145 | 26 | ✅ |
|
||||
| Windows x86_64 | 145 | 48 | ✅ |
|
||||
| Windows x86_64 | 146 | 57 | ✅ Latest |
|
||||
|
||||
The wrapper auto-downloads the correct binary for your platform.
|
||||
|
||||
@@ -864,7 +941,47 @@ page.goto("https://heavily-protected-site.com") # passes DataDome, etc.
|
||||
browser.close()
|
||||
```
|
||||
|
||||
This runs a real headed browser rendered on a virtual display — no physical monitor needed. Combined with a residential proxy, this passes even the most aggressive detection services. Datacenter IPs are often flagged by IP reputation regardless of browser fingerprint — a residential proxy makes the difference.
|
||||
This runs a real headed browser rendered on a virtual display — no physical monitor needed. Combine with the recommended config below for maximum stealth.
|
||||
|
||||
---
|
||||
|
||||
### Recommended config for anti-bot sites
|
||||
|
||||
Most blocks come from missing one of these three things, not from browser fingerprint detection:
|
||||
|
||||
```python
|
||||
browser = launch(
|
||||
proxy="http://your-residential-proxy:port", # residential IP — datacenter IPs get blocked by reputation alone
|
||||
geoip=True, # matches timezone + locale to proxy exit IP (without this: UTC + en-US = bot signal)
|
||||
headless=False, # headed mode — some sites detect headless even with C++ patches
|
||||
humanize=True, # human-like mouse, keyboard, scroll behavior
|
||||
)
|
||||
```
|
||||
|
||||
```javascript
|
||||
const browser = await launch({
|
||||
proxy: 'http://your-residential-proxy:port',
|
||||
geoip: true,
|
||||
headless: false,
|
||||
humanize: true,
|
||||
});
|
||||
```
|
||||
|
||||
If your proxy supports SOCKS5, use it for better compatibility — SOCKS5 tunnels raw TCP, avoiding HTTP CONNECT issues that some proxies have with HTTP/2:
|
||||
|
||||
```python
|
||||
browser = launch(proxy="socks5://user:pass@proxy:1080", geoip=True, headless=False, humanize=True)
|
||||
```
|
||||
|
||||
If you're still blocked after this, check the font setup below.
|
||||
|
||||
---
|
||||
|
||||
### Blocked on Kasada / Akamai sites despite correct config?
|
||||
|
||||
On minimal Linux environments, missing font packages cause canvas emoji rendering to produce hashes that anti-bot systems don't recognize. This is the most common cause of blocks on aggressive sites after proxy, geoip, and headed mode are already set up correctly.
|
||||
|
||||
Install the font packages listed in [Font Setup on Linux](#font-setup-on-linux) above.
|
||||
|
||||
---
|
||||
|
||||
@@ -900,7 +1017,7 @@ await ctx.close();
|
||||
ctx = await launchPersistentContext({ userDataDir: './profile' });
|
||||
```
|
||||
|
||||
For stateless/ephemeral use cases, `launch(args=["--disable-http2"])` forces HTTP/1.1 which bypasses the check. Only use this flag for sites that require it — most work fine with HTTP/2.
|
||||
For stateless/ephemeral use cases, `launch(args=["--disable-http2"])` forces HTTP/1.1 which bypasses the check. Only use this flag for sites that require it — most work fine with HTTP/2. If your proxy supports SOCKS5, use `proxy="socks5://user:pass@host:port"` instead — SOCKS5 bypasses HTTP CONNECT entirely.
|
||||
|
||||
---
|
||||
|
||||
@@ -1020,15 +1137,15 @@ A: Camoufox patches Firefox. We patch Chromium. Chromium means native Playwright
|
||||
A: Possibly. Bot detection is an arms race. Source-level patches are harder to detect than config-level patches, but not impossible. We actively monitor and update when detection evolves.
|
||||
|
||||
**Q: Can I use my own proxy?**
|
||||
A: Yes. Pass `proxy="http://user:pass@host:port"` to `launch()`.
|
||||
A: Yes. Pass `proxy="http://user:pass@host:port"` or `proxy="socks5://user:pass@host:port"` to `launch()`. Both HTTP and SOCKS5 proxies are supported natively.
|
||||
|
||||
## Roadmap
|
||||
|
||||
| Feature | Status |
|
||||
|---------|--------|
|
||||
| Linux x64 — Chromium 146 (49 patches) | ✅ Released |
|
||||
| Linux x64 — Chromium 146 (57 patches) | ✅ Released |
|
||||
| macOS arm64/x64 — Chromium 145 (26 patches) | ✅ Released |
|
||||
| Windows x64 — Chromium 145 (33 patches) | ✅ Released |
|
||||
| Windows x64 — Chromium 146 (57 patches) | ✅ Released |
|
||||
| JavaScript/Puppeteer + Playwright support | ✅ Released |
|
||||
| Fingerprint rotation per session | ✅ Released |
|
||||
| Built-in proxy rotation | 📋 Planned |
|
||||
@@ -1050,7 +1167,7 @@ All releases are signed for supply chain verification.
|
||||
```bash
|
||||
# Verify GPG signature (binary release tag)
|
||||
gpg --keyserver keyserver.ubuntu.com --recv-keys C60C0DDC9D0DE2DD
|
||||
git verify-tag chromium-v146.0.7680.177.1
|
||||
git verify-tag chromium-v146.0.7680.177.3
|
||||
|
||||
# Verify GitHub binary attestation (Sigstore)
|
||||
gh attestation verify cloakbrowser-linux-x64.tar.gz --repo CloakHQ/cloakbrowser
|
||||
@@ -1076,3 +1193,7 @@ Issues and PRs welcome. If something isn't working, [open an issue](https://gith
|
||||
- [@evelaa123](https://github.com/evelaa123) — humanize behavior, persistent contexts, Windows fix
|
||||
- [@yahooguntu](https://github.com/yahooguntu) — persistent contexts
|
||||
- [@kitiho](https://github.com/kitiho) — null viewport fix
|
||||
- [@eofreternal](https://github.com/eofreternal) — humanConfig type fix, humanized method option types
|
||||
- [@manaskarra](https://github.com/manaskarra) — iframe scope fix for humanized frame actions, GeoIP timeout guard
|
||||
- [@Youhai020616](https://github.com/Youhai020616) — SOCKS5 credential encoding logging
|
||||
- [@AlexTech314](https://github.com/AlexTech314) — AWS Lambda integration
|
||||
|
||||
+33
-8
@@ -22,6 +22,7 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
@@ -36,7 +37,7 @@ import aiohttp
|
||||
import websockets
|
||||
from aiohttp import web
|
||||
|
||||
from cloakbrowser.browser import build_args, maybe_resolve_geoip, _resolve_webrtc_args
|
||||
from cloakbrowser.browser import build_args, maybe_resolve_geoip, _resolve_webrtc_args, _normalize_socks_string_url
|
||||
from cloakbrowser.download import ensure_binary
|
||||
|
||||
logging.basicConfig(
|
||||
@@ -61,6 +62,9 @@ BASE_CHROME_ARGS = [
|
||||
|
||||
BASE_CDP_PORT = 5100
|
||||
|
||||
SAFE_SEED_RE = re.compile(r"^[A-Za-z0-9_-]{1,128}$")
|
||||
RESERVED_SEEDS = {"__default__"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ChromeProcess — one running Chrome instance
|
||||
@@ -111,6 +115,14 @@ class ChromePool:
|
||||
self._locks[seed] = asyncio.Lock()
|
||||
return self._locks[seed]
|
||||
|
||||
def _safe_rmtree(self, path: str) -> None:
|
||||
resolved = Path(path).resolve()
|
||||
data_resolved = Path(self._data_dir).resolve()
|
||||
if resolved == data_resolved or not resolved.is_relative_to(data_resolved):
|
||||
logger.error("Refusing to delete path outside data_dir: %s", resolved)
|
||||
return
|
||||
shutil.rmtree(path, True)
|
||||
|
||||
def _allocate_port(self) -> int:
|
||||
"""Find a free port starting from _next_port."""
|
||||
for _ in range(100):
|
||||
@@ -159,6 +171,11 @@ class ChromePool:
|
||||
seed_key = "__default__"
|
||||
actual_seed = str(random.randint(10000, 99999))
|
||||
else:
|
||||
if not SAFE_SEED_RE.match(seed) or seed in RESERVED_SEEDS:
|
||||
raise web.HTTPBadRequest(
|
||||
text=json.dumps({"error": "Invalid fingerprint seed"}),
|
||||
content_type="application/json",
|
||||
)
|
||||
seed_key = seed
|
||||
actual_seed = seed
|
||||
|
||||
@@ -189,7 +206,7 @@ class ChromePool:
|
||||
if extra_args:
|
||||
fp_extra.extend(extra_args)
|
||||
if proxy:
|
||||
fp_extra.append(f"--proxy-server={proxy}")
|
||||
fp_extra.append(f"--proxy-server={_normalize_socks_string_url(proxy)}")
|
||||
|
||||
# WebRTC IP spoofing: resolve auto, inject geoip exit IP
|
||||
fp_extra = _resolve_webrtc_args(fp_extra, proxy)
|
||||
@@ -232,7 +249,7 @@ class ChromePool:
|
||||
if not await self._wait_for_cdp(port):
|
||||
process.kill()
|
||||
await asyncio.to_thread(process.wait, timeout=5)
|
||||
await asyncio.to_thread(shutil.rmtree, user_data_dir, True)
|
||||
await asyncio.to_thread(self._safe_rmtree, user_data_dir)
|
||||
raise web.HTTPBadGateway(
|
||||
text=json.dumps({"error": "Chrome failed to start"}),
|
||||
content_type="application/json",
|
||||
@@ -266,8 +283,7 @@ class ChromePool:
|
||||
await asyncio.to_thread(proc.process.wait, timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.process.kill()
|
||||
# Clean up user data dir (can be slow for large profiles)
|
||||
await asyncio.to_thread(shutil.rmtree, proc.user_data_dir, True)
|
||||
await asyncio.to_thread(self._safe_rmtree, proc.user_data_dir)
|
||||
if self._default is proc:
|
||||
self._default = None
|
||||
self._locks.pop(key, None)
|
||||
@@ -559,8 +575,8 @@ async def on_shutdown(app: web.Application) -> None:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _default_data_dir() -> str:
|
||||
"""Smart default: Docker → /tmp/cloakserve, bare metal → ~/.cloakbrowser/cloakserve."""
|
||||
if os.path.exists("/.dockerenv"):
|
||||
"""Smart default: container → /tmp/cloakserve, bare metal → ~/.cloakbrowser/cloakserve."""
|
||||
if os.path.exists("/.dockerenv") or os.path.exists("/run/.containerenv"):
|
||||
return "/tmp/cloakserve"
|
||||
return str(Path.home() / ".cloakbrowser" / "cloakserve")
|
||||
|
||||
@@ -624,6 +640,13 @@ def main() -> None:
|
||||
binary = ensure_binary()
|
||||
config, global_args = parse_cli_args(sys.argv[1:])
|
||||
|
||||
if config["default_seed"] and (
|
||||
not SAFE_SEED_RE.match(config["default_seed"])
|
||||
or config["default_seed"] in RESERVED_SEEDS
|
||||
):
|
||||
logger.error("Invalid --fingerprint seed: %s", config["default_seed"])
|
||||
sys.exit(1)
|
||||
|
||||
pool = ChromePool(
|
||||
binary=binary,
|
||||
global_args=global_args,
|
||||
@@ -662,7 +685,9 @@ def main() -> None:
|
||||
port,
|
||||
)
|
||||
|
||||
web.run_app(app, host="0.0.0.0", port=port, print=None)
|
||||
in_container = os.path.exists("/.dockerenv") or os.path.exists("/run/.containerenv")
|
||||
host = "0.0.0.0" if in_container else "127.0.0.1"
|
||||
web.run_app(app, host=host, port=port, print=None)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -11,7 +11,7 @@ Usage:
|
||||
browser.close()
|
||||
"""
|
||||
|
||||
from .browser import launch, launch_async, launch_context, launch_persistent_context, launch_persistent_context_async, ProxySettings, build_args, maybe_resolve_geoip
|
||||
from .browser import launch, launch_async, launch_context, launch_context_async, launch_persistent_context, launch_persistent_context_async, ProxySettings, build_args, maybe_resolve_geoip
|
||||
from .config import CHROMIUM_VERSION, get_default_stealth_args
|
||||
from .download import binary_info, check_for_update, clear_cache, ensure_binary
|
||||
from ._version import __version__
|
||||
@@ -32,6 +32,7 @@ __all__ = [
|
||||
"launch",
|
||||
"launch_async",
|
||||
"launch_context",
|
||||
"launch_context_async",
|
||||
"launch_persistent_context",
|
||||
"launch_persistent_context_async",
|
||||
"ensure_binary",
|
||||
|
||||
@@ -1 +1 @@
|
||||
__version__ = "0.3.23"
|
||||
__version__ = "0.3.28"
|
||||
|
||||
+314
-43
@@ -17,13 +17,15 @@ from __future__ import annotations
|
||||
import logging
|
||||
import os
|
||||
from typing import Any, Literal, TypedDict
|
||||
from urllib.parse import unquote, urlparse, urlunparse
|
||||
from urllib.parse import quote, unquote, urlparse, urlunparse
|
||||
|
||||
from .config import DEFAULT_VIEWPORT, IGNORE_DEFAULT_ARGS, get_default_stealth_args
|
||||
from .download import ensure_binary
|
||||
from .human.config import HumanConfigOverrides, HumanPreset
|
||||
|
||||
logger = logging.getLogger("cloakbrowser")
|
||||
|
||||
|
||||
# Sentinel to distinguish "viewport not provided" from "viewport=None" (disable emulation)
|
||||
_VIEWPORT_UNSET = object()
|
||||
|
||||
@@ -60,8 +62,8 @@ def launch(
|
||||
geoip: bool = False,
|
||||
backend: str | None = None,
|
||||
humanize: bool = False,
|
||||
human_preset: str = "default",
|
||||
human_config: dict | None = None,
|
||||
human_preset: HumanPreset = "default",
|
||||
human_config: HumanConfigOverrides | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""Launch stealth Chromium browser. Returns a Playwright Browser object.
|
||||
@@ -87,7 +89,7 @@ def launch(
|
||||
Override globally with CLOAKBROWSER_BACKEND env var.
|
||||
humanize: Enable human-like mouse, keyboard, scroll behavior (default False).
|
||||
human_preset: Humanize preset — 'default' or 'careful' (default 'default').
|
||||
human_config: Custom humanize config dict to override preset values.
|
||||
human_config: Custom humanize config mapping to override preset values.
|
||||
**kwargs: Passed directly to playwright.chromium.launch().
|
||||
|
||||
Returns:
|
||||
@@ -105,11 +107,12 @@ def launch(
|
||||
|
||||
binary_path = ensure_binary()
|
||||
timezone, locale, exit_ip = maybe_resolve_geoip(geoip, proxy, timezone, locale)
|
||||
proxy_kwargs, proxy_extra_args = _resolve_proxy_config(proxy)
|
||||
args = _resolve_webrtc_args(args, proxy)
|
||||
if exit_ip and not (args and any(a.startswith("--fingerprint-webrtc-ip") for a in args)):
|
||||
args = list(args or [])
|
||||
args.append(f"--fingerprint-webrtc-ip={exit_ip}")
|
||||
chrome_args = build_args(stealth_args, args, timezone=timezone, locale=locale, headless=headless)
|
||||
chrome_args = build_args(stealth_args, (args or []) + proxy_extra_args, timezone=timezone, locale=locale, headless=headless)
|
||||
|
||||
logger.debug("Launching stealth Chromium (headless=%s, args=%d)", headless, len(chrome_args))
|
||||
|
||||
@@ -119,7 +122,7 @@ def launch(
|
||||
headless=headless,
|
||||
args=chrome_args,
|
||||
ignore_default_args=IGNORE_DEFAULT_ARGS,
|
||||
**_build_proxy_kwargs(proxy),
|
||||
**proxy_kwargs,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@@ -154,8 +157,8 @@ async def launch_async( # noqa: C901
|
||||
geoip: bool = False,
|
||||
backend: str | None = None,
|
||||
humanize: bool = False,
|
||||
human_preset: str = "default",
|
||||
human_config: dict | None = None,
|
||||
human_preset: HumanPreset = "default",
|
||||
human_config: HumanConfigOverrides | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""Async version of launch(). Returns a Playwright Browser object.
|
||||
@@ -171,7 +174,7 @@ async def launch_async( # noqa: C901
|
||||
backend: Playwright backend — 'playwright' (default) or 'patchright'.
|
||||
humanize: Enable human-like mouse, keyboard, scroll behavior (default False).
|
||||
human_preset: Humanize preset — 'default' or 'careful' (default 'default').
|
||||
human_config: Custom humanize config dict to override preset values.
|
||||
human_config: Custom humanize config mapping to override preset values.
|
||||
**kwargs: Passed directly to playwright.chromium.launch().
|
||||
|
||||
Returns:
|
||||
@@ -194,11 +197,12 @@ async def launch_async( # noqa: C901
|
||||
|
||||
binary_path = ensure_binary()
|
||||
timezone, locale, exit_ip = maybe_resolve_geoip(geoip, proxy, timezone, locale)
|
||||
proxy_kwargs, proxy_extra_args = _resolve_proxy_config(proxy)
|
||||
args = _resolve_webrtc_args(args, proxy)
|
||||
if exit_ip and not (args and any(a.startswith("--fingerprint-webrtc-ip") for a in args)):
|
||||
args = list(args or [])
|
||||
args.append(f"--fingerprint-webrtc-ip={exit_ip}")
|
||||
chrome_args = build_args(stealth_args, args, timezone=timezone, locale=locale, headless=headless)
|
||||
chrome_args = build_args(stealth_args, (args or []) + proxy_extra_args, timezone=timezone, locale=locale, headless=headless)
|
||||
|
||||
logger.debug("Launching stealth Chromium async (headless=%s, args=%d)", headless, len(chrome_args))
|
||||
|
||||
@@ -208,7 +212,7 @@ async def launch_async( # noqa: C901
|
||||
headless=headless,
|
||||
args=chrome_args,
|
||||
ignore_default_args=IGNORE_DEFAULT_ARGS,
|
||||
**_build_proxy_kwargs(proxy),
|
||||
**proxy_kwargs,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@@ -247,8 +251,8 @@ def launch_persistent_context(
|
||||
geoip: bool = False,
|
||||
backend: str | None = None,
|
||||
humanize: bool = False,
|
||||
human_preset: str = "default",
|
||||
human_config: dict | None = None,
|
||||
human_preset: HumanPreset = "default",
|
||||
human_config: HumanConfigOverrides | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""Launch stealth browser with a persistent profile and return a BrowserContext.
|
||||
@@ -277,7 +281,7 @@ def launch_persistent_context(
|
||||
backend: Playwright backend — 'playwright' (default) or 'patchright'.
|
||||
humanize: Enable human-like mouse, keyboard, scroll behavior (default False).
|
||||
human_preset: Humanize preset — 'default' or 'careful' (default 'default').
|
||||
human_config: Custom humanize config dict to override preset values.
|
||||
human_config: Custom humanize config mapping to override preset values.
|
||||
**kwargs: Passed directly to playwright.chromium.launch_persistent_context().
|
||||
|
||||
Returns:
|
||||
@@ -297,11 +301,12 @@ def launch_persistent_context(
|
||||
|
||||
binary_path = ensure_binary()
|
||||
timezone, locale, exit_ip = maybe_resolve_geoip(geoip, proxy, timezone, locale)
|
||||
proxy_kwargs, proxy_extra_args = _resolve_proxy_config(proxy)
|
||||
args = _resolve_webrtc_args(args, proxy)
|
||||
if exit_ip and not (args and any(a.startswith("--fingerprint-webrtc-ip") for a in args)):
|
||||
args = list(args or [])
|
||||
args.append(f"--fingerprint-webrtc-ip={exit_ip}")
|
||||
chrome_args = build_args(stealth_args, args, timezone=timezone, locale=locale, headless=headless)
|
||||
chrome_args = build_args(stealth_args, (args or []) + proxy_extra_args, timezone=timezone, locale=locale, headless=headless)
|
||||
|
||||
logger.debug(
|
||||
"Launching persistent stealth Chromium (headless=%s, user_data_dir=%s)",
|
||||
@@ -331,7 +336,7 @@ def launch_persistent_context(
|
||||
headless=headless,
|
||||
args=chrome_args,
|
||||
ignore_default_args=IGNORE_DEFAULT_ARGS,
|
||||
**_build_proxy_kwargs(proxy),
|
||||
**proxy_kwargs,
|
||||
**context_kwargs,
|
||||
)
|
||||
|
||||
@@ -370,8 +375,8 @@ async def launch_persistent_context_async(
|
||||
geoip: bool = False,
|
||||
backend: str | None = None,
|
||||
humanize: bool = False,
|
||||
human_preset: str = "default",
|
||||
human_config: dict | None = None,
|
||||
human_preset: HumanPreset = "default",
|
||||
human_config: HumanConfigOverrides | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""Async version of launch_persistent_context().
|
||||
@@ -397,7 +402,7 @@ async def launch_persistent_context_async(
|
||||
backend: Playwright backend — 'playwright' (default) or 'patchright'.
|
||||
humanize: Enable human-like mouse, keyboard, scroll behavior (default False).
|
||||
human_preset: Humanize preset — 'default' or 'careful' (default 'default').
|
||||
human_config: Custom humanize config dict to override preset values.
|
||||
human_config: Custom humanize config mapping to override preset values.
|
||||
**kwargs: Passed directly to playwright.chromium.launch_persistent_context().
|
||||
|
||||
Returns:
|
||||
@@ -422,11 +427,12 @@ async def launch_persistent_context_async(
|
||||
|
||||
binary_path = ensure_binary()
|
||||
timezone, locale, exit_ip = maybe_resolve_geoip(geoip, proxy, timezone, locale)
|
||||
proxy_kwargs, proxy_extra_args = _resolve_proxy_config(proxy)
|
||||
args = _resolve_webrtc_args(args, proxy)
|
||||
if exit_ip and not (args and any(a.startswith("--fingerprint-webrtc-ip") for a in args)):
|
||||
args = list(args or [])
|
||||
args.append(f"--fingerprint-webrtc-ip={exit_ip}")
|
||||
chrome_args = build_args(stealth_args, args, timezone=timezone, locale=locale, headless=headless)
|
||||
chrome_args = build_args(stealth_args, (args or []) + proxy_extra_args, timezone=timezone, locale=locale, headless=headless)
|
||||
|
||||
logger.debug(
|
||||
"Launching persistent stealth Chromium async (headless=%s, user_data_dir=%s)",
|
||||
@@ -456,7 +462,7 @@ async def launch_persistent_context_async(
|
||||
headless=headless,
|
||||
args=chrome_args,
|
||||
ignore_default_args=IGNORE_DEFAULT_ARGS,
|
||||
**_build_proxy_kwargs(proxy),
|
||||
**proxy_kwargs,
|
||||
**context_kwargs,
|
||||
)
|
||||
|
||||
@@ -494,8 +500,8 @@ def launch_context(
|
||||
geoip: bool = False,
|
||||
backend: str | None = None,
|
||||
humanize: bool = False,
|
||||
human_preset: str = "default",
|
||||
human_config: dict | None = None,
|
||||
human_preset: HumanPreset = "default",
|
||||
human_config: HumanConfigOverrides | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""Launch stealth browser and return a BrowserContext with common options pre-set.
|
||||
@@ -519,7 +525,7 @@ def launch_context(
|
||||
backend: Playwright backend — 'playwright' (default) or 'patchright'.
|
||||
humanize: Enable human-like mouse, keyboard, scroll behavior (default False).
|
||||
human_preset: Humanize preset — 'default' or 'careful' (default 'default').
|
||||
human_config: Custom humanize config dict to override preset values.
|
||||
human_config: Custom humanize config mapping to override preset values.
|
||||
**kwargs: Passed to browser.new_context().
|
||||
|
||||
Returns:
|
||||
@@ -580,6 +586,130 @@ def launch_context(
|
||||
return context
|
||||
|
||||
|
||||
async def launch_context_async(
|
||||
headless: bool = True,
|
||||
proxy: str | ProxySettings | None = None,
|
||||
args: list[str] | None = None,
|
||||
stealth_args: bool = True,
|
||||
user_agent: str | None = None,
|
||||
viewport: dict | None = _VIEWPORT_UNSET,
|
||||
locale: str | None = None,
|
||||
timezone: str | None = None,
|
||||
color_scheme: Literal["light", "dark", "no-preference"] | None = None,
|
||||
geoip: bool = False,
|
||||
backend: str | None = None,
|
||||
humanize: bool = False,
|
||||
human_preset: HumanPreset = "default",
|
||||
human_config: HumanConfigOverrides | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
"""Async version of launch_context().
|
||||
|
||||
Launch stealth browser and return a BrowserContext with common options pre-set.
|
||||
All extra kwargs are forwarded to ``browser.new_context()`` — use this for
|
||||
``storage_state``, ``permissions``, ``extra_http_headers``, etc. without needing
|
||||
a persistent profile folder.
|
||||
|
||||
Args:
|
||||
headless: Run in headless mode (default True).
|
||||
proxy: Proxy URL string or Playwright proxy dict (see launch() for details).
|
||||
args: Additional Chromium CLI arguments.
|
||||
stealth_args: Include default stealth fingerprint args (default True).
|
||||
user_agent: Custom user agent string.
|
||||
viewport: Viewport size dict, e.g. {"width": 1920, "height": 1080}.
|
||||
Pass None to disable viewport emulation (use OS window size).
|
||||
locale: Browser locale, e.g. "en-US".
|
||||
timezone: IANA timezone (e.g. 'America/New_York').
|
||||
color_scheme: Color scheme preference — 'light', 'dark', or 'no-preference'.
|
||||
geoip: Auto-detect timezone/locale from proxy IP (default False).
|
||||
backend: Playwright backend — 'playwright' (default) or 'patchright'.
|
||||
humanize: Enable human-like mouse, keyboard, scroll behavior (default False).
|
||||
human_preset: Humanize preset — 'default' or 'careful' (default 'default').
|
||||
human_config: Custom humanize config mapping to override preset values.
|
||||
**kwargs: Passed to browser.new_context() — e.g. storage_state, permissions.
|
||||
|
||||
Returns:
|
||||
Playwright BrowserContext object (async API).
|
||||
Call ``await .close()`` when done — this also closes the underlying browser.
|
||||
|
||||
Example:
|
||||
>>> import asyncio
|
||||
>>> from cloakbrowser import launch_context_async
|
||||
>>>
|
||||
>>> async def main():
|
||||
... # Load saved session (cookies, localStorage)
|
||||
... ctx = await launch_context_async(
|
||||
... headless=True,
|
||||
... storage_state="state.json",
|
||||
... )
|
||||
... page = await ctx.new_page()
|
||||
... await page.goto("https://example.com")
|
||||
... # Save state back
|
||||
... await ctx.storage_state(path="state.json")
|
||||
... await ctx.close()
|
||||
>>>
|
||||
>>> asyncio.run(main())
|
||||
"""
|
||||
timezone = _resolve_timezone(timezone, kwargs)
|
||||
|
||||
# Resolve geoip BEFORE launch_async() to avoid double-resolution and ensure
|
||||
# resolved values flow to binary flags
|
||||
timezone, locale, exit_ip = maybe_resolve_geoip(geoip, proxy, timezone, locale)
|
||||
if exit_ip and not (args and any(a.startswith("--fingerprint-webrtc-ip") for a in args)):
|
||||
args = list(args or [])
|
||||
args.append(f"--fingerprint-webrtc-ip={exit_ip}")
|
||||
# --fingerprint-timezone is process-wide (reads CommandLine in renderer),
|
||||
# so it applies to ALL contexts, not just the default one.
|
||||
# locale and timezone are set via binary flags only — no CDP emulation.
|
||||
browser = await launch_async(headless=headless, proxy=proxy, args=args, stealth_args=stealth_args,
|
||||
timezone=timezone, locale=locale, backend=backend)
|
||||
|
||||
context_kwargs: dict[str, Any] = {}
|
||||
if user_agent:
|
||||
context_kwargs["user_agent"] = user_agent
|
||||
if viewport is _VIEWPORT_UNSET:
|
||||
context_kwargs["viewport"] = DEFAULT_VIEWPORT
|
||||
elif viewport is None:
|
||||
context_kwargs["no_viewport"] = True
|
||||
else:
|
||||
context_kwargs["viewport"] = viewport
|
||||
if color_scheme:
|
||||
context_kwargs["color_scheme"] = color_scheme
|
||||
context_kwargs.update(kwargs)
|
||||
|
||||
# Catch BaseException (not just Exception) so that asyncio.CancelledError
|
||||
# triggers browser cleanup — otherwise the underlying Chromium process
|
||||
# leaks when the awaiting task is cancelled.
|
||||
try:
|
||||
context = await browser.new_context(**context_kwargs)
|
||||
except BaseException:
|
||||
try:
|
||||
await browser.close()
|
||||
except BaseException:
|
||||
pass
|
||||
raise
|
||||
|
||||
# Patch close() to also close the browser (and its Playwright instance)
|
||||
_original_ctx_close = context.close
|
||||
|
||||
async def _close_context_with_cleanup() -> None:
|
||||
try:
|
||||
await _original_ctx_close()
|
||||
finally:
|
||||
await browser.close()
|
||||
|
||||
context.close = _close_context_with_cleanup
|
||||
|
||||
# Human-like behavioral patching (async variant)
|
||||
if humanize:
|
||||
from .human import patch_context_async
|
||||
from .human.config import resolve_config
|
||||
cfg = resolve_config(human_preset, human_config)
|
||||
patch_context_async(context, cfg)
|
||||
|
||||
return context
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backend resolution
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -631,14 +761,120 @@ def _ensure_proxy_scheme(proxy_url: str) -> str:
|
||||
return proxy_url if "://" in proxy_url else f"http://{proxy_url}"
|
||||
|
||||
|
||||
def _assemble_socks_url(
|
||||
scheme: str,
|
||||
host: str,
|
||||
port: int | None,
|
||||
enc_user: str,
|
||||
enc_pass: str | None,
|
||||
path: str = "",
|
||||
params: str = "",
|
||||
query: str = "",
|
||||
fragment: str = "",
|
||||
) -> str:
|
||||
"""Build a SOCKS URL from already-percent-encoded credentials and host parts.
|
||||
|
||||
``enc_pass is None`` means no password (no colon in userinfo). Empty string
|
||||
means present-but-empty (colon preserved). This mirrors the distinction
|
||||
urlparse makes between ``user@host`` and ``user:@host``.
|
||||
"""
|
||||
if ":" in host: # IPv6 literal — re-add brackets
|
||||
host = f"[{host}]"
|
||||
if enc_pass is not None:
|
||||
userinfo = f"{enc_user}:{enc_pass}@"
|
||||
elif enc_user:
|
||||
userinfo = f"{enc_user}@"
|
||||
else:
|
||||
userinfo = ""
|
||||
netloc = f"{userinfo}{host}"
|
||||
if port is not None:
|
||||
netloc += f":{port}"
|
||||
return urlunparse((scheme, netloc, path, params, query, fragment))
|
||||
|
||||
|
||||
def _reconstruct_socks_url(proxy: ProxySettings) -> str:
|
||||
"""Reconstruct a SOCKS5 URL with inline credentials from a Playwright proxy dict."""
|
||||
server = proxy.get("server", "")
|
||||
username = proxy.get("username", "")
|
||||
password = proxy.get("password", "")
|
||||
if not username:
|
||||
return server
|
||||
parsed = urlparse(server)
|
||||
enc_user = quote(username, safe="")
|
||||
# Dict convention: empty/missing password → no colon.
|
||||
enc_pass = quote(password, safe="") if password else None
|
||||
return _assemble_socks_url(
|
||||
parsed.scheme, parsed.hostname or "", parsed.port,
|
||||
enc_user, enc_pass, parsed.path,
|
||||
)
|
||||
|
||||
|
||||
def _normalize_socks_string_url(url: str) -> str:
|
||||
"""Re-encode credentials in a SOCKS5 URL string so Chromium's parser doesn't
|
||||
truncate them at special chars like '='. Idempotent: pre-encoded input stays
|
||||
the same (decoded then re-encoded).
|
||||
|
||||
Emits an INFO log when re-encoding actually changes the URL, so users who
|
||||
previously hit silent SOCKS5 fallback (#157) can see what the wrapper did.
|
||||
Silent on already-encoded inputs (no false-positive noise).
|
||||
|
||||
On unparseable input (invalid port, broken IPv6 literal, etc.) logs a
|
||||
warning and returns the original string — preserves pre-fix pass-through
|
||||
behavior so Chromium's own error handling kicks in.
|
||||
"""
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
# Accessing .port raises ValueError on invalid port strings.
|
||||
_ = parsed.port
|
||||
except ValueError as e:
|
||||
logger.warning("Malformed SOCKS5 proxy URL, passing through unchanged: %s", e)
|
||||
return url
|
||||
# Skip only if no credentials at all (username AND password both absent).
|
||||
# urlparse returns None for absent components, "" for present-but-empty.
|
||||
if parsed.username is None and parsed.password is None:
|
||||
return url
|
||||
raw_user = parsed.username or ""
|
||||
enc_user = quote(unquote(raw_user), safe="") if raw_user else ""
|
||||
# Preserve the colon separator when password component is present, even if
|
||||
# empty, so `user:@host` stays `user:@host`.
|
||||
if parsed.password is not None:
|
||||
raw_pass = parsed.password
|
||||
enc_pass = quote(unquote(raw_pass), safe="") if raw_pass else ""
|
||||
else:
|
||||
raw_pass = None
|
||||
enc_pass = None
|
||||
normalized = _assemble_socks_url(
|
||||
parsed.scheme, parsed.hostname or "", parsed.port,
|
||||
enc_user, enc_pass,
|
||||
parsed.path, parsed.params, parsed.query, parsed.fragment,
|
||||
)
|
||||
# Compare credentials, not the full URL: urlparse cosmetically lowercases
|
||||
# scheme and hostname, so a full-string compare would falsely fire on
|
||||
# `socks5://USER:pass@HOST.com:1080` even when no encoding work happened.
|
||||
if enc_user != raw_user or enc_pass != raw_pass:
|
||||
logger.info(
|
||||
"Auto URL-encoded SOCKS5 proxy credentials (special characters "
|
||||
"detected). Pre-encode the URL to suppress this notice."
|
||||
)
|
||||
return normalized
|
||||
|
||||
|
||||
def _extract_proxy_url(proxy: str | ProxySettings | None) -> str | None:
|
||||
"""Extract and normalize proxy URL string from proxy param."""
|
||||
"""Extract and normalize proxy URL string from proxy param.
|
||||
|
||||
For SOCKS5 dicts with separate username/password fields, reconstructs
|
||||
the full URL with inline credentials so SOCKS5 auth works.
|
||||
"""
|
||||
if proxy is None:
|
||||
return None
|
||||
raw = proxy.get("server") if isinstance(proxy, dict) else proxy
|
||||
if not raw:
|
||||
return None
|
||||
return _ensure_proxy_scheme(raw)
|
||||
if isinstance(proxy, dict):
|
||||
server = proxy.get("server", "")
|
||||
if not server:
|
||||
return None
|
||||
if _is_socks_proxy(proxy):
|
||||
return _reconstruct_socks_url(proxy)
|
||||
return _ensure_proxy_scheme(server)
|
||||
return _ensure_proxy_scheme(proxy)
|
||||
|
||||
|
||||
def maybe_resolve_geoip(
|
||||
@@ -655,7 +891,7 @@ def maybe_resolve_geoip(
|
||||
if not geoip or not proxy:
|
||||
return timezone, locale, None
|
||||
|
||||
from .geoip import resolve_proxy_geo_with_ip
|
||||
from .geoip import resolve_proxy_exit_ip, resolve_proxy_geo_with_ip
|
||||
|
||||
proxy_url = _extract_proxy_url(proxy)
|
||||
if not proxy_url:
|
||||
@@ -663,8 +899,7 @@ def maybe_resolve_geoip(
|
||||
|
||||
# When both tz/locale are explicit, still resolve exit IP for WebRTC
|
||||
if timezone is not None and locale is not None:
|
||||
from .geoip import _resolve_exit_ip
|
||||
exit_ip = _resolve_exit_ip(proxy_url)
|
||||
exit_ip = resolve_proxy_exit_ip(proxy_url)
|
||||
return timezone, locale, exit_ip
|
||||
|
||||
geo_tz, geo_locale, exit_ip = resolve_proxy_geo_with_ip(proxy_url)
|
||||
@@ -694,15 +929,15 @@ def _resolve_webrtc_args(
|
||||
return args
|
||||
proxy_url = _extract_proxy_url(proxy)
|
||||
if not proxy_url:
|
||||
logger.debug("--fingerprint-webrtc-ip=auto but no proxy set — removing flag")
|
||||
logger.warning("--fingerprint-webrtc-ip=auto requires a proxy; removing flag")
|
||||
args = list(args)
|
||||
del args[idx]
|
||||
return args
|
||||
try:
|
||||
from .geoip import _resolve_exit_ip
|
||||
exit_ip = _resolve_exit_ip(proxy_url)
|
||||
from .geoip import resolve_proxy_exit_ip
|
||||
exit_ip = resolve_proxy_exit_ip(proxy_url)
|
||||
except Exception:
|
||||
logger.debug("WebRTC IP resolution failed — removing flag")
|
||||
logger.warning("Failed to resolve proxy exit IP for WebRTC spoofing; removing --fingerprint-webrtc-ip=auto")
|
||||
args = list(args)
|
||||
del args[idx]
|
||||
return args
|
||||
@@ -710,6 +945,7 @@ def _resolve_webrtc_args(
|
||||
args = list(args)
|
||||
args[idx] = f"--fingerprint-webrtc-ip={exit_ip}"
|
||||
else:
|
||||
logger.warning("Could not resolve proxy exit IP for WebRTC spoofing; removing --fingerprint-webrtc-ip=auto")
|
||||
args = list(args)
|
||||
del args[idx]
|
||||
return args
|
||||
@@ -768,11 +1004,14 @@ def build_args(
|
||||
|
||||
|
||||
def _parse_proxy_url(proxy: str) -> dict[str, Any]:
|
||||
"""Parse proxy URL, extracting credentials into separate Playwright fields.
|
||||
"""Parse HTTP(S) proxy URL, extracting credentials into separate Playwright fields.
|
||||
|
||||
Handles: http://user:pass@host:port -> {server: "http://host:port", username: "user", password: "pass"}
|
||||
Also handles: no credentials, URL-encoded special chars, socks5://, missing port,
|
||||
Also handles: no credentials, URL-encoded special chars, missing port,
|
||||
and bare proxy strings without a scheme (e.g. 'user:pass@host:port' -> treated as http).
|
||||
|
||||
SOCKS5 URLs are NOT handled here — they take a dedicated path via
|
||||
``_normalize_socks_string_url`` in ``_resolve_proxy_config``.
|
||||
"""
|
||||
# Bare format: "user:pass@host:port" — urlparse needs a scheme to extract credentials.
|
||||
normalized = proxy
|
||||
@@ -799,10 +1038,42 @@ def _parse_proxy_url(proxy: str) -> dict[str, Any]:
|
||||
return result
|
||||
|
||||
|
||||
def _build_proxy_kwargs(proxy: str | ProxySettings | None) -> dict[str, Any]:
|
||||
"""Build proxy kwargs for Playwright launch."""
|
||||
def _is_socks_proxy(proxy: str | ProxySettings | None) -> bool:
|
||||
"""Check if the proxy uses SOCKS5 protocol."""
|
||||
if proxy is None:
|
||||
return {}
|
||||
return False
|
||||
url = proxy.get("server", "") if isinstance(proxy, dict) else proxy
|
||||
return url.lower().startswith(("socks5://", "socks5h://"))
|
||||
|
||||
|
||||
def _resolve_proxy_config(
|
||||
proxy: str | ProxySettings | None,
|
||||
) -> tuple[dict[str, Any], list[str]]:
|
||||
"""Resolve proxy into Playwright kwargs and Chrome args.
|
||||
|
||||
Playwright rejects SOCKS5 proxies with credentials in its proxy dict,
|
||||
so SOCKS5 is passed via --proxy-server Chrome arg instead.
|
||||
|
||||
Returns:
|
||||
(proxy_kwargs, extra_chrome_args) — one or both will be empty.
|
||||
"""
|
||||
if proxy is None:
|
||||
return {}, []
|
||||
|
||||
if _is_socks_proxy(proxy):
|
||||
# SOCKS5: bypass Playwright, pass directly to Chrome via --proxy-server.
|
||||
# Chrome handles SOCKS5 auth natively from the URL.
|
||||
if isinstance(proxy, dict):
|
||||
url = _reconstruct_socks_url(proxy)
|
||||
extra_args = [f"--proxy-server={url}"]
|
||||
if proxy.get("bypass"):
|
||||
extra_args.append(f"--proxy-bypass-list={proxy['bypass']}")
|
||||
return {}, extra_args
|
||||
# String URL — re-encode creds to work around Chromium parser truncating
|
||||
# passwords at '=' and other special chars (#157).
|
||||
return {}, [f"--proxy-server={_normalize_socks_string_url(proxy)}"]
|
||||
|
||||
# HTTP/HTTPS: use Playwright's proxy dict as before
|
||||
if isinstance(proxy, dict):
|
||||
return {"proxy": proxy}
|
||||
return {"proxy": _parse_proxy_url(proxy)}
|
||||
return {"proxy": proxy}, []
|
||||
return {"proxy": _parse_proxy_url(proxy)}, []
|
||||
|
||||
@@ -15,14 +15,14 @@ from ._version import __version__
|
||||
# CHROMIUM_VERSION is the latest across all platforms (for display/reference).
|
||||
# Use get_chromium_version() for the current platform's actual version.
|
||||
# ---------------------------------------------------------------------------
|
||||
CHROMIUM_VERSION = "146.0.7680.177.1"
|
||||
CHROMIUM_VERSION = "146.0.7680.177.3"
|
||||
|
||||
PLATFORM_CHROMIUM_VERSIONS: dict[str, str] = {
|
||||
"linux-x64": "146.0.7680.177.1",
|
||||
"linux-arm64": "145.0.7632.159.7",
|
||||
"linux-x64": "146.0.7680.177.3",
|
||||
"linux-arm64": "146.0.7680.177.3",
|
||||
"darwin-arm64": "145.0.7632.109.2",
|
||||
"darwin-x64": "145.0.7632.109.2",
|
||||
"windows-x64": "145.0.7632.159.7",
|
||||
"windows-x64": "146.0.7680.177.4",
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -60,7 +60,6 @@ def _show_welcome() -> None:
|
||||
sys.stderr.write(" CloakBrowser — stealth Chromium for automation\n")
|
||||
sys.stderr.write(" https://github.com/CloakHQ/CloakBrowser\n")
|
||||
sys.stderr.write("\n")
|
||||
sys.stderr.write(" Issues? https://github.com/CloakHQ/CloakBrowser/issues\n")
|
||||
sys.stderr.write(" Donate? https://ko-fi.com/cloakhq\n")
|
||||
sys.stderr.write(" Star us if CloakBrowser helps your project!\n")
|
||||
sys.stderr.write("\n")
|
||||
|
||||
+73
-8
@@ -12,6 +12,8 @@ from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import socket
|
||||
import tempfile
|
||||
import threading
|
||||
@@ -27,6 +29,8 @@ GEOIP_DB_URL = (
|
||||
)
|
||||
GEOIP_DB_FILENAME = "GeoLite2-City.mmdb"
|
||||
GEOIP_UPDATE_INTERVAL = 30 * 86_400 # 30 days
|
||||
DEFAULT_GEOIP_TIMEOUT_SECONDS = 5.0
|
||||
GEOIP_TIMEOUT_ENV = "CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS"
|
||||
|
||||
# Country ISO code → BCP 47 locale (covers ~90 % of proxy traffic)
|
||||
COUNTRY_LOCALE_MAP: dict[str, str] = {
|
||||
@@ -77,11 +81,16 @@ def resolve_proxy_geo_with_ip(
|
||||
if db_path is None:
|
||||
return None, None, None
|
||||
|
||||
timeout = _get_geoip_timeout_seconds()
|
||||
deadline = _deadline_from_timeout(timeout)
|
||||
|
||||
# Exit IP (through proxy) is most accurate — gateway DNS may differ from exit
|
||||
ip = _resolve_exit_ip(proxy_url)
|
||||
if ip is None:
|
||||
ip = _resolve_exit_ip(proxy_url, timeout=_remaining_seconds(deadline))
|
||||
if ip is None and not _deadline_expired(deadline):
|
||||
ip = _resolve_proxy_ip(proxy_url)
|
||||
if ip is None:
|
||||
if ip is None or _deadline_expired(deadline):
|
||||
if deadline is not None and _deadline_expired(deadline):
|
||||
logger.warning("GeoIP resolution timed out after %.1fs; continuing without GeoIP", timeout)
|
||||
return None, None, None
|
||||
|
||||
try:
|
||||
@@ -96,7 +105,7 @@ def resolve_proxy_geo_with_ip(
|
||||
)
|
||||
return timezone, locale, ip
|
||||
except Exception as exc:
|
||||
logger.debug("GeoIP lookup failed for %s: %s", ip, exc)
|
||||
logger.warning("GeoIP lookup failed for %s: %s", ip, exc)
|
||||
return None, None, ip
|
||||
|
||||
|
||||
@@ -132,7 +141,7 @@ def _resolve_proxy_ip(proxy_url: str) -> str | None:
|
||||
return ip
|
||||
return None
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to resolve proxy hostname: %s", exc)
|
||||
logger.warning("Failed to resolve proxy hostname: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
@@ -152,22 +161,78 @@ _IP_ECHO_URLS = [
|
||||
]
|
||||
|
||||
|
||||
def _resolve_exit_ip(proxy_url: str) -> str | None:
|
||||
def _get_geoip_timeout_seconds() -> float:
|
||||
raw = os.getenv(GEOIP_TIMEOUT_ENV)
|
||||
if not raw:
|
||||
return DEFAULT_GEOIP_TIMEOUT_SECONDS
|
||||
try:
|
||||
timeout = float(raw)
|
||||
except ValueError:
|
||||
timeout = float("nan")
|
||||
if not math.isfinite(timeout):
|
||||
logger.warning(
|
||||
"Invalid %s=%r; using %.1fs",
|
||||
GEOIP_TIMEOUT_ENV,
|
||||
raw,
|
||||
DEFAULT_GEOIP_TIMEOUT_SECONDS,
|
||||
)
|
||||
return DEFAULT_GEOIP_TIMEOUT_SECONDS
|
||||
return max(timeout, 0.0)
|
||||
|
||||
|
||||
def _deadline_from_timeout(timeout: float) -> float | None:
|
||||
if timeout <= 0:
|
||||
return None
|
||||
return time.monotonic() + timeout
|
||||
|
||||
|
||||
def _remaining_seconds(deadline: float | None) -> float | None:
|
||||
if deadline is None:
|
||||
return None
|
||||
return max(deadline - time.monotonic(), 0.0)
|
||||
|
||||
|
||||
def _deadline_expired(deadline: float | None) -> bool:
|
||||
return deadline is not None and time.monotonic() >= deadline
|
||||
|
||||
|
||||
def resolve_proxy_exit_ip(proxy_url: str) -> str | None:
|
||||
"""Resolve only the proxy exit IP, bounded by the GeoIP timeout."""
|
||||
timeout = _get_geoip_timeout_seconds()
|
||||
deadline = _deadline_from_timeout(timeout)
|
||||
ip = _resolve_exit_ip(proxy_url, timeout=timeout)
|
||||
if ip is None and _deadline_expired(deadline):
|
||||
logger.warning("GeoIP resolution timed out after %.1fs; continuing without GeoIP", timeout)
|
||||
return ip
|
||||
|
||||
|
||||
def _resolve_exit_ip(proxy_url: str, timeout: float | None = None) -> str | None:
|
||||
"""Discover the proxy's actual exit IP by connecting through it."""
|
||||
import httpx
|
||||
|
||||
deadline = _deadline_from_timeout(timeout or 0)
|
||||
|
||||
for url in _IP_ECHO_URLS:
|
||||
try:
|
||||
resp = httpx.get(url, proxy=proxy_url, timeout=10.0)
|
||||
remaining = _remaining_seconds(deadline)
|
||||
if remaining is not None and remaining <= 0:
|
||||
return None
|
||||
request_timeout = min(10.0, remaining) if remaining is not None else 10.0
|
||||
resp = httpx.get(url, proxy=proxy_url, timeout=request_timeout)
|
||||
resp.raise_for_status()
|
||||
ip = resp.text.strip()
|
||||
# Validate it looks like an IP
|
||||
ipaddress.ip_address(ip)
|
||||
logger.debug("Exit IP via %s: %s", url, ip)
|
||||
return ip
|
||||
except httpx.UnsupportedProtocol:
|
||||
logger.warning(
|
||||
"SOCKS5 proxy requires socksio: pip install cloakbrowser[geoip]"
|
||||
)
|
||||
return None
|
||||
except Exception:
|
||||
continue
|
||||
logger.debug("Failed to discover exit IP through proxy")
|
||||
logger.warning("Failed to discover exit IP through proxy")
|
||||
return None
|
||||
|
||||
|
||||
|
||||
+917
-59
File diff suppressed because it is too large
Load Diff
@@ -10,7 +10,7 @@ import math
|
||||
import random
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Literal, Tuple
|
||||
from typing import Literal, Tuple, TypedDict
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Type alias
|
||||
@@ -20,6 +20,50 @@ Range = Tuple[float, float]
|
||||
HumanPreset = Literal["default", "careful"]
|
||||
|
||||
|
||||
class HumanConfigOverrides(TypedDict, total=False):
|
||||
typing_delay: float
|
||||
typing_delay_spread: float
|
||||
typing_pause_chance: float
|
||||
typing_pause_range: Range
|
||||
shift_down_delay: Range
|
||||
shift_up_delay: Range
|
||||
key_hold: Range
|
||||
field_switch_delay: Range
|
||||
mistype_chance: float
|
||||
mistype_delay_notice: Range
|
||||
mistype_delay_correct: Range
|
||||
mouse_steps_divisor: float
|
||||
mouse_min_steps: int
|
||||
mouse_max_steps: int
|
||||
mouse_wobble_max: float
|
||||
mouse_overshoot_chance: float
|
||||
mouse_overshoot_px: Range
|
||||
mouse_burst_size: Range
|
||||
mouse_burst_pause: Range
|
||||
click_aim_delay_input: Range
|
||||
click_aim_delay_button: Range
|
||||
click_hold_input: Range
|
||||
click_hold_button: Range
|
||||
click_input_x_range: Range
|
||||
idle_drift_px: float
|
||||
idle_pause_range: Range
|
||||
scroll_delta_base: Range
|
||||
scroll_delta_variance: float
|
||||
scroll_pause_fast: Range
|
||||
scroll_pause_slow: Range
|
||||
scroll_accel_steps: Range
|
||||
scroll_decel_steps: Range
|
||||
scroll_overshoot_chance: float
|
||||
scroll_overshoot_px: Range
|
||||
scroll_settle_delay: Range
|
||||
scroll_target_zone: Range
|
||||
scroll_pre_move_delay: Range
|
||||
initial_cursor_x: Range
|
||||
initial_cursor_y: Range
|
||||
idle_between_actions: bool
|
||||
idle_between_duration: Range
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Configuration dataclass
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -130,13 +174,13 @@ _PRESETS: dict[str, HumanConfig] = {
|
||||
|
||||
def resolve_config(
|
||||
preset: HumanPreset = "default",
|
||||
overrides: dict | None = None,
|
||||
overrides: HumanConfigOverrides | None = None,
|
||||
) -> HumanConfig:
|
||||
"""Resolve a preset name + optional overrides into a full HumanConfig.
|
||||
|
||||
Args:
|
||||
preset: 'default' or 'careful'.
|
||||
overrides: Dict of field names to override values.
|
||||
overrides: Typed mapping of HumanConfig field names to override values.
|
||||
|
||||
Returns:
|
||||
A new HumanConfig instance.
|
||||
@@ -157,6 +201,25 @@ def resolve_config(
|
||||
return HumanConfig(**merged)
|
||||
|
||||
|
||||
def merge_config(base: HumanConfig, overrides: dict | None) -> HumanConfig:
|
||||
"""Merge ``overrides`` (a dict of HumanConfig field names → values) on top of
|
||||
``base``. Returns a new HumanConfig — ``base`` is never mutated.
|
||||
|
||||
Used by per-call overrides like ``page.type(sel, text, human_config={...})``
|
||||
so the same page can use different timings for different inputs without
|
||||
re-patching.
|
||||
|
||||
Unknown keys are ignored silently to keep this forgiving for callers.
|
||||
"""
|
||||
if not overrides:
|
||||
return base
|
||||
merged = {k: getattr(base, k) for k in base.__dataclass_fields__}
|
||||
for k, v in overrides.items():
|
||||
if k in base.__dataclass_fields__:
|
||||
merged[k] = v
|
||||
return HumanConfig(**merged)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Utility functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
import math
|
||||
import random
|
||||
from typing import Any, Optional, Tuple
|
||||
from typing import Any, Callable, Optional, Tuple
|
||||
|
||||
from .config import HumanConfig, rand, rand_range, rand_int_range, sleep_ms
|
||||
from .mouse import RawMouse, human_move
|
||||
@@ -18,10 +18,15 @@ def _is_in_viewport(bounds: dict, viewport_height: int, cfg: HumanConfig) -> boo
|
||||
return top_edge >= zone_top and bottom_edge <= zone_bottom
|
||||
|
||||
|
||||
def _get_element_box(page: Any, selector: str) -> Optional[dict]:
|
||||
def _get_element_box(page: Any, selector: str, timeout: float = 30000) -> Optional[dict]:
|
||||
"""Locate ``selector`` and return its bounding box.
|
||||
|
||||
The ``timeout`` is forwarded to Playwright's ``boundingBox(timeout=...)``
|
||||
so callers can extend it for slow-loading elements (#172).
|
||||
"""
|
||||
try:
|
||||
el = page.locator(selector).first
|
||||
return el.bounding_box(timeout=2000)
|
||||
return el.bounding_box(timeout=timeout)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@@ -39,13 +44,21 @@ def _smooth_wheel(raw: RawMouse, delta: int, cfg: HumanConfig) -> None:
|
||||
sleep_ms(rand(8, 20))
|
||||
|
||||
|
||||
def scroll_to_element(
|
||||
def human_scroll_into_view(
|
||||
page: Any,
|
||||
raw: RawMouse,
|
||||
selector: str,
|
||||
get_box: Callable[[], Optional[dict]],
|
||||
cursor_x: float, cursor_y: float,
|
||||
cfg: HumanConfig,
|
||||
) -> Tuple[dict, float, float]:
|
||||
"""Humanized scrolling that uses an arbitrary ``get_box`` callable
|
||||
instead of a CSS selector.
|
||||
|
||||
Used both by ``scroll_to_element`` (selector-based) and by
|
||||
``ElementHandle.scroll_into_view_if_needed`` / ``Locator.scroll_into_view_if_needed``
|
||||
(handle-based) so the same accelerate \u2192 cruise \u2192 decelerate \u2192 overshoot
|
||||
behavior runs everywhere.
|
||||
"""
|
||||
viewport = page.viewport_size
|
||||
if not viewport:
|
||||
raise RuntimeError("Viewport size not available")
|
||||
@@ -53,12 +66,9 @@ def scroll_to_element(
|
||||
viewport_height = viewport["height"]
|
||||
viewport_width = viewport["width"]
|
||||
|
||||
box = _get_element_box(page, selector)
|
||||
box = get_box()
|
||||
if box is None:
|
||||
sleep_ms(200)
|
||||
box = _get_element_box(page, selector)
|
||||
if box is None:
|
||||
raise RuntimeError(f"Element not found: {selector}")
|
||||
raise RuntimeError("Element not found while scrolling into view")
|
||||
|
||||
if _is_in_viewport(box, viewport_height, cfg):
|
||||
return box, cursor_x, cursor_y
|
||||
@@ -105,7 +115,7 @@ def scroll_to_element(
|
||||
|
||||
# Check visibility every 3 steps
|
||||
if i % 3 == 2 or i == total_clicks - 1:
|
||||
box = _get_element_box(page, selector)
|
||||
box = get_box()
|
||||
if box and _is_in_viewport(box, viewport_height, cfg):
|
||||
break
|
||||
if scrolled >= abs_distance * 1.1:
|
||||
@@ -125,8 +135,29 @@ def scroll_to_element(
|
||||
# Settle
|
||||
sleep_ms(rand_range(cfg.scroll_settle_delay))
|
||||
|
||||
box = _get_element_box(page, selector)
|
||||
box = get_box()
|
||||
if box is None:
|
||||
raise RuntimeError(f"Element lost after scrolling: {selector}")
|
||||
raise RuntimeError("Element lost after scrolling into view")
|
||||
|
||||
return box, cursor_x, cursor_y
|
||||
|
||||
|
||||
def scroll_to_element(
|
||||
page: Any,
|
||||
raw: RawMouse,
|
||||
selector: str,
|
||||
cursor_x: float, cursor_y: float,
|
||||
cfg: HumanConfig,
|
||||
timeout: float = 30000,
|
||||
) -> Tuple[dict, float, float]:
|
||||
"""Selector-based humanized scroll.
|
||||
|
||||
``timeout`` is forwarded to ``locator.bounding_box(timeout=...)`` so callers
|
||||
such as ``page.click('#x', timeout=5000)`` can wait longer for slow elements
|
||||
(#172). Default matches Playwright's 30000ms when not specified.
|
||||
"""
|
||||
return human_scroll_into_view(
|
||||
page, raw,
|
||||
lambda: _get_element_box(page, selector, timeout),
|
||||
cursor_x, cursor_y, cfg,
|
||||
)
|
||||
|
||||
@@ -8,17 +8,22 @@ from __future__ import annotations
|
||||
|
||||
import math
|
||||
import random
|
||||
from typing import Any, Optional, Tuple
|
||||
from typing import Any, Awaitable, Callable, Optional, Tuple
|
||||
|
||||
from .config import HumanConfig, rand, rand_range, rand_int_range, async_sleep_ms
|
||||
from .mouse_async import AsyncRawMouse, async_human_move
|
||||
from .scroll import _is_in_viewport
|
||||
|
||||
|
||||
async def _get_element_box_async(page: Any, selector: str) -> Optional[dict]:
|
||||
async def _get_element_box_async(
|
||||
page: Any, selector: str, timeout: float = 30000,
|
||||
) -> Optional[dict]:
|
||||
"""Async variant. ``timeout`` is forwarded to Playwright's
|
||||
``boundingBox(timeout=...)`` so callers can extend it for slow-loading
|
||||
elements (#172)."""
|
||||
try:
|
||||
el = page.locator(selector).first
|
||||
return await el.bounding_box(timeout=2000)
|
||||
return await el.bounding_box(timeout=timeout)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@@ -36,13 +41,20 @@ async def _async_smooth_wheel(raw: AsyncRawMouse, delta: int, cfg: HumanConfig)
|
||||
await async_sleep_ms(rand(8, 20))
|
||||
|
||||
|
||||
async def async_scroll_to_element(
|
||||
async def async_human_scroll_into_view(
|
||||
page: Any,
|
||||
raw: AsyncRawMouse,
|
||||
selector: str,
|
||||
get_box: Callable[[], Awaitable[Optional[dict]]],
|
||||
cursor_x: float, cursor_y: float,
|
||||
cfg: HumanConfig,
|
||||
) -> Tuple[dict, float, float]:
|
||||
"""Humanized scrolling using an arbitrary async ``get_box`` callable.
|
||||
|
||||
Used by both ``async_scroll_to_element`` (selector-based) and the
|
||||
ElementHandle / Locator ``scroll_into_view_if_needed`` patches so all
|
||||
scrolling paths share the same accelerate \u2192 cruise \u2192 decelerate
|
||||
\u2192 overshoot behavior.
|
||||
"""
|
||||
viewport = page.viewport_size
|
||||
if not viewport:
|
||||
raise RuntimeError("Viewport size not available")
|
||||
@@ -50,12 +62,9 @@ async def async_scroll_to_element(
|
||||
viewport_height = viewport["height"]
|
||||
viewport_width = viewport["width"]
|
||||
|
||||
box = await _get_element_box_async(page, selector)
|
||||
box = await get_box()
|
||||
if box is None:
|
||||
await async_sleep_ms(200)
|
||||
box = await _get_element_box_async(page, selector)
|
||||
if box is None:
|
||||
raise RuntimeError(f"Element not found: {selector}")
|
||||
raise RuntimeError("Element not found while scrolling into view")
|
||||
|
||||
if _is_in_viewport(box, viewport_height, cfg):
|
||||
return box, cursor_x, cursor_y
|
||||
@@ -102,7 +111,7 @@ async def async_scroll_to_element(
|
||||
|
||||
# Check visibility every 3 steps
|
||||
if i % 3 == 2 or i == total_clicks - 1:
|
||||
box = await _get_element_box_async(page, selector)
|
||||
box = await get_box()
|
||||
if box and _is_in_viewport(box, viewport_height, cfg):
|
||||
break
|
||||
if scrolled >= abs_distance * 1.1:
|
||||
@@ -122,8 +131,29 @@ async def async_scroll_to_element(
|
||||
# Settle
|
||||
await async_sleep_ms(rand_range(cfg.scroll_settle_delay))
|
||||
|
||||
box = await _get_element_box_async(page, selector)
|
||||
box = await get_box()
|
||||
if box is None:
|
||||
raise RuntimeError(f"Element lost after scrolling: {selector}")
|
||||
raise RuntimeError("Element lost after scrolling into view")
|
||||
|
||||
return box, cursor_x, cursor_y
|
||||
|
||||
|
||||
async def async_scroll_to_element(
|
||||
page: Any,
|
||||
raw: AsyncRawMouse,
|
||||
selector: str,
|
||||
cursor_x: float, cursor_y: float,
|
||||
cfg: HumanConfig,
|
||||
timeout: float = 30000,
|
||||
) -> Tuple[dict, float, float]:
|
||||
"""Selector-based humanized scroll (async).
|
||||
|
||||
``timeout`` is forwarded to ``locator.bounding_box(timeout=...)`` so callers
|
||||
such as ``page.click('#x', timeout=5000)`` can wait longer for slow elements
|
||||
(#172). Default matches Playwright's 30000ms when not specified.
|
||||
"""
|
||||
async def _get():
|
||||
return await _get_element_box_async(page, selector, timeout)
|
||||
return await async_human_scroll_into_view(
|
||||
page, raw, _get, cursor_x, cursor_y, cfg,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# CloakBrowser on AWS Lambda — derived from the official CloakHQ image.
|
||||
#
|
||||
# `FROM cloakhq/cloakbrowser:<tag>` is an official distribution channel under
|
||||
# the CloakBrowser Binary License — pulling it isn't redistribution. We just
|
||||
# layer Lambda glue on top: the Lambda Runtime Interface Client (awslambdaric),
|
||||
# the Lambda Runtime Interface Emulator (for local `docker run` testing), the
|
||||
# dual-mode entrypoint, and the handler module.
|
||||
#
|
||||
# This directory is self-contained — copy/clone it anywhere and build from
|
||||
# inside it. No files outside this directory are referenced.
|
||||
#
|
||||
# ─── Lambda invocation (default CMD) ──────────────────────────────────────────
|
||||
# # From inside this directory:
|
||||
# docker buildx build --platform linux/arm64 -t cloakbrowser-lambda:arm64 --load .
|
||||
#
|
||||
# # Or from a parent dir, pointing at this directory as the build context:
|
||||
# docker buildx build --platform linux/arm64 \
|
||||
# -f path/to/aws_lambda/Dockerfile -t cloakbrowser-lambda:arm64 --load \
|
||||
# path/to/aws_lambda
|
||||
#
|
||||
# docker run --rm -p 9000:8080 cloakbrowser-lambda:arm64
|
||||
# curl -XPOST http://localhost:9000/2015-03-31/functions/function/invocations \
|
||||
# -d '{"url":"https://example.com"}'
|
||||
#
|
||||
# ─── Same as the canonical CloakHQ image (CMD overridden) ─────────────────────
|
||||
# docker run --rm -it cloakbrowser-lambda:arm64 python # REPL
|
||||
# docker run --rm cloakbrowser-lambda:arm64 python examples/basic.py # examples
|
||||
# docker run --rm -p 9222:9222 cloakbrowser-lambda:arm64 cloakserve --port=9222 # CDP server
|
||||
# docker run --rm cloakbrowser-lambda:arm64 cloaktest # stealth tests
|
||||
# docker run --rm -it cloakbrowser-lambda:arm64 node # JS wrapper
|
||||
# docker run --rm -it cloakbrowser-lambda:arm64 bash # shell
|
||||
#
|
||||
# Pin a specific tag (e.g. cloakhq/cloakbrowser:0.3.25) for reproducible builds;
|
||||
# `latest` floats with CloakHQ's release cadence.
|
||||
|
||||
FROM cloakhq/cloakbrowser:latest
|
||||
|
||||
# ─── Lambda Runtime Interface Client ──────────────────────────────────────────
|
||||
RUN pip install --no-cache-dir awslambdaric
|
||||
|
||||
# ─── Lambda Runtime Interface Emulator (local `docker run` testing) ───────────
|
||||
# Bundled into the image so users can hit the standard local-invoke endpoint
|
||||
# without mounting the RIE separately. TARGETARCH is provided by buildx.
|
||||
ARG TARGETARCH
|
||||
ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie-${TARGETARCH} \
|
||||
/usr/local/bin/aws-lambda-rie
|
||||
RUN chmod +x /usr/local/bin/aws-lambda-rie
|
||||
|
||||
# ─── Lambda glue ──────────────────────────────────────────────────────────────
|
||||
# Dual-mode entrypoint replaces the canonical bin/docker-entrypoint.sh: same
|
||||
# Xvfb startup, plus routing for `module.func` CMDs through awslambdaric.
|
||||
COPY lambda-entrypoint.sh /entrypoint.sh
|
||||
RUN chmod +x /entrypoint.sh
|
||||
|
||||
# Handler sits at /app (already on Python's import path in the canonical image,
|
||||
# WORKDIR=/app), imports cloakbrowser as a normal library.
|
||||
COPY lambda_handler.py /app/lambda_handler.py
|
||||
|
||||
# ─── Lambda non-root readability fix ──────────────────────────────────────────
|
||||
# The canonical image bakes the Chromium binary at /root/.cloakbrowser/ (root's
|
||||
# HOME at build time). Lambda runs the container as a non-root user that can't
|
||||
# read /root by default (mode 750). Make the whole binary tree world-readable
|
||||
# and traversable. Also restore the .welcome_shown marker the canonical image
|
||||
# rm's (Lambda's read-only runtime FS can't recreate it, so the welcome would
|
||||
# print to CloudWatch on every cold start otherwise).
|
||||
RUN touch /root/.cloakbrowser/.welcome_shown \
|
||||
&& chmod -R o+rX /root /root/.cloakbrowser
|
||||
|
||||
# ─── Lambda runtime env ───────────────────────────────────────────────────────
|
||||
# HOME=/tmp gives Chromium a writable scratch dir (Lambda only allows writes
|
||||
# under /tmp). CLOAKBROWSER_CACHE_DIR points at the baked binary location since
|
||||
# HOME=/tmp would otherwise make get_cache_dir() resolve to /tmp/.cloakbrowser
|
||||
# (empty). Auto-update is disabled because the runtime FS is read-only.
|
||||
ENV HOME=/tmp \
|
||||
CLOAKBROWSER_CACHE_DIR=/root/.cloakbrowser \
|
||||
CLOAKBROWSER_AUTO_UPDATE=false
|
||||
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
CMD ["lambda_handler.handler"]
|
||||
@@ -0,0 +1,181 @@
|
||||
# CloakBrowser on AWS Lambda
|
||||
|
||||
Run stealth Chromium one-shot scrapes inside an AWS Lambda function (container image package type). The image derives directly from the official CloakHQ Docker Hub image (`cloakhq/cloakbrowser`) and adds Lambda runtime support on top — Lambda is an additional invocation surface, not a replacement. Every other surface from the canonical image (`python`, `cloakserve`, `cloaktest`, `node`, `bash`, examples) keeps working.
|
||||
|
||||
This document covers what the image is, how to build and locally test it, and the event/response contract. **It does not prescribe a deployment method** — push the resulting image to ECR and create the Lambda function however you prefer (AWS CLI, CDK, Terraform, SAM, console, etc.). Configuration tips for whichever tool you use are at the bottom.
|
||||
|
||||
## Files in this directory
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `Dockerfile` | `FROM cloakhq/cloakbrowser` plus a thin Lambda layer. Self-contained — no files outside this directory are referenced. |
|
||||
| `lambda-entrypoint.sh` | Dual-mode entrypoint. Starts Xvfb, then routes `module.func` CMDs through `awslambdaric` (via the bundled `aws-lambda-rie` locally, or the AWS Runtime API in production), and execs everything else (`python`, `cloakserve`, `cloaktest`, `node`, `bash`) directly. |
|
||||
| `lambda_handler.py` | Default handler. Takes `{url, ...}`, returns `{title, url, html, screenshot_b64?}`. Always headed via Xvfb. |
|
||||
| `INSTRUCTIONS.md` | This file. |
|
||||
|
||||
The Lambda layer is ~30 lines on top of the official image — no apt list, no Node install, no JS-wrapper build, no Chromium download. The canonical CloakHQ image owns those.
|
||||
|
||||
This directory is **standalone**: copy or clone it anywhere (its own repo, a subdirectory of an existing project, a CI artifact bundle) and the build still works. It depends only on the upstream `cloakhq/cloakbrowser` image on Docker Hub and the `aws-lambda-rie` binary on GitHub Releases — both fetched at build time.
|
||||
|
||||
## Build
|
||||
|
||||
From inside this directory:
|
||||
|
||||
```bash
|
||||
docker buildx build --platform linux/arm64 -t cloakbrowser-lambda:arm64 --load .
|
||||
```
|
||||
|
||||
Or from anywhere, pointing at this directory as the build context:
|
||||
|
||||
```bash
|
||||
docker buildx build --platform linux/arm64 \
|
||||
-f path/to/aws_lambda/Dockerfile \
|
||||
-t cloakbrowser-lambda:arm64 --load \
|
||||
path/to/aws_lambda
|
||||
```
|
||||
|
||||
The build pulls `cloakhq/cloakbrowser:latest` from Docker Hub and adds the Lambda layer on top. Pin a specific tag (e.g. `cloakhq/cloakbrowser:0.3.25`) in the `FROM` line for reproducible builds; `latest` floats with the upstream release cadence.
|
||||
|
||||
For x86_64, switch `--platform linux/amd64` (slower on Apple Silicon under emulation).
|
||||
|
||||
## Local smoke test (no AWS account needed)
|
||||
|
||||
> **What's the RIE?** Lambda container images can't be run with a plain `docker run` — they expect to talk to AWS's Runtime API (the HTTP service Lambda exposes inside its sandbox to deliver events and collect responses). AWS publishes a small binary called the **Runtime Interface Emulator** that stands up a fake Runtime API on localhost so you can test the container exactly the way Lambda will invoke it, without deploying. We bake the RIE into the image, and the dual-mode entrypoint uses it automatically when `AWS_LAMBDA_RUNTIME_API` isn't set (i.e. you're not running in real Lambda).
|
||||
|
||||
The image bakes in `aws-lambda-rie`, so the standard Lambda local-invoke endpoint works without mounting anything:
|
||||
|
||||
```bash
|
||||
docker run --rm -p 9000:8080 cloakbrowser-lambda:arm64
|
||||
|
||||
# In another shell:
|
||||
curl -sS -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" \
|
||||
-d '{"url":"https://example.com"}'
|
||||
```
|
||||
|
||||
Other invocation surfaces stay intact (these match the canonical CloakHQ image):
|
||||
|
||||
```bash
|
||||
docker run --rm -it cloakbrowser-lambda:arm64 python # REPL
|
||||
docker run --rm cloakbrowser-lambda:arm64 python examples/basic.py # examples
|
||||
docker run --rm -p 9222:9222 cloakbrowser-lambda:arm64 cloakserve --port=9222 # CDP server
|
||||
docker run --rm cloakbrowser-lambda:arm64 cloaktest # stealth tests
|
||||
docker run --rm -it cloakbrowser-lambda:arm64 node # JS wrapper
|
||||
```
|
||||
|
||||
## Event schema
|
||||
|
||||
Only `url` is required. Everything else is optional.
|
||||
|
||||
### Launch options (forwarded to `cloakbrowser.launch_context_async`)
|
||||
|
||||
| Field | Type | Default |
|
||||
|---|---|---|
|
||||
| `url` | str | required |
|
||||
| `proxy` | str / dict | none — `http://user:pass@host:port` or a Playwright proxy dict |
|
||||
| `humanize` | bool | `false` — enable human-like mouse / keyboard / scroll |
|
||||
| `human_preset` | str | `"default"` or `"careful"` |
|
||||
| `geoip` | bool | `false` — auto timezone+locale from proxy IP |
|
||||
| `timezone` | str | none — IANA tz, e.g. `"America/New_York"` |
|
||||
| `locale` | str | none — BCP-47, e.g. `"en-US"` |
|
||||
| `viewport` | `{width,height}` | `1920x947` (cloakbrowser default) |
|
||||
| `user_agent` | str | none |
|
||||
| `extra_args` | `list[str]` | `[]` — extra Chromium CLI flags |
|
||||
|
||||
### Navigation
|
||||
|
||||
| Field | Type | Default |
|
||||
|---|---|---|
|
||||
| `wait_until` | str | `"domcontentloaded"` — `load` / `domcontentloaded` / `networkidle` / `commit` |
|
||||
| `goto_timeout_ms` | int | `30000` |
|
||||
|
||||
### Post-navigation waits
|
||||
|
||||
`smart_wait` is the default when no other wait is specified. It polls `document.documentElement.outerHTML.length` and returns when the size hasn't changed for `dom_stable_ms`. Robust for at-scale scraping because it ignores network activity (analytics beacons, long-poll, websockets) that doesn't mutate the DOM — `wait_until: "networkidle"` is unreliable on modern SPAs for exactly this reason.
|
||||
|
||||
| Field | Type | Default |
|
||||
|---|---|---|
|
||||
| `smart_wait` | bool | `true` if no other wait is set |
|
||||
| `dom_stable_ms` | int | `1500` |
|
||||
| `max_settle_ms` | int | `15000` |
|
||||
| `wait_for_load_state` | str | none — `load` / `domcontentloaded` / `networkidle` |
|
||||
| `wait_for_load_state_timeout_ms` | int | `30000` |
|
||||
| `wait_for_selector` | str | none — CSS or XPath |
|
||||
| `wait_for_selector_state` | str | `"visible"` — also `attached` / `detached` / `hidden` |
|
||||
| `wait_for_selector_timeout_ms` | int | `30000` |
|
||||
| `wait_for_function` | str | none — JS expression returning truthy when ready |
|
||||
| `wait_for_function_timeout_ms` | int | `30000` |
|
||||
| `wait_ms` | int | none — fixed pause |
|
||||
|
||||
### Capture
|
||||
|
||||
| Field | Type | Default |
|
||||
|---|---|---|
|
||||
| `screenshot` | bool | `true` |
|
||||
| `full_page_screenshot` | bool | `false` |
|
||||
|
||||
### Retry orchestration
|
||||
|
||||
The handler retries transient navigation failures inline within the same Lambda invocation. Two layers, both built-in:
|
||||
|
||||
- **Launch retries** — 3 attempts with 0.3 s + 0.6 s backoff. Recovers Xvfb / Chromium spawn races at cold start. Fast and cheap; not configurable.
|
||||
- **Strategy retries** — default 1 attempt, configurable via the `retries` event field. Recovers specific post-launch error classes by relaunching with adjusted Chromium args / page-load budgets.
|
||||
|
||||
| Field | Type | Default |
|
||||
|---|---|---|
|
||||
| `retries` | int | `1` — number of strategy-retry attempts after the first failure. Set to `0` to disable retry entirely. |
|
||||
|
||||
Strategies (priority order — first match wins):
|
||||
|
||||
| Error pattern | Strategy applied |
|
||||
|---|---|
|
||||
| `ERR_CERT_*` (any cert error) | `extra_args: ["--ignore-certificate-errors"]`, `goto_timeout_ms: 60000` |
|
||||
| `Timeout … exceeded` | `goto_timeout_ms: 90000`, `max_settle_ms: 25000` |
|
||||
| `ERR_CONNECTION_TIMED_OUT` | same as `Timeout … exceeded` |
|
||||
|
||||
Errors that are **not retried** (no anonymous scraper can recover): `ERR_NAME_NOT_RESOLVED`, `ERR_SSL_PROTOCOL_ERROR`, `ERR_CONNECTION_REFUSED`, `ERR_HTTP_RESPONSE_CODE_FAILURE`. These bail immediately.
|
||||
|
||||
On final failure, the raised `RuntimeError`'s message includes a `retry_history` block listing every attempt (strategy applied + error seen). Successful invocations return the standard response shape unchanged — no surprise fields when retries didn't fire.
|
||||
|
||||
### Response
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "...",
|
||||
"url": "https://example.com/",
|
||||
"html": "<!DOCTYPE html>...",
|
||||
"screenshot_b64": "<base64 PNG>"
|
||||
}
|
||||
```
|
||||
|
||||
## Lambda-specific Chromium hardening (baked in, do not remove)
|
||||
|
||||
Two flags are forced on every launch by `lambda_handler.py`:
|
||||
|
||||
- `--disable-dev-shm-usage` — Lambda's `/dev/shm` is ~64 MB; Chromium's renderer crashes mid-paint without this.
|
||||
- `--no-zygote` — Lambda's restricted process model can't fork from Chromium's zygote process; without this the browser launches but child renderers fail to spawn and the first `page.new_page()` raises `TargetClosedError`.
|
||||
|
||||
## Function configuration recommendations
|
||||
|
||||
Whatever tool you use to create the Lambda function (CLI, CDK, Terraform, SAM, console), apply these settings:
|
||||
|
||||
| Setting | Value | Why |
|
||||
|---|---|---|
|
||||
| Package type | Image | Required — this is a container image, not a zip. |
|
||||
| Architecture | `arm64` | Roughly 20% cheaper than x86_64. Native build on Apple Silicon. Match the architecture you built for. |
|
||||
| Memory | 3008 MB | Memory in Lambda is tied to vCPU. Below ~1769 MB Chromium starts noticeably slower. |
|
||||
| Timeout | 120–180 s | Single-attempt scrapes complete in 3–15 s warm; under retry, a `Timeout`-class first failure (30 s default) plus a longer-budget retry (90 s) plus cleanup can total ~120-130 s. 180 s leaves headroom; below 120 s the function will time out before the retry completes. Cold-start init adds 5-10 s on top. |
|
||||
| Ephemeral storage (`/tmp`) | 1024 MB | Chromium profile dirs and screenshots can fill the 512 MB default. |
|
||||
| Networking | Default (no VPC) | Binary is baked in, no network needed at cold start. Add VPC + NAT only if your proxy egress requires it. |
|
||||
| Execution role | `AWSLambdaBasicExecutionRole` | Just CloudWatch Logs. Add more permissions only if your handler needs them. |
|
||||
|
||||
## Cold start
|
||||
|
||||
First invocation in a new container takes ~80–90 s (image extraction, Chromium binary mmap, JS engine warmup, no DNS/TLS caches). Subsequent warm invocations on the same container are 3–15 s.
|
||||
|
||||
For latency-sensitive use cases: provision concurrency, schedule a CloudWatch/EventBridge warmer ping, or accept the cold tail.
|
||||
|
||||
If you see empty/missing dynamic content on cold-start invocations, raise `max_settle_ms` in the event payload (e.g. `25000`) — the default `15000` is tuned for warm runs.
|
||||
|
||||
## License
|
||||
|
||||
The patched Chromium binary inside the upstream `cloakhq/cloakbrowser` image is governed by the **CloakBrowser Binary License** (published at https://github.com/CloakHQ/CloakBrowser/blob/main/BINARY-LICENSE.md). Internal organizational use (private ECR, your own scraping pipelines, your own business) is free. Exposing this Lambda as a paid API to third-party customers — i.e. browser-as-a-service — requires an OEM/SaaS license from CloakHQ (`cloakhq@pm.me`). Do not push the resulting image to a public registry; that would be redistribution and is prohibited.
|
||||
@@ -0,0 +1,52 @@
|
||||
#!/bin/sh
|
||||
# Dual-mode entrypoint for the CloakBrowser Lambda image.
|
||||
#
|
||||
# 1. Always start Xvfb on :99 (same as the canonical bin/docker-entrypoint.sh)
|
||||
# so headed Chromium works no matter how the container is invoked.
|
||||
# 2. Detect whether the CMD looks like a Lambda handler (a single
|
||||
# `module.func`-shaped argument). If yes, route through the Lambda runtime
|
||||
# client (using the bundled aws-lambda-rie locally, or talking to the real
|
||||
# Lambda Runtime API when AWS_LAMBDA_RUNTIME_API is set in production).
|
||||
# 3. Otherwise exec the CMD directly — preserving the canonical Dockerfile's
|
||||
# interaction surface (`python`, `cloakserve`, `cloaktest`, `node`, `bash`,
|
||||
# `python examples/basic.py`, etc.).
|
||||
set -e
|
||||
|
||||
mkdir -p /tmp/.X11-unix
|
||||
chmod 1777 /tmp/.X11-unix 2>/dev/null || true
|
||||
|
||||
# Clean any stale Xvfb state. If a previous Xvfb died and left its lock file
|
||||
# behind (we observed this in cold-start storms), a new Xvfb refuses to start
|
||||
# with "Server is already active for display 99". Removing both files makes
|
||||
# Xvfb start cleanly every time.
|
||||
rm -f /tmp/.X99-lock /tmp/.X11-unix/X99
|
||||
|
||||
Xvfb :99 -screen 0 1920x1080x24 -nolisten tcp >/tmp/Xvfb.log 2>&1 &
|
||||
|
||||
# Wait for the X11 socket to appear AND for Xvfb to be ready to serve. The
|
||||
# socket file appears at bind(), but listen() and the first accept() come
|
||||
# slightly later — under cold-start CPU contention this gap matters.
|
||||
i=0
|
||||
while [ ! -e /tmp/.X11-unix/X99 ] && [ "$i" -lt 200 ]; do
|
||||
i=$((i + 1))
|
||||
sleep 0.05
|
||||
done
|
||||
# Small buffer after the socket appears so Xvfb has a moment to call listen()
|
||||
# and start accepting clients. Cheap insurance against the bind/listen gap.
|
||||
sleep 0.2
|
||||
|
||||
# Lambda handler shape: exactly one arg, dotted identifier (no spaces, no slashes,
|
||||
# no leading dot). `python`, `cloakserve`, `cloaktest`, `bash`, `node` all fail
|
||||
# this test and pass through to plain exec.
|
||||
if [ $# -eq 1 ] && \
|
||||
echo "$1" | grep -qE '^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)+$'; then
|
||||
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
|
||||
# Local invocation via bundled RIE.
|
||||
exec /usr/local/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric "$@"
|
||||
else
|
||||
# Real Lambda — runtime API endpoint already provided by the platform.
|
||||
exec /usr/local/bin/python -m awslambdaric "$@"
|
||||
fi
|
||||
fi
|
||||
|
||||
exec "$@"
|
||||
@@ -0,0 +1,336 @@
|
||||
"""AWS Lambda handler for one-off stealth-browser invocations.
|
||||
|
||||
Always runs **headed** via the Xvfb display started by `lambda-entrypoint.sh`.
|
||||
|
||||
Event schema (all fields except `url` are optional):
|
||||
|
||||
Launch options (passed to cloakbrowser.launch_context_async):
|
||||
url str required, the page to scrape
|
||||
proxy str|dict http://user:pass@host:port or Playwright proxy dict
|
||||
humanize bool False — enable human-like mouse/keyboard/scroll
|
||||
human_preset str "default" | "careful"
|
||||
geoip bool False — auto timezone+locale from proxy IP
|
||||
timezone str IANA tz, e.g. "America/New_York"
|
||||
locale str BCP-47, e.g. "en-US"
|
||||
viewport {width,height} defaults to 1920x947 (cloakbrowser DEFAULT_VIEWPORT)
|
||||
user_agent str custom UA (rare — cloakbrowser sets one already)
|
||||
extra_args list[str] additional Chromium CLI flags
|
||||
|
||||
Navigation options (passed to page.goto):
|
||||
wait_until str "load"|"domcontentloaded"|"networkidle"|"commit"
|
||||
default "domcontentloaded"
|
||||
goto_timeout_ms int 30000
|
||||
|
||||
Post-navigation waits (run in this order if specified):
|
||||
smart_wait bool ON by default if no other wait is set.
|
||||
Polls document.outerHTML.length and bails when it
|
||||
hasn't changed for `dom_stable_ms`. Handles lazy
|
||||
hydration, async chunks, and lazy images, and is
|
||||
immune to analytics beacons / long-poll that keep
|
||||
the network busy without mutating the DOM.
|
||||
dom_stable_ms int 1500 — how long DOM must be quiet
|
||||
max_settle_ms int 15000 — hard cap on smart_wait
|
||||
wait_for_load_state str "load"|"domcontentloaded"|"networkidle"
|
||||
wait_for_load_state_timeout_ms int 30000
|
||||
wait_for_selector str CSS or XPath selector
|
||||
wait_for_selector_state str "attached"|"detached"|"visible"|"hidden", default "visible"
|
||||
wait_for_selector_timeout_ms int 30000
|
||||
wait_for_function str JS expression that returns truthy when ready
|
||||
wait_for_function_timeout_ms int 30000
|
||||
wait_ms int fixed pause in ms (page.wait_for_timeout)
|
||||
|
||||
Capture options:
|
||||
screenshot bool True
|
||||
full_page_screenshot bool False — capture entire scrollable page
|
||||
|
||||
Retry orchestration:
|
||||
retries int default 1. Number of retry attempts after the first
|
||||
failure. Set to 0 to disable retries entirely (the
|
||||
handler will fail fast on the first error).
|
||||
Retried errors:
|
||||
ERR_CERT_* -> retry with --ignore-certificate-errors
|
||||
Timeout exceeded -> retry with goto_timeout_ms=90000, max_settle_ms=25000
|
||||
ERR_CONNECTION_TIMED_OUT -> same as Timeout
|
||||
Not retried (unrecoverable): ERR_NAME_NOT_RESOLVED,
|
||||
ERR_SSL_PROTOCOL_ERROR, generic ERR_CONNECTION_REFUSED.
|
||||
On final failure, the error message includes a
|
||||
retry_history block with strategy + error per attempt.
|
||||
|
||||
Returns:
|
||||
{"title": ..., "url": ..., "html": ..., "screenshot_b64"?: ...}
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from cloakbrowser import launch_context_async
|
||||
|
||||
logger = logging.getLogger("cloakbrowser.lambda")
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def _diag_snapshot() -> str:
|
||||
"""Capture Xvfb status, Xvfb log, X11 socket state, and env for error reports."""
|
||||
import os
|
||||
parts = []
|
||||
try:
|
||||
r = subprocess.run(["pgrep", "-fa", "Xvfb"], capture_output=True, text=True)
|
||||
parts.append(f"pgrep Xvfb: rc={r.returncode} stdout={r.stdout.strip()!r}")
|
||||
except Exception as e:
|
||||
parts.append(f"pgrep failed: {e}")
|
||||
try:
|
||||
r = subprocess.run(["ls", "-la", "/tmp/.X11-unix"], capture_output=True, text=True)
|
||||
parts.append(f"ls /tmp/.X11-unix:\n{r.stdout}{r.stderr}")
|
||||
except Exception as e:
|
||||
parts.append(f"ls /tmp/.X11-unix failed: {e}")
|
||||
try:
|
||||
log = Path("/tmp/Xvfb.log").read_text()
|
||||
parts.append(f"/tmp/Xvfb.log:\n{log}")
|
||||
except Exception as e:
|
||||
parts.append(f"Xvfb log unreadable: {e}")
|
||||
parts.append(f"env: DISPLAY={os.environ.get('DISPLAY')!r} HOME={os.environ.get('HOME')!r}")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def handler(event: dict, context: Any) -> dict:
|
||||
return asyncio.run(_run(event))
|
||||
|
||||
|
||||
def _build_launch_kwargs(event: dict) -> dict:
|
||||
"""Translate the event dict into kwargs for launch_context_async.
|
||||
|
||||
Only includes keys explicitly set in the event so cloakbrowser's defaults
|
||||
(DEFAULT_VIEWPORT etc.) kick in when fields are absent — passing
|
||||
viewport=None would *disable* viewport emulation, which we don't want.
|
||||
"""
|
||||
kwargs: dict = {
|
||||
"headless": False, # always headed via Xvfb
|
||||
"args": [
|
||||
# Lambda /dev/shm is ~64 MB — Chromium crashes mid-render without this.
|
||||
"--disable-dev-shm-usage",
|
||||
# Lambda's restricted process model can't fork from Chromium's zygote
|
||||
# — without this, child renderer processes fail to spawn.
|
||||
"--no-zygote",
|
||||
*event.get("extra_args", []),
|
||||
],
|
||||
}
|
||||
for key in ("proxy", "humanize", "human_preset", "geoip",
|
||||
"timezone", "locale", "viewport", "user_agent"):
|
||||
if key in event:
|
||||
kwargs[key] = event[key]
|
||||
return kwargs
|
||||
|
||||
|
||||
async def _smart_wait(page, dom_stable_ms: int = 1500, max_settle_ms: int = 15000) -> None:
|
||||
"""Wait until the document HTML hasn't changed for `dom_stable_ms`.
|
||||
|
||||
Generic stopping condition for at-scale scraping when you can't tune
|
||||
selectors per site. More robust than `networkidle` because it ignores
|
||||
network activity that doesn't mutate the DOM (analytics beacons,
|
||||
long-poll, websockets, web vitals streams).
|
||||
"""
|
||||
js = f"""
|
||||
(() => {{
|
||||
if (!window.__cb_settle) {{
|
||||
window.__cb_settle = {{ len: -1, since: Date.now() }};
|
||||
}}
|
||||
const cur = document.documentElement.outerHTML.length;
|
||||
const s = window.__cb_settle;
|
||||
if (cur !== s.len) {{
|
||||
s.len = cur;
|
||||
s.since = Date.now();
|
||||
return false;
|
||||
}}
|
||||
return (Date.now() - s.since) >= {int(dom_stable_ms)};
|
||||
}})()
|
||||
"""
|
||||
try:
|
||||
await page.wait_for_function(js, timeout=max_settle_ms, polling=200)
|
||||
except Exception:
|
||||
# Hit max_settle_ms cap — return what we have rather than fail the whole invoke
|
||||
logger.warning("smart_wait hit max_settle_ms=%d cap", max_settle_ms)
|
||||
|
||||
|
||||
_EXPLICIT_WAIT_KEYS = (
|
||||
"wait_for_load_state", "wait_for_selector", "wait_for_function", "wait_ms",
|
||||
)
|
||||
|
||||
|
||||
async def _post_nav_waits(page, event: dict) -> None:
|
||||
"""Run waits in priority order. smart_wait is the default unless the
|
||||
caller asked for a more specific stopping condition."""
|
||||
explicit = any(k in event for k in _EXPLICIT_WAIT_KEYS)
|
||||
if event.get("smart_wait", not explicit):
|
||||
await _smart_wait(
|
||||
page,
|
||||
dom_stable_ms=event.get("dom_stable_ms", 1500),
|
||||
max_settle_ms=event.get("max_settle_ms", 15000),
|
||||
)
|
||||
if "wait_for_load_state" in event:
|
||||
await page.wait_for_load_state(
|
||||
event["wait_for_load_state"],
|
||||
timeout=event.get("wait_for_load_state_timeout_ms", 30000),
|
||||
)
|
||||
if "wait_for_selector" in event:
|
||||
await page.wait_for_selector(
|
||||
event["wait_for_selector"],
|
||||
state=event.get("wait_for_selector_state", "visible"),
|
||||
timeout=event.get("wait_for_selector_timeout_ms", 30000),
|
||||
)
|
||||
if "wait_for_function" in event:
|
||||
await page.wait_for_function(
|
||||
event["wait_for_function"],
|
||||
timeout=event.get("wait_for_function_timeout_ms", 30000),
|
||||
)
|
||||
if "wait_ms" in event:
|
||||
await page.wait_for_timeout(event["wait_ms"])
|
||||
|
||||
|
||||
async def _launch_with_retry(event: dict, attempts: int = 3, backoff_s: float = 0.3):
|
||||
"""Retry launch_context_async up to `attempts` times with linear backoff.
|
||||
|
||||
Lambda cold-start storms occasionally race Xvfb readiness or hit transient
|
||||
Chromium spawn failures — both surface as "Target page, context or browser
|
||||
has been closed" at launch. The failure is fast (~0.5s) so retries are
|
||||
cheap, and a retry on a now-warm container almost always succeeds.
|
||||
|
||||
Pairs with the lock-cleanup + socket-poll in lambda-entrypoint.sh: the
|
||||
entrypoint catches the common case at container init; this catches the
|
||||
residual race when the first invocation hits before Xvfb is fully ready.
|
||||
"""
|
||||
last_err: Exception | None = None
|
||||
for i in range(attempts):
|
||||
try:
|
||||
return await launch_context_async(**_build_launch_kwargs(event))
|
||||
except Exception as e:
|
||||
last_err = e
|
||||
logger.warning("launch attempt %d/%d failed: %s",
|
||||
i + 1, attempts, str(e)[:200])
|
||||
if i + 1 < attempts:
|
||||
await asyncio.sleep(backoff_s * (i + 1)) # 0.3s, 0.6s
|
||||
raise last_err # type: ignore[misc]
|
||||
|
||||
|
||||
def _classify_error(err: Exception) -> dict | None:
|
||||
"""Map a Playwright error to a retry-strategy override dict, or None
|
||||
if the error is unrecoverable.
|
||||
|
||||
Match on str(e) because Playwright errors carry their codes inside the
|
||||
message (Error.__str__ includes ERR_CERT_AUTHORITY_INVALID etc.); there
|
||||
is no stable structured `.error_code` attribute to rely on.
|
||||
|
||||
Strategies (priority order — first match wins):
|
||||
ERR_CERT_* -> --ignore-certificate-errors + 60s goto budget
|
||||
Timeout exceeded -> 90s goto budget + 25s smart_wait cap
|
||||
ERR_CONNECTION_TIMED_OUT -> same as Timeout
|
||||
Returns None for unrecoverable site issues (DNS, SSL, refused, HTTP 4xx/5xx).
|
||||
"""
|
||||
msg = str(err)
|
||||
if "ERR_CERT" in msg:
|
||||
return {
|
||||
"extra_args": ["--ignore-certificate-errors"],
|
||||
"goto_timeout_ms": 60000,
|
||||
}
|
||||
if ("Timeout" in msg and "exceeded" in msg) or "ERR_CONNECTION_TIMED_OUT" in msg:
|
||||
return {
|
||||
"goto_timeout_ms": 90000,
|
||||
"max_settle_ms": 25000,
|
||||
}
|
||||
return None
|
||||
|
||||
|
||||
async def _attempt_scrape(url: str, event: dict) -> dict:
|
||||
"""One self-contained scrape attempt: launch, navigate, wait, capture, close.
|
||||
|
||||
Extracted from `_run` so the retry loop can call it repeatedly with an
|
||||
overridden event dict. Each attempt relaunches the browser — uniform
|
||||
behavior across strategies (the cert-bypass strategy *requires* a relaunch
|
||||
because `--ignore-certificate-errors` is a Chromium CLI arg, not a per-
|
||||
context switch), and the ~3-5s relaunch cost is fine on the slow path.
|
||||
"""
|
||||
ctx = await _launch_with_retry(event)
|
||||
try:
|
||||
page = await ctx.new_page()
|
||||
await page.goto(
|
||||
url,
|
||||
wait_until=event.get("wait_until", "domcontentloaded"),
|
||||
timeout=event.get("goto_timeout_ms", 30000),
|
||||
)
|
||||
|
||||
await _post_nav_waits(page, event)
|
||||
|
||||
result: dict = {
|
||||
"title": await page.title(),
|
||||
"url": page.url,
|
||||
"html": await page.content(),
|
||||
}
|
||||
|
||||
if event.get("screenshot", True):
|
||||
png = await page.screenshot(
|
||||
full_page=event.get("full_page_screenshot", False),
|
||||
)
|
||||
result["screenshot_b64"] = base64.b64encode(png).decode()
|
||||
|
||||
return result
|
||||
finally:
|
||||
try:
|
||||
await ctx.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _raise_with_history(err: Exception, history: list[dict]) -> None:
|
||||
"""Surface a final failure with a retry_history block embedded in the
|
||||
error message, so callers see what was tried before bailing."""
|
||||
diag = _diag_snapshot()
|
||||
if history:
|
||||
diag = "retry_history: " + json.dumps(history, default=str) + "\n\n" + diag
|
||||
logger.error("scrape failed (after %d retries): %s\nDIAG:\n%s",
|
||||
len(history), err, diag)
|
||||
raise RuntimeError(f"scrape failed: {err}\n--- DIAG ---\n{diag}") from err
|
||||
|
||||
|
||||
async def _run(event: dict) -> dict:
|
||||
"""Top-level scrape with strategy-based retry orchestration.
|
||||
|
||||
First attempt uses the event verbatim. If it fails with a classifiable
|
||||
error (cert / timeout), retry with that strategy's overrides merged into
|
||||
the event. `retries` bounds the number of strategy retries (default 1;
|
||||
set to 0 to disable retry entirely).
|
||||
"""
|
||||
url = event["url"]
|
||||
retries_left = max(0, int(event.get("retries", 1)))
|
||||
history: list[dict] = []
|
||||
current_event = event
|
||||
|
||||
while True:
|
||||
try:
|
||||
return await _attempt_scrape(url, current_event)
|
||||
except Exception as e:
|
||||
if retries_left <= 0:
|
||||
_raise_with_history(e, history)
|
||||
strategy = _classify_error(e)
|
||||
if strategy is None:
|
||||
_raise_with_history(e, history)
|
||||
history.append({
|
||||
"attempt": len(history) + 1,
|
||||
"error": str(e)[:300],
|
||||
"strategy": strategy,
|
||||
})
|
||||
logger.warning("attempt %d failed (%s); retrying with strategy=%s",
|
||||
len(history), str(e)[:120], strategy)
|
||||
merged_args = list(current_event.get("extra_args", [])) + list(strategy.get("extra_args", []))
|
||||
current_event = {**current_event, **strategy, "extra_args": merged_args}
|
||||
retries_left -= 1
|
||||
# No backoff: strategy overrides change goto budget directly;
|
||||
# the prior failure was either fast (cert reject) or already
|
||||
# waited its full timeout. Container is warm.
|
||||
|
||||
|
||||
+6
-3
@@ -63,10 +63,13 @@ await browser.close();
|
||||
```javascript
|
||||
import { launch, launchContext, launchPersistentContext } from 'cloakbrowser';
|
||||
|
||||
// With proxy
|
||||
// With proxy (HTTP or SOCKS5)
|
||||
const browser = await launch({
|
||||
proxy: 'http://user:pass@proxy:8080',
|
||||
});
|
||||
const browser = await launch({
|
||||
proxy: 'socks5://user:pass@proxy:1080',
|
||||
});
|
||||
|
||||
// With proxy object (bypass, separate auth fields)
|
||||
const browser = await launch({
|
||||
@@ -211,8 +214,8 @@ const page = await browser.newPage();
|
||||
|
||||
## Requirements
|
||||
|
||||
- Node.js >= 18
|
||||
- One of: `playwright-core` >= 1.40 or `puppeteer-core` >= 21
|
||||
- Node.js >= 20
|
||||
- One of: `playwright-core` >= 1.53 or `puppeteer-core` >= 21
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
Generated
+57
-9
@@ -1,31 +1,36 @@
|
||||
{
|
||||
"name": "cloakbrowser",
|
||||
"version": "0.3.9",
|
||||
"version": "0.3.23",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "cloakbrowser",
|
||||
"version": "0.3.9",
|
||||
"version": "0.3.23",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"tar": "^7.0.0"
|
||||
},
|
||||
"bin": {
|
||||
"cloakbrowser": "dist/cli.js"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.10.0",
|
||||
"mmdb-lib": "^3.0.2",
|
||||
"playwright-core": "^1.40.0",
|
||||
"puppeteer-core": "^21.0.0",
|
||||
"socks-proxy-agent": "^10.0.0",
|
||||
"typescript": "^5.3.0",
|
||||
"vitest": "^1.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
"node": ">=20.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"mmdb-lib": ">=2.0.0",
|
||||
"playwright-core": ">=1.40.0",
|
||||
"puppeteer-core": ">=21.0.0"
|
||||
"puppeteer-core": ">=21.0.0",
|
||||
"socks-proxy-agent": ">=8.0.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"mmdb-lib": {
|
||||
@@ -36,6 +41,9 @@
|
||||
},
|
||||
"puppeteer-core": {
|
||||
"optional": true
|
||||
},
|
||||
"socks-proxy-agent": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -2003,6 +2011,21 @@
|
||||
"node": ">= 14"
|
||||
}
|
||||
},
|
||||
"node_modules/pac-proxy-agent/node_modules/socks-proxy-agent": {
|
||||
"version": "8.0.5",
|
||||
"resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz",
|
||||
"integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"agent-base": "^7.1.2",
|
||||
"debug": "^4.3.4",
|
||||
"socks": "^2.8.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 14"
|
||||
}
|
||||
},
|
||||
"node_modules/pac-resolver": {
|
||||
"version": "7.0.1",
|
||||
"resolved": "https://registry.npmjs.org/pac-resolver/-/pac-resolver-7.0.1.tgz",
|
||||
@@ -2164,6 +2187,21 @@
|
||||
"node": ">= 14"
|
||||
}
|
||||
},
|
||||
"node_modules/proxy-agent/node_modules/socks-proxy-agent": {
|
||||
"version": "8.0.5",
|
||||
"resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz",
|
||||
"integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"agent-base": "^7.1.2",
|
||||
"debug": "^4.3.4",
|
||||
"socks": "^2.8.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 14"
|
||||
}
|
||||
},
|
||||
"node_modules/proxy-from-env": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
|
||||
@@ -2332,18 +2370,28 @@
|
||||
}
|
||||
},
|
||||
"node_modules/socks-proxy-agent": {
|
||||
"version": "8.0.5",
|
||||
"resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz",
|
||||
"integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==",
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-10.0.0.tgz",
|
||||
"integrity": "sha512-pyp2YR3mNxAMu0mGLtzs4g7O3uT4/9sQOLAKcViAkaS9fJWkud7nmaf6ZREFqQEi24IPkBcjfHjXhPTUWjo3uA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"agent-base": "^7.1.2",
|
||||
"agent-base": "9.0.0",
|
||||
"debug": "^4.3.4",
|
||||
"socks": "^2.8.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 14"
|
||||
"node": ">= 20"
|
||||
}
|
||||
},
|
||||
"node_modules/socks-proxy-agent/node_modules/agent-base": {
|
||||
"version": "9.0.0",
|
||||
"resolved": "https://registry.npmjs.org/agent-base/-/agent-base-9.0.0.tgz",
|
||||
"integrity": "sha512-TQf59BsZnytt8GdJKLPfUZ54g/iaUL2OWDSFCCvMOhsHduDQxO8xC4PNeyIkVcA5KwL2phPSv0douC0fgWzmnA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 20"
|
||||
}
|
||||
},
|
||||
"node_modules/source-map": {
|
||||
|
||||
+10
-5
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "cloakbrowser",
|
||||
"version": "0.3.23",
|
||||
"version": "0.3.28",
|
||||
"description": "Stealth Chromium that passes every bot detection test. Drop-in Playwright/Puppeteer replacement with source-level fingerprint patches.",
|
||||
"type": "module",
|
||||
"main": "dist/index.js",
|
||||
@@ -55,12 +55,13 @@
|
||||
},
|
||||
"homepage": "https://github.com/CloakHQ/cloakbrowser#javascript--nodejs",
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
"node": ">=20.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"mmdb-lib": ">=2.0.0",
|
||||
"playwright-core": ">=1.40.0",
|
||||
"puppeteer-core": ">=21.0.0"
|
||||
"playwright-core": ">=1.53.0",
|
||||
"puppeteer-core": ">=21.0.0",
|
||||
"socks-proxy-agent": ">=10.0.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"playwright-core": {
|
||||
@@ -71,6 +72,9 @@
|
||||
},
|
||||
"mmdb-lib": {
|
||||
"optional": true
|
||||
},
|
||||
"socks-proxy-agent": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"dependencies": {
|
||||
@@ -79,7 +83,8 @@
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.10.0",
|
||||
"mmdb-lib": "^3.0.2",
|
||||
"playwright-core": "^1.40.0",
|
||||
"socks-proxy-agent": "^10.0.0",
|
||||
"playwright-core": "^1.53.0",
|
||||
"puppeteer-core": "^21.0.0",
|
||||
"typescript": "^5.3.0",
|
||||
"vitest": "^1.0.0"
|
||||
|
||||
+4
-4
@@ -27,14 +27,14 @@ export { WRAPPER_VERSION };
|
||||
// CHROMIUM_VERSION is the latest across all platforms (for display/reference).
|
||||
// Use getChromiumVersion() for the current platform's actual version.
|
||||
// ---------------------------------------------------------------------------
|
||||
export const CHROMIUM_VERSION = "146.0.7680.177.1";
|
||||
export const CHROMIUM_VERSION = "146.0.7680.177.3";
|
||||
|
||||
export const PLATFORM_CHROMIUM_VERSIONS: Record<string, string> = {
|
||||
"linux-x64": "146.0.7680.177.1",
|
||||
"linux-arm64": "145.0.7632.159.7",
|
||||
"linux-x64": "146.0.7680.177.3",
|
||||
"linux-arm64": "146.0.7680.177.3",
|
||||
"darwin-arm64": "145.0.7632.109.2",
|
||||
"darwin-x64": "145.0.7632.109.2",
|
||||
"windows-x64": "145.0.7632.159.7",
|
||||
"windows-x64": "146.0.7680.177.4",
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
+112
-17
@@ -15,13 +15,14 @@ import dns from "node:dns/promises";
|
||||
import net from "node:net";
|
||||
import { getCacheDir } from "./config.js";
|
||||
import type { LaunchOptions } from "./types.js";
|
||||
import { ensureProxyScheme } from "./proxy.js";
|
||||
import { ensureProxyScheme, isSocksProxy, reconstructSocksUrl, type ProxyDict } from "./proxy.js";
|
||||
|
||||
// P3TERX mirror of MaxMind GeoLite2-City — no license key needed
|
||||
const GEOIP_DB_URL =
|
||||
"https://github.com/P3TERX/GeoLite.mmdb/raw/download/GeoLite2-City.mmdb";
|
||||
const GEOIP_DB_FILENAME = "GeoLite2-City.mmdb";
|
||||
const GEOIP_UPDATE_INTERVAL_MS = 30 * 86_400_000; // 30 days
|
||||
const DEFAULT_GEOIP_TIMEOUT_MS = 5_000;
|
||||
|
||||
/** Country ISO code → BCP 47 locale (covers ~90% of proxy traffic). */
|
||||
export const COUNTRY_LOCALE_MAP: Record<string, string> = {
|
||||
@@ -68,10 +69,18 @@ export async function resolveProxyGeo(
|
||||
const dbPath = await ensureGeoipDb();
|
||||
if (!dbPath) return { timezone: null, locale: null, exitIp: null };
|
||||
|
||||
const timeoutMs = getGeoipTimeoutMs();
|
||||
const deadline = deadlineFromTimeout(timeoutMs);
|
||||
|
||||
// Exit IP (through proxy) is most accurate — gateway DNS may differ from exit
|
||||
let ip = await resolveExitIp(proxyUrl);
|
||||
if (!ip) ip = await resolveProxyIp(proxyUrl);
|
||||
if (!ip) return { timezone: null, locale: null, exitIp: null };
|
||||
let ip = await resolveExitIp(proxyUrl, remainingMs(deadline));
|
||||
if (!ip && !deadlineExpired(deadline)) ip = await resolveProxyIp(proxyUrl);
|
||||
if (!ip || deadlineExpired(deadline)) {
|
||||
if (deadlineExpired(deadline)) {
|
||||
console.warn(`[cloakbrowser] GeoIP resolution timed out after ${timeoutMs}ms; continuing without GeoIP`);
|
||||
}
|
||||
return { timezone: null, locale: null, exitIp: null };
|
||||
}
|
||||
|
||||
try {
|
||||
const buf = fs.readFileSync(dbPath);
|
||||
@@ -87,6 +96,30 @@ export async function resolveProxyGeo(
|
||||
}
|
||||
}
|
||||
|
||||
function getGeoipTimeoutMs(): number {
|
||||
const raw = process.env.CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS;
|
||||
if (!raw) return DEFAULT_GEOIP_TIMEOUT_MS;
|
||||
const timeoutSeconds = Number(raw);
|
||||
if (!Number.isFinite(timeoutSeconds)) {
|
||||
console.warn(`[cloakbrowser] Invalid CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS=${raw}; using ${DEFAULT_GEOIP_TIMEOUT_MS / 1000}s`);
|
||||
return DEFAULT_GEOIP_TIMEOUT_MS;
|
||||
}
|
||||
return Math.max(timeoutSeconds, 0) * 1000;
|
||||
}
|
||||
|
||||
function deadlineFromTimeout(timeoutMs: number): number | null {
|
||||
return timeoutMs > 0 ? performance.now() + timeoutMs : null;
|
||||
}
|
||||
|
||||
function remainingMs(deadline: number | null): number | undefined {
|
||||
if (deadline === null) return undefined;
|
||||
return Math.max(deadline - performance.now(), 0);
|
||||
}
|
||||
|
||||
function deadlineExpired(deadline: number | null): boolean {
|
||||
return deadline !== null && performance.now() >= deadline;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Proxy IP resolution
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -128,16 +161,56 @@ const IP_ECHO_URLS = [
|
||||
"https://ifconfig.me/ip",
|
||||
];
|
||||
|
||||
async function resolveExitIp(proxyUrl: string): Promise<string | null> {
|
||||
// Node.js fetch doesn't support proxy natively — use a CONNECT tunnel via http
|
||||
// For simplicity, use a direct HTTP request to a plain-text IP echo service
|
||||
// through the proxy using Node's http module
|
||||
async function resolveExitIp(proxyUrl: string, timeoutMs?: number): Promise<string | null> {
|
||||
const deadline = timeoutMs && timeoutMs > 0 ? performance.now() + timeoutMs : null;
|
||||
const isSocks = isSocksProxy(proxyUrl);
|
||||
|
||||
// SOCKS5: tunnel through the SOCKS5 proxy via socks-proxy-agent
|
||||
if (isSocks) {
|
||||
let SocksProxyAgent: typeof import("socks-proxy-agent").SocksProxyAgent;
|
||||
try {
|
||||
({ SocksProxyAgent } = await import("socks-proxy-agent"));
|
||||
} catch {
|
||||
console.warn("[cloakbrowser] socks-proxy-agent not installed — cannot resolve exit IP through SOCKS5 proxy. Install it: npm install socks-proxy-agent");
|
||||
return null;
|
||||
}
|
||||
const { default: https } = await import("node:https");
|
||||
const agent = new SocksProxyAgent(proxyUrl);
|
||||
|
||||
for (const echoUrl of IP_ECHO_URLS) {
|
||||
const remaining = remainingMs(deadline);
|
||||
if (remaining !== undefined && remaining <= 0) return null;
|
||||
try {
|
||||
const ip = await new Promise<string | null>((resolve) => {
|
||||
const req = https.request(echoUrl, { agent, timeout: Math.min(10_000, remaining ?? 10_000) }, (res) => {
|
||||
let data = "";
|
||||
res.on("data", (chunk: Buffer) => (data += chunk.toString()));
|
||||
res.on("end", () => {
|
||||
const ip = data.trim();
|
||||
resolve(net.isIP(ip) ? ip : null);
|
||||
});
|
||||
});
|
||||
req.on("error", () => resolve(null));
|
||||
req.on("timeout", () => { req.destroy(); resolve(null); });
|
||||
req.end();
|
||||
});
|
||||
if (ip) return ip;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// HTTP/HTTPS: use a CONNECT tunnel via http
|
||||
try {
|
||||
const { default: http } = await import("node:http");
|
||||
const { default: https } = await import("node:https");
|
||||
const proxyUrlObj = new URL(proxyUrl);
|
||||
|
||||
for (const echoUrl of IP_ECHO_URLS) {
|
||||
const remaining = remainingMs(deadline);
|
||||
if (remaining !== undefined && remaining <= 0) return null;
|
||||
try {
|
||||
const ip = await new Promise<string | null>((resolve, reject) => {
|
||||
const targetUrl = new URL(echoUrl);
|
||||
@@ -155,13 +228,14 @@ async function resolveExitIp(proxyUrl: string): Promise<string | null> {
|
||||
).toString("base64"),
|
||||
}
|
||||
: {},
|
||||
timeout: 10_000,
|
||||
timeout: Math.min(10_000, remaining ?? 10_000),
|
||||
});
|
||||
|
||||
connectReq.on("connect", (_res, socket) => {
|
||||
const innerRemaining = remainingMs(deadline);
|
||||
const req = https.request(
|
||||
echoUrl,
|
||||
{ socket, timeout: 5_000 } as any,
|
||||
{ socket, timeout: Math.min(5_000, innerRemaining ?? 5_000) } as any,
|
||||
(res) => {
|
||||
let data = "";
|
||||
res.on("data", (chunk: Buffer) => (data += chunk.toString()));
|
||||
@@ -172,6 +246,7 @@ async function resolveExitIp(proxyUrl: string): Promise<string | null> {
|
||||
}
|
||||
);
|
||||
req.on("error", () => resolve(null));
|
||||
req.on("timeout", () => { req.destroy(); resolve(null); });
|
||||
req.end();
|
||||
});
|
||||
|
||||
@@ -226,7 +301,9 @@ async function downloadGeoipDb(dest: string): Promise<void> {
|
||||
|
||||
const tmpPath = `${dest}.tmp.${Date.now()}`;
|
||||
try {
|
||||
const response = await fetch(GEOIP_DB_URL, { redirect: "follow" });
|
||||
const response = await fetch(GEOIP_DB_URL, {
|
||||
redirect: "follow",
|
||||
});
|
||||
if (!response.ok || !response.body) {
|
||||
throw new Error(`HTTP ${response.status}`);
|
||||
}
|
||||
@@ -264,6 +341,22 @@ function maybeTriggerUpdate(dbPath: string): void {
|
||||
downloadGeoipDb(dbPath).catch(() => {});
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract a usable proxy URL from LaunchOptions.proxy.
|
||||
* For SOCKS5 dicts with separate credentials, reconstructs the full URL
|
||||
* with inline credentials so SOCKS5 auth works.
|
||||
*/
|
||||
function extractProxyUrl(proxy: string | ProxyDict | undefined): string | null {
|
||||
if (!proxy) return null;
|
||||
if (typeof proxy === "string") return ensureProxyScheme(proxy);
|
||||
const p = proxy as ProxyDict;
|
||||
if (!p.server) return null;
|
||||
if (p.username && isSocksProxy(p)) {
|
||||
return reconstructSocksUrl(p);
|
||||
}
|
||||
return ensureProxyScheme(p.server);
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-fill timezone/locale from proxy IP when geoip is enabled.
|
||||
* Also returns exitIp as a free bonus (reused for WebRTC spoofing).
|
||||
@@ -273,13 +366,13 @@ export async function maybeResolveGeoip(
|
||||
): Promise<{ timezone?: string; locale?: string; exitIp?: string }> {
|
||||
if (!options.geoip || !options.proxy) return { timezone: options.timezone, locale: options.locale };
|
||||
|
||||
let proxyUrl = typeof options.proxy === "string" ? options.proxy : options.proxy.server;
|
||||
const proxyUrl = extractProxyUrl(options.proxy);
|
||||
if (!proxyUrl) return { timezone: options.timezone, locale: options.locale };
|
||||
proxyUrl = ensureProxyScheme(proxyUrl);
|
||||
|
||||
// When both tz/locale are explicit, still resolve exit IP for WebRTC
|
||||
if (options.timezone && options.locale) {
|
||||
const exitIp = await resolveExitIp(proxyUrl) ?? undefined;
|
||||
const timeoutMs = getGeoipTimeoutMs();
|
||||
const exitIp = await resolveExitIp(proxyUrl, timeoutMs) ?? undefined;
|
||||
return { timezone: options.timezone, locale: options.locale, exitIp };
|
||||
}
|
||||
|
||||
@@ -304,24 +397,26 @@ export async function resolveWebrtcArgs(
|
||||
const idx = args.findIndex(a => a === "--fingerprint-webrtc-ip=auto");
|
||||
if (idx === -1) return args;
|
||||
|
||||
let proxyUrl = typeof options.proxy === "string" ? options.proxy : options.proxy?.server;
|
||||
const proxyUrl = extractProxyUrl(options.proxy);
|
||||
if (!proxyUrl) {
|
||||
console.warn("[cloakbrowser] --fingerprint-webrtc-ip=auto requires a proxy; removing flag");
|
||||
const result = [...args];
|
||||
result.splice(idx, 1);
|
||||
return result;
|
||||
}
|
||||
proxyUrl = ensureProxyScheme(proxyUrl);
|
||||
|
||||
try {
|
||||
const ip = await resolveExitIp(proxyUrl);
|
||||
const ip = await resolveExitIp(proxyUrl, getGeoipTimeoutMs());
|
||||
const result = [...args];
|
||||
if (ip) {
|
||||
result[idx] = `--fingerprint-webrtc-ip=${ip}`;
|
||||
} else {
|
||||
console.warn("[cloakbrowser] Could not resolve proxy exit IP for WebRTC spoofing; removing --fingerprint-webrtc-ip=auto");
|
||||
result.splice(idx, 1);
|
||||
}
|
||||
return result;
|
||||
} catch {
|
||||
console.warn("[cloakbrowser] Failed to resolve proxy exit IP for WebRTC spoofing; removing --fingerprint-webrtc-ip=auto");
|
||||
const result = [...args];
|
||||
result.splice(idx, 1);
|
||||
return result;
|
||||
|
||||
+141
-55
@@ -47,17 +47,17 @@
|
||||
*/
|
||||
|
||||
import type { Browser, Page, Frame, CDPSession, ElementHandle, BrowserContext } from 'puppeteer-core';
|
||||
import type { HumanConfig } from '../human/config.js';
|
||||
import { resolveConfig, rand, randRange, sleep } from '../human/config.js';
|
||||
import type { HumanConfig, HumanActionOptions } from '../human/config.js';
|
||||
import { resolveConfig, mergeConfig, rand, randRange, sleep } from '../human/config.js';
|
||||
import { RawMouse, RawKeyboard, humanMove, humanClick, clickTarget, humanIdle } from '../human/mouse.js';
|
||||
import { humanType } from './keyboard.js';
|
||||
import { scrollToElement, smoothWheel } from './scroll.js';
|
||||
import { scrollToElement, humanScrollIntoView, smoothWheel } from './scroll.js';
|
||||
|
||||
export type { HumanConfig } from '../human/config.js';
|
||||
export { resolveConfig } from '../human/config.js';
|
||||
export { resolveConfig, mergeConfig } from '../human/config.js';
|
||||
export { humanMove, humanClick, clickTarget, humanIdle } from '../human/mouse.js';
|
||||
export { humanType } from './keyboard.js';
|
||||
export { scrollToElement } from './scroll.js';
|
||||
export { scrollToElement, humanScrollIntoView } from './scroll.js';
|
||||
|
||||
|
||||
// ============================================================================
|
||||
@@ -319,7 +319,11 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
}
|
||||
|
||||
// ==== goto ====
|
||||
const humanGoto = async (url: string, options?: any) => {
|
||||
const humanGoto = async (url: string, options?: {
|
||||
referer?: string;
|
||||
timeout?: number;
|
||||
waitUntil?: 'load' | 'domcontentloaded' | 'networkidle0' | 'networkidle2';
|
||||
}) => {
|
||||
const response = await originals.goto(url, options);
|
||||
stealth.invalidate();
|
||||
patchFrames(page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
@@ -327,54 +331,64 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
};
|
||||
|
||||
// ==== click (with clickCount support for dblclick) ====
|
||||
const humanClickFn = async (selector: string, options?: any) => {
|
||||
const humanClickFn = async (selector: string, options?: HumanActionOptions & {
|
||||
button?: 'left' | 'right' | 'middle' | 'back' | 'forward';
|
||||
clickCount?: number;
|
||||
count?: number;
|
||||
delay?: number;
|
||||
}) => {
|
||||
await ensureCursorInit();
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, cfg);
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, callCfg, options?.timeout);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
const isInput = await isInputElement(stealth, page, selector);
|
||||
const target = clickTarget(box, isInput, cfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, cfg);
|
||||
const target = clickTarget(box, isInput, callCfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
|
||||
const clickCount = options?.clickCount ?? options?.count ?? 1;
|
||||
if (clickCount >= 2) {
|
||||
await humanClick(raw, isInput, cfg);
|
||||
await humanClick(raw, isInput, callCfg);
|
||||
await sleep(rand(40, 90));
|
||||
await raw.down({ clickCount: 2 });
|
||||
await sleep(rand(30, 60));
|
||||
await raw.up({ clickCount: 2 });
|
||||
} else {
|
||||
await humanClick(raw, isInput, cfg);
|
||||
await humanClick(raw, isInput, callCfg);
|
||||
}
|
||||
};
|
||||
|
||||
// ==== hover ====
|
||||
const humanHoverFn = async (selector: string, options?: any) => {
|
||||
const humanHoverFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
await ensureCursorInit();
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, cfg);
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, callCfg, options?.timeout);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
const target = clickTarget(box, false, cfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, cfg);
|
||||
const target = clickTarget(box, false, callCfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
};
|
||||
|
||||
// ==== type ====
|
||||
const humanTypeFn = async (selector: string, text: string, options?: any) => {
|
||||
await sleep(randRange(cfg.field_switch_delay));
|
||||
await humanClickFn(selector);
|
||||
const humanTypeFn = async (selector: string, text: string, options?: HumanActionOptions & {
|
||||
delay?: number;
|
||||
}) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
await sleep(randRange(callCfg.field_switch_delay));
|
||||
await humanClickFn(selector, options);
|
||||
await sleep(rand(100, 250));
|
||||
const cdp = await ensureCdp();
|
||||
await humanType(page, rawKb, text, cfg, cdp);
|
||||
await humanType(page, rawKb, text, callCfg, cdp);
|
||||
};
|
||||
|
||||
// ==== select ====
|
||||
@@ -392,7 +406,7 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
};
|
||||
|
||||
// ==== tap ====
|
||||
const humanTapFn = async (selector: string, options?: any) => {
|
||||
const humanTapFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
await humanClickFn(selector, options);
|
||||
};
|
||||
|
||||
@@ -410,14 +424,19 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
// ============================================================
|
||||
// Mouse patches
|
||||
// ============================================================
|
||||
page.mouse.move = async (x: number, y: number, options?: any) => {
|
||||
page.mouse.move = async (x: number, y: number, options?: { steps?: number }) => {
|
||||
await ensureCursorInit();
|
||||
await humanMove(raw, cursor.x, cursor.y, x, y, cfg);
|
||||
cursor.x = x;
|
||||
cursor.y = y;
|
||||
};
|
||||
|
||||
page.mouse.click = async (x: number, y: number, options?: any) => {
|
||||
page.mouse.click = async (x: number, y: number, options?: {
|
||||
button?: 'left' | 'right' | 'middle' | 'back' | 'forward';
|
||||
clickCount?: number;
|
||||
count?: number;
|
||||
delay?: number;
|
||||
}) => {
|
||||
await ensureCursorInit();
|
||||
await humanMove(raw, cursor.x, cursor.y, x, y, cfg);
|
||||
cursor.x = x;
|
||||
@@ -452,7 +471,7 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
(page.mouse as any).dragAndDrop = async (
|
||||
start: { x: number; y: number },
|
||||
target: { x: number; y: number },
|
||||
options?: any,
|
||||
options?: { delay?: number },
|
||||
) => {
|
||||
await ensureCursorInit();
|
||||
await humanMove(raw, cursor.x, cursor.y, start.x, start.y, cfg);
|
||||
@@ -472,12 +491,12 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
// ============================================================
|
||||
// Keyboard patches
|
||||
// ============================================================
|
||||
page.keyboard.type = async (text: string, options?: any) => {
|
||||
page.keyboard.type = async (text: string, options?: { delay?: number }) => {
|
||||
const cdp = await ensureCdp();
|
||||
await humanType(page, rawKb, text, cfg, cdp);
|
||||
};
|
||||
|
||||
page.keyboard.press = async (key: any, options?: any) => {
|
||||
page.keyboard.press = async (key: any, options?: { delay?: number }) => {
|
||||
await sleep(rand(20, 60));
|
||||
await originals.keyboardDown(key as any);
|
||||
await sleep(randRange(cfg.key_hold));
|
||||
@@ -548,7 +567,11 @@ function patchElementHandle(
|
||||
return els;
|
||||
};
|
||||
|
||||
(page as any).waitForSelector = async (selector: string, options?: any) => {
|
||||
(page as any).waitForSelector = async (selector: string, options?: {
|
||||
hidden?: boolean;
|
||||
timeout?: number;
|
||||
visible?: boolean;
|
||||
}) => {
|
||||
const el = await origWaitForSelector(selector, options);
|
||||
if (el) patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return el;
|
||||
@@ -577,6 +600,9 @@ function patchSingleElementHandle(
|
||||
const origElDragAndDrop = (el as any).dragAndDrop?.bind(el);
|
||||
const origElSelect = (el as any).select?.bind(el);
|
||||
const origElDrop = (el as any).drop?.bind(el);
|
||||
// Puppeteer v22+ adds ElementHandle.scrollIntoView(); earlier versions
|
||||
// expose it implicitly via evaluate(node => node.scrollIntoView()).
|
||||
const origElScrollIntoView = (el as any).scrollIntoView?.bind(el);
|
||||
|
||||
// --- Nested selectors ---
|
||||
const origEl$ = el.$.bind(el);
|
||||
@@ -597,45 +623,69 @@ function patchSingleElementHandle(
|
||||
return children;
|
||||
};
|
||||
|
||||
(el as any).waitForSelector = async (selector: string, options?: any) => {
|
||||
(el as any).waitForSelector = async (selector: string, options?: {
|
||||
hidden?: boolean;
|
||||
timeout?: number;
|
||||
visible?: boolean;
|
||||
}) => {
|
||||
const child = await origElWaitForSelector(selector, options);
|
||||
if (child) patchSingleElementHandle(child, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return child;
|
||||
};
|
||||
|
||||
// --- Helper: get box and move cursor ---
|
||||
const moveToElement = async () => {
|
||||
// --- Helper: get box and move cursor. Accepts a per-call ``callCfg``
|
||||
// so type/fill overrides like ``el.type(text, { typing_delay: 30 })``
|
||||
// carry through to mouse timing for that single call. Also scrolls into
|
||||
// view first so off-screen elements work (#129, #172 follow-up).
|
||||
const moveToElement = async (callCfg: HumanConfig = cfg) => {
|
||||
await (page as any)._ensureCursorInit();
|
||||
|
||||
try {
|
||||
const { cursorX, cursorY } = await humanScrollIntoView(
|
||||
page, raw,
|
||||
() => el.boundingBox().then(b => b ?? null),
|
||||
cursor.x, cursor.y, callCfg,
|
||||
);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
} catch { /* let boundingBox() decide */ }
|
||||
|
||||
const box = await el.boundingBox();
|
||||
if (!box) return null;
|
||||
|
||||
const isInp = await isInputElementHandle(stealth, el);
|
||||
const target = clickTarget(box, isInp, cfg);
|
||||
const target = clickTarget(box, isInp, callCfg);
|
||||
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, cfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
return { box, isInp };
|
||||
};
|
||||
|
||||
// --- el.click() ---
|
||||
(el as any).click = async (options?: any) => {
|
||||
const info = await moveToElement();
|
||||
(el as any).click = async (options?: HumanActionOptions & {
|
||||
button?: 'left' | 'right' | 'middle' | 'back' | 'forward';
|
||||
clickCount?: number;
|
||||
count?: number;
|
||||
delay?: number;
|
||||
}) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const info = await moveToElement(callCfg);
|
||||
if (!info) return origElClick(options);
|
||||
|
||||
const clickCount = options?.clickCount ?? options?.count ?? 1;
|
||||
if (clickCount >= 2) {
|
||||
await humanClick(raw, info.isInp, cfg);
|
||||
await humanClick(raw, info.isInp, callCfg);
|
||||
await sleep(rand(40, 90));
|
||||
await raw.down({ clickCount: 2 });
|
||||
await sleep(rand(30, 60));
|
||||
await raw.up({ clickCount: 2 });
|
||||
} else {
|
||||
await humanClick(raw, info.isInp, cfg);
|
||||
await humanClick(raw, info.isInp, callCfg);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -646,18 +696,43 @@ function patchSingleElementHandle(
|
||||
};
|
||||
|
||||
// --- el.type() ---
|
||||
(el as any).type = async (text: string, options?: any) => {
|
||||
const info = await moveToElement();
|
||||
(el as any).type = async (text: string, options?: HumanActionOptions & { delay?: number }) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const info = await moveToElement(callCfg);
|
||||
if (!info) return origElType(text, options);
|
||||
await humanClick(raw, info.isInp, cfg);
|
||||
await humanClick(raw, info.isInp, callCfg);
|
||||
await sleep(rand(100, 250));
|
||||
const cdp = await stealth.getCdpSession().catch(() => null);
|
||||
await humanType(page, rawKb, text, cfg, cdp);
|
||||
await humanType(page, rawKb, text, callCfg, cdp);
|
||||
};
|
||||
|
||||
// --- el.scrollIntoView() ---
|
||||
// Puppeteer-only equivalent of Playwright's scrollIntoViewIfNeeded.
|
||||
// Replaces the native snap-scroll (a strong bot signal) with the same
|
||||
// accelerate → cruise → decelerate → overshoot wheel sequence used by
|
||||
// page.click(). Only patched when the underlying ElementHandle exposes
|
||||
// ``scrollIntoView`` (Puppeteer v22+).
|
||||
if (origElScrollIntoView) {
|
||||
(el as any).scrollIntoView = async (options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
await (page as any)._ensureCursorInit();
|
||||
try {
|
||||
const { cursorX, cursorY } = await humanScrollIntoView(
|
||||
page, raw,
|
||||
() => el.boundingBox().then(b => b ?? null),
|
||||
cursor.x, cursor.y, callCfg,
|
||||
);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
} catch {
|
||||
return origElScrollIntoView(options);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// --- el.press() ---
|
||||
if (origElPress) {
|
||||
(el as any).press = async (key: string, options?: any) => {
|
||||
(el as any).press = async (key: string, options?: { delay?: number }) => {
|
||||
await sleep(rand(20, 60));
|
||||
await originals.keyboardDown(key as any);
|
||||
await sleep(randRange(cfg.key_hold));
|
||||
@@ -696,7 +771,7 @@ function patchSingleElementHandle(
|
||||
|
||||
// --- el.drop() ---
|
||||
if (origElDrop) {
|
||||
(el as any).drop = async (draggable: ElementHandle, options?: any) => {
|
||||
(el as any).drop = async (draggable: ElementHandle, options?: { delay?: number }) => {
|
||||
const srcBox = await draggable.boundingBox();
|
||||
const tgtBox = await el.boundingBox();
|
||||
|
||||
@@ -726,7 +801,7 @@ function patchSingleElementHandle(
|
||||
|
||||
// --- el.dragAndDrop() ---
|
||||
if (origElDragAndDrop) {
|
||||
(el as any).dragAndDrop = async (targetEl: ElementHandle, options?: any) => {
|
||||
(el as any).dragAndDrop = async (targetEl: ElementHandle, options?: { delay?: number }) => {
|
||||
const srcBox = await el.boundingBox();
|
||||
const tgtBox = await targetEl.boundingBox();
|
||||
|
||||
@@ -790,15 +865,22 @@ function patchSingleFrame(
|
||||
|
||||
const origFrameSelect = frame.select.bind(frame);
|
||||
|
||||
(frame as any).click = async (selector: string, options?: any) => {
|
||||
(frame as any).click = async (selector: string, options?: HumanActionOptions & {
|
||||
button?: 'left' | 'right' | 'middle' | 'back' | 'forward';
|
||||
clickCount?: number;
|
||||
count?: number;
|
||||
delay?: number;
|
||||
}) => {
|
||||
await (page as any).click(selector, options);
|
||||
};
|
||||
|
||||
(frame as any).hover = async (selector: string, options?: any) => {
|
||||
(frame as any).hover = async (selector: string, options?: HumanActionOptions) => {
|
||||
await (page as any).hover(selector, options);
|
||||
};
|
||||
|
||||
(frame as any).type = async (selector: string, text: string, options?: any) => {
|
||||
(frame as any).type = async (selector: string, text: string, options?: HumanActionOptions & {
|
||||
delay?: number;
|
||||
}) => {
|
||||
await (page as any).type(selector, text, options);
|
||||
};
|
||||
|
||||
@@ -812,7 +894,7 @@ function patchSingleFrame(
|
||||
await (page as any).focus(selector);
|
||||
};
|
||||
|
||||
(frame as any).tap = async (selector: string, options?: any) => {
|
||||
(frame as any).tap = async (selector: string, options?: HumanActionOptions) => {
|
||||
await (page as any).click(selector, options);
|
||||
};
|
||||
|
||||
@@ -835,7 +917,11 @@ function patchSingleFrame(
|
||||
return els;
|
||||
};
|
||||
|
||||
(frame as any).waitForSelector = async (selector: string, options?: any) => {
|
||||
(frame as any).waitForSelector = async (selector: string, options?: {
|
||||
hidden?: boolean;
|
||||
timeout?: number;
|
||||
visible?: boolean;
|
||||
}) => {
|
||||
const el = await origFrameWaitForSelector(selector, options);
|
||||
if (el) patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return el;
|
||||
@@ -881,7 +967,7 @@ export function patchBrowser(browser: Browser, cfg: HumanConfig): void {
|
||||
for (const methodName of ['createBrowserContext', 'createIncognitoBrowserContext'] as const) {
|
||||
if (typeof (browser as any)[methodName] === 'function') {
|
||||
const origCreateContext = (browser as any)[methodName].bind(browser);
|
||||
(browser as any)[methodName] = async (options?: any) => {
|
||||
(browser as any)[methodName] = async (options?: Parameters<typeof origCreateContext>[0]) => {
|
||||
const context: BrowserContext = await origCreateContext(options);
|
||||
|
||||
const origCtxNewPage = context.newPage.bind(context);
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
* Changes from Playwright version:
|
||||
* - page.viewport() instead of page.viewportSize()
|
||||
* - page.$(selector) + el.boundingBox() instead of page.locator().boundingBox()
|
||||
* - No timeout parameter on boundingBox()
|
||||
* - boundingBox() has no timeout param — we poll page.$() up to ``timeout`` ms
|
||||
*/
|
||||
|
||||
import type { Page } from 'puppeteer-core';
|
||||
@@ -55,22 +55,40 @@ export async function smoothWheel(
|
||||
}
|
||||
}
|
||||
|
||||
async function getElementBox(page: Page, selector: string): Promise<ElementBounds | null> {
|
||||
try {
|
||||
const el = await page.$(selector);
|
||||
if (!el) return null;
|
||||
const box = await el.boundingBox();
|
||||
if (!box) return null;
|
||||
return { x: box.x, y: box.y, width: box.width, height: box.height };
|
||||
} catch {
|
||||
return null;
|
||||
/**
|
||||
* Poll ``page.$(selector)`` for up to ``timeout`` ms, returning the element's
|
||||
* bounding box when found. ``timeout`` defaults to 30000ms when not specified.
|
||||
*/
|
||||
async function getElementBox(
|
||||
page: Page,
|
||||
selector: string,
|
||||
timeout: number = 30000,
|
||||
): Promise<ElementBounds | null> {
|
||||
const start = Date.now();
|
||||
const pollInterval = 100;
|
||||
while (true) {
|
||||
try {
|
||||
const el = await page.$(selector);
|
||||
if (el) {
|
||||
const box = await el.boundingBox();
|
||||
if (box) return { x: box.x, y: box.y, width: box.width, height: box.height };
|
||||
}
|
||||
} catch { /* keep polling */ }
|
||||
|
||||
if (Date.now() - start >= timeout) return null;
|
||||
await sleep(pollInterval);
|
||||
}
|
||||
}
|
||||
|
||||
export async function scrollToElement(
|
||||
/**
|
||||
* Humanized scrolling that takes an arbitrary ``getBox`` callable.
|
||||
* Used by both ``scrollToElement`` (selector-based) and the ElementHandle
|
||||
* ``scrollIntoView`` patch.
|
||||
*/
|
||||
export async function humanScrollIntoView(
|
||||
page: Page,
|
||||
raw: RawMouse,
|
||||
selector: string,
|
||||
getBox: () => Promise<ElementBounds | null>,
|
||||
cursorX: number,
|
||||
cursorY: number,
|
||||
cfg: HumanConfig,
|
||||
@@ -78,12 +96,8 @@ export async function scrollToElement(
|
||||
const viewport = page.viewport();
|
||||
if (!viewport) throw new Error('Viewport size not available');
|
||||
|
||||
let box = await getElementBox(page, selector);
|
||||
if (!box) {
|
||||
await sleep(200);
|
||||
box = await getElementBox(page, selector);
|
||||
if (!box) throw new Error(`Element not found: ${selector}`);
|
||||
}
|
||||
let box = await getBox();
|
||||
if (!box) throw new Error('Element not found while scrolling into view');
|
||||
|
||||
if (isInViewport(box, viewport.height, cfg)) {
|
||||
return { box, cursorX, cursorY };
|
||||
@@ -134,7 +148,7 @@ export async function scrollToElement(
|
||||
await sleep(pause);
|
||||
|
||||
if (i % 3 === 2 || i === totalClicks - 1) {
|
||||
box = await getElementBox(page, selector);
|
||||
box = await getBox();
|
||||
if (box && isInViewport(box, viewport.height, cfg)) {
|
||||
break;
|
||||
}
|
||||
@@ -159,8 +173,31 @@ export async function scrollToElement(
|
||||
|
||||
await sleep(randRange(cfg.scroll_settle_delay));
|
||||
|
||||
box = await getElementBox(page, selector);
|
||||
if (!box) throw new Error(`Element lost after scrolling: ${selector}`);
|
||||
box = await getBox();
|
||||
if (!box) throw new Error('Element lost after scrolling into view');
|
||||
|
||||
return { box, cursorX, cursorY };
|
||||
}
|
||||
|
||||
/**
|
||||
* Selector-based humanized scroll (Puppeteer).
|
||||
*
|
||||
* ``timeout`` controls how long we poll ``page.$(selector)`` before giving up,
|
||||
* so callers like ``page.click('#x', { timeout: 5000 })`` can wait longer for
|
||||
* slow-loading elements (#172). Default matches Playwright's 30000ms when not specified.
|
||||
*/
|
||||
export async function scrollToElement(
|
||||
page: Page,
|
||||
raw: RawMouse,
|
||||
selector: string,
|
||||
cursorX: number,
|
||||
cursorY: number,
|
||||
cfg: HumanConfig,
|
||||
timeout?: number,
|
||||
): Promise<{ box: ElementBounds; cursorX: number; cursorY: number }> {
|
||||
return humanScrollIntoView(
|
||||
page, raw,
|
||||
() => getElementBox(page, selector, timeout),
|
||||
cursorX, cursorY, cfg,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -70,6 +70,11 @@ export interface HumanConfig {
|
||||
|
||||
export type HumanPreset = 'default' | 'careful';
|
||||
|
||||
export type HumanActionOptions = Partial<HumanConfig> & {
|
||||
timeout?: number;
|
||||
human_config?: Partial<HumanConfig>;
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Default preset
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -201,6 +206,22 @@ export function resolveConfig(
|
||||
return { ...base, ...overrides };
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge a partial overrides object on top of an existing HumanConfig.
|
||||
* Returns a new object — the original ``cfg`` is never mutated.
|
||||
*
|
||||
* Used by per-call overrides such as ``page.type(sel, text, { human_config: { typing_delay: 30 } })``
|
||||
* so the same patched page can type different fields at different speeds
|
||||
* without re-patching.
|
||||
*/
|
||||
export function mergeConfig(
|
||||
cfg: HumanConfig,
|
||||
overrides?: Partial<HumanConfig> | null,
|
||||
): HumanConfig {
|
||||
if (!overrides) return cfg;
|
||||
return { ...cfg, ...overrides };
|
||||
}
|
||||
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Utility: random number in range
|
||||
|
||||
@@ -0,0 +1,487 @@
|
||||
/**
|
||||
* ElementHandle humanization for Playwright.
|
||||
*
|
||||
* Mirrors Puppeteer's ElementHandle patching architecture.
|
||||
* Patches page.$(), page.$$(), page.waitForSelector() to return humanized handles,
|
||||
* and patches all interaction methods on each ElementHandle instance.
|
||||
*
|
||||
* Playwright ElementHandle methods patched:
|
||||
* click, dblclick, hover, type, fill, press, selectOption,
|
||||
* check, uncheck, setChecked, tap, focus
|
||||
* + $, $$, waitForSelector (nested elements are also patched)
|
||||
*
|
||||
* Stealth-aware:
|
||||
* - Uses CDP DOM.describeNode when available to check element type
|
||||
* (no main-world JS execution)
|
||||
* - Falls back to el.evaluate() only when CDP is unavailable
|
||||
*/
|
||||
|
||||
import type { Page, Frame, ElementHandle, CDPSession } from 'playwright-core';
|
||||
import type { HumanConfig, HumanActionOptions } from './config.js';
|
||||
import { rand, randRange, sleep, mergeConfig } from './config.js';
|
||||
import { RawMouse, RawKeyboard, humanMove, humanClick, clickTarget, humanIdle } from './mouse.js';
|
||||
import { humanType } from './keyboard.js';
|
||||
import { humanScrollIntoView } from './scroll.js';
|
||||
|
||||
// --- Platform-aware select-all shortcut ---
|
||||
const SELECT_ALL = process.platform === 'darwin' ? 'Meta+a' : 'Control+a';
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// Stealth ElementHandle input check — uses CDP DOM.describeNode
|
||||
// ============================================================================
|
||||
|
||||
async function isInputElementHandle(
|
||||
stealth: any, // StealthEval from index.ts
|
||||
el: ElementHandle,
|
||||
): Promise<boolean> {
|
||||
// Try CDP DOM.describeNode first (no main-world JS execution)
|
||||
if (stealth) {
|
||||
try {
|
||||
const cdp: CDPSession = await stealth.getCdpSession();
|
||||
// Playwright exposes the JSHandle's internal preview via _objectId or similar
|
||||
// We need the remote object ID. Try to get it via internal API.
|
||||
const impl = (el as any)._impl ?? (el as any)._object ?? el;
|
||||
const guid = (impl as any)._guid;
|
||||
|
||||
// Use el.evaluate as a reliable fallback within stealth context
|
||||
// Playwright doesn't expose remoteObject directly like Puppeteer
|
||||
} catch { /* fallthrough */ }
|
||||
}
|
||||
|
||||
// Fallback: el.evaluate (works reliably in Playwright)
|
||||
try {
|
||||
return await el.evaluate((node: any) => {
|
||||
const tag = node.tagName?.toLowerCase();
|
||||
return tag === 'input' || tag === 'textarea'
|
||||
|| node.getAttribute?.('contenteditable') === 'true';
|
||||
});
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// CursorState type (matches index.ts)
|
||||
// ============================================================================
|
||||
|
||||
interface CursorState {
|
||||
x: number;
|
||||
y: number;
|
||||
initialized: boolean;
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// Patch a single Playwright ElementHandle
|
||||
// ============================================================================
|
||||
|
||||
export function patchSingleElementHandle(
|
||||
el: ElementHandle,
|
||||
page: Page,
|
||||
cfg: HumanConfig,
|
||||
cursor: CursorState,
|
||||
raw: RawMouse,
|
||||
rawKb: RawKeyboard,
|
||||
originals: any,
|
||||
stealth: any,
|
||||
): void {
|
||||
if ((el as any)._humanPatched) return;
|
||||
(el as any)._humanPatched = true;
|
||||
|
||||
// Save originals
|
||||
const origElClick = el.click.bind(el);
|
||||
const origElDblclick = el.dblclick.bind(el);
|
||||
const origElHover = el.hover.bind(el);
|
||||
const origElType = el.type.bind(el);
|
||||
const origElFill = el.fill.bind(el);
|
||||
const origElPress = el.press.bind(el);
|
||||
const origElSelectOption = el.selectOption.bind(el);
|
||||
const origElCheck = el.check.bind(el);
|
||||
const origElUncheck = el.uncheck.bind(el);
|
||||
const origElSetChecked = (el as any).setChecked?.bind(el);
|
||||
const origElTap = el.tap.bind(el);
|
||||
const origElFocus = el.focus.bind(el);
|
||||
const origElScrollIntoViewIfNeeded = (el as any).scrollIntoViewIfNeeded?.bind(el);
|
||||
|
||||
// Nested selectors
|
||||
const origEl$ = el.$.bind(el);
|
||||
const origEl$$ = el.$$.bind(el);
|
||||
const origElWaitForSelector = el.waitForSelector.bind(el);
|
||||
|
||||
// --- Nested elements are also patched ---
|
||||
(el as any).$ = async (selector: string) => {
|
||||
const child = await origEl$(selector);
|
||||
if (child) patchSingleElementHandle(child, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return child;
|
||||
};
|
||||
|
||||
(el as any).$$ = async (selector: string) => {
|
||||
const children = await origEl$$(selector);
|
||||
for (const child of children) {
|
||||
patchSingleElementHandle(child, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
}
|
||||
return children;
|
||||
};
|
||||
|
||||
(el as any).waitForSelector = async (selector: string, options?: {
|
||||
state?: 'attached' | 'detached' | 'visible' | 'hidden';
|
||||
strict?: boolean;
|
||||
timeout?: number;
|
||||
}) => {
|
||||
const child = await origElWaitForSelector(selector, options ?? {});
|
||||
if (child) patchSingleElementHandle(child, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return child;
|
||||
};
|
||||
|
||||
// --- Helper: get bounding box and move cursor to element ---
|
||||
// Accepts a per-call ``callCfg`` so type/fill overrides like
|
||||
// ``el.type(text, { human_config: { typing_delay: 30 } })`` or
|
||||
// ``el.type(text, { typing_delay: 30 })`` carry through to mouse movement
|
||||
// & idle timing for that single call.
|
||||
// Also scrolls the element into view first so off-screen elements work
|
||||
// (#129, #172 follow-up): otherwise boundingBox() returns null and we'd
|
||||
// silently fall back to the unpatched native method.
|
||||
const moveToElement = async (callCfg: HumanConfig = cfg) => {
|
||||
// Ensure cursor is initialized
|
||||
const ensureCursorInit = (page as any)._ensureCursorInit;
|
||||
if (ensureCursorInit) await ensureCursorInit();
|
||||
|
||||
// Scroll into view first so boundingBox() returns coordinates even when
|
||||
// the element starts below the fold. Best-effort — if humanScrollIntoView
|
||||
// throws (e.g. detached element), we let boundingBox() decide whether to
|
||||
// proceed or fall back to the original method.
|
||||
try {
|
||||
const { cursorX, cursorY } = await humanScrollIntoView(
|
||||
page, raw,
|
||||
() => el.boundingBox(),
|
||||
cursor.x, cursor.y, callCfg,
|
||||
);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
} catch { /* let boundingBox() decide */ }
|
||||
|
||||
const box = await el.boundingBox();
|
||||
if (!box) return null;
|
||||
|
||||
const isInp = await isInputElementHandle(stealth, el);
|
||||
const target = clickTarget(box, isInp, callCfg);
|
||||
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
return { box, isInp };
|
||||
};
|
||||
|
||||
// --- el.click() ---
|
||||
(el as any).click = async (options?: HumanActionOptions & {
|
||||
button?: 'left' | 'right' | 'middle';
|
||||
clickCount?: number;
|
||||
delay?: number;
|
||||
force?: boolean;
|
||||
modifiers?: Array<'Alt' | 'Control' | 'ControlOrMeta' | 'Meta' | 'Shift'>;
|
||||
noWaitAfter?: boolean;
|
||||
position?: { x: number; y: number };
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const info = await moveToElement(callCfg);
|
||||
if (!info) return origElClick(options);
|
||||
await humanClick(raw, info.isInp, callCfg);
|
||||
};
|
||||
|
||||
// --- el.dblclick() ---
|
||||
(el as any).dblclick = async (options?: HumanActionOptions & {
|
||||
button?: 'left' | 'right' | 'middle';
|
||||
delay?: number;
|
||||
force?: boolean;
|
||||
modifiers?: Array<'Alt' | 'Control' | 'ControlOrMeta' | 'Meta' | 'Shift'>;
|
||||
noWaitAfter?: boolean;
|
||||
position?: { x: number; y: number };
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const info = await moveToElement(callCfg);
|
||||
if (!info) return origElDblclick(options);
|
||||
await raw.down({ clickCount: 2 });
|
||||
await sleep(rand(30, 60));
|
||||
await raw.up({ clickCount: 2 });
|
||||
};
|
||||
|
||||
// --- el.hover() ---
|
||||
(el as any).hover = async (options?: HumanActionOptions & {
|
||||
force?: boolean;
|
||||
modifiers?: Array<'Alt' | 'Control' | 'ControlOrMeta' | 'Meta' | 'Shift'>;
|
||||
position?: { x: number; y: number };
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const info = await moveToElement(callCfg);
|
||||
if (!info) return origElHover(options);
|
||||
// Just move — no click
|
||||
};
|
||||
|
||||
// --- el.type() ---
|
||||
(el as any).type = async (text: string, options?: HumanActionOptions & {
|
||||
delay?: number;
|
||||
noWaitAfter?: boolean;
|
||||
}) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const info = await moveToElement(callCfg);
|
||||
if (!info) return origElType(text, options);
|
||||
await humanClick(raw, info.isInp, callCfg);
|
||||
await sleep(rand(100, 250));
|
||||
let cdpSession: CDPSession | null = null;
|
||||
try { cdpSession = await stealth?.getCdpSession(); } catch {}
|
||||
await humanType(page, rawKb, text, callCfg, cdpSession);
|
||||
};
|
||||
|
||||
// --- el.fill() ---
|
||||
(el as any).fill = async (value: string, options?: HumanActionOptions & {
|
||||
force?: boolean;
|
||||
noWaitAfter?: boolean;
|
||||
}) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const info = await moveToElement(callCfg);
|
||||
if (!info) return origElFill(value, options);
|
||||
await humanClick(raw, info.isInp, callCfg);
|
||||
await sleep(rand(100, 250));
|
||||
// Clear existing content
|
||||
await originals.keyboardPress(SELECT_ALL);
|
||||
await sleep(rand(30, 80));
|
||||
await originals.keyboardPress('Backspace');
|
||||
await sleep(rand(50, 150));
|
||||
let cdpSession: CDPSession | null = null;
|
||||
try { cdpSession = await stealth?.getCdpSession(); } catch {}
|
||||
await humanType(page, rawKb, value, callCfg, cdpSession);
|
||||
};
|
||||
|
||||
// --- el.press() ---
|
||||
(el as any).press = async (key: string, options?: { delay?: number; noWaitAfter?: boolean; timeout?: number }) => {
|
||||
await sleep(rand(20, 60));
|
||||
await originals.keyboardDown(key);
|
||||
await sleep(randRange(cfg.key_hold));
|
||||
await originals.keyboardUp(key);
|
||||
};
|
||||
|
||||
// --- el.selectOption() ---
|
||||
(el as any).selectOption = async (values: any, options?: {
|
||||
force?: boolean;
|
||||
noWaitAfter?: boolean;
|
||||
timeout?: number;
|
||||
}) => {
|
||||
const info = await moveToElement();
|
||||
if (!info) return origElSelectOption(values, options);
|
||||
await humanClick(raw, false, cfg);
|
||||
await sleep(rand(100, 300));
|
||||
return origElSelectOption(values, options);
|
||||
};
|
||||
|
||||
// --- el.check() ---
|
||||
(el as any).check = async (options?: {
|
||||
force?: boolean;
|
||||
noWaitAfter?: boolean;
|
||||
position?: { x: number; y: number };
|
||||
timeout?: number;
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
try {
|
||||
const checked = await el.isChecked();
|
||||
if (checked) return; // Already checked
|
||||
} catch {}
|
||||
const info = await moveToElement();
|
||||
if (!info) return origElCheck(options);
|
||||
await humanClick(raw, info.isInp, cfg);
|
||||
};
|
||||
|
||||
// --- el.uncheck() ---
|
||||
(el as any).uncheck = async (options?: {
|
||||
force?: boolean;
|
||||
noWaitAfter?: boolean;
|
||||
position?: { x: number; y: number };
|
||||
timeout?: number;
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
try {
|
||||
const checked = await el.isChecked();
|
||||
if (!checked) return; // Already unchecked
|
||||
} catch {}
|
||||
const info = await moveToElement();
|
||||
if (!info) return origElUncheck(options);
|
||||
await humanClick(raw, info.isInp, cfg);
|
||||
};
|
||||
|
||||
// --- el.setChecked() ---
|
||||
if (origElSetChecked) {
|
||||
(el as any).setChecked = async (checked: boolean, options?: {
|
||||
force?: boolean;
|
||||
noWaitAfter?: boolean;
|
||||
position?: { x: number; y: number };
|
||||
timeout?: number;
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
try {
|
||||
const current = await el.isChecked();
|
||||
if (current === checked) return;
|
||||
} catch {}
|
||||
const info = await moveToElement();
|
||||
if (!info) return origElSetChecked(checked, options);
|
||||
await humanClick(raw, info.isInp, cfg);
|
||||
};
|
||||
}
|
||||
|
||||
// --- el.tap() ---
|
||||
(el as any).tap = async (options?: {
|
||||
force?: boolean;
|
||||
modifiers?: Array<'Alt' | 'Control' | 'ControlOrMeta' | 'Meta' | 'Shift'>;
|
||||
noWaitAfter?: boolean;
|
||||
position?: { x: number; y: number };
|
||||
timeout?: number;
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
const info = await moveToElement();
|
||||
if (!info) return origElTap(options);
|
||||
await humanClick(raw, info.isInp, cfg);
|
||||
};
|
||||
|
||||
// --- el.focus() ---
|
||||
// Move cursor humanly but use programmatic focus (no click side-effects).
|
||||
// Stock Playwright el.focus() never clicks — clicking would trigger onclick,
|
||||
// submit forms, navigate links, etc.
|
||||
(el as any).focus = async () => {
|
||||
await moveToElement(); // human-like Bézier cursor movement
|
||||
await origElFocus(); // programmatic focus, no click
|
||||
};
|
||||
|
||||
// --- el.scrollIntoViewIfNeeded() ---
|
||||
// Playwright's native version snaps the page — a strong bot signal.
|
||||
// Replace with the same accelerate → cruise → decelerate → overshoot
|
||||
// wheel sequence used by page.click() etc. Falls back to the native
|
||||
// method if the element is detached or scrolling fails.
|
||||
if (origElScrollIntoViewIfNeeded) {
|
||||
(el as any).scrollIntoViewIfNeeded = async (options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
const ensureCursorInit = (page as any)._ensureCursorInit;
|
||||
if (ensureCursorInit) await ensureCursorInit();
|
||||
try {
|
||||
const { cursorX, cursorY } = await humanScrollIntoView(
|
||||
page, raw,
|
||||
() => el.boundingBox(),
|
||||
cursor.x, cursor.y, callCfg,
|
||||
);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
} catch {
|
||||
return origElScrollIntoViewIfNeeded(options);
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// Page-level ElementHandle patching
|
||||
// ============================================================================
|
||||
|
||||
export function patchPageElementHandles(
|
||||
page: Page,
|
||||
cfg: HumanConfig,
|
||||
cursor: CursorState,
|
||||
raw: RawMouse,
|
||||
rawKb: RawKeyboard,
|
||||
originals: any,
|
||||
stealth: any,
|
||||
): void {
|
||||
// Patch page.$() — only if the method exists
|
||||
if (typeof page.$ === 'function') {
|
||||
const orig$ = page.$.bind(page);
|
||||
(page as any).$ = async (selector: string) => {
|
||||
const el = await orig$(selector);
|
||||
if (el) patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return el;
|
||||
};
|
||||
}
|
||||
|
||||
// Patch page.$$()
|
||||
if (typeof page.$$ === 'function') {
|
||||
const orig$$ = page.$$.bind(page);
|
||||
(page as any).$$ = async (selector: string) => {
|
||||
const els = await orig$$(selector);
|
||||
for (const el of els) {
|
||||
patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
}
|
||||
return els;
|
||||
};
|
||||
}
|
||||
|
||||
// Patch page.waitForSelector()
|
||||
if (typeof page.waitForSelector === 'function') {
|
||||
const origWaitForSelector = page.waitForSelector.bind(page);
|
||||
(page as any).waitForSelector = async (selector: string, options?: {
|
||||
state?: 'attached' | 'detached' | 'visible' | 'hidden';
|
||||
strict?: boolean;
|
||||
timeout?: number;
|
||||
}) => {
|
||||
const el = await origWaitForSelector(selector, options ?? {});
|
||||
if (el) patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return el;
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// Frame-level ElementHandle patching
|
||||
// ============================================================================
|
||||
|
||||
export function patchFrameElementHandles(
|
||||
frame: Frame,
|
||||
page: Page,
|
||||
cfg: HumanConfig,
|
||||
cursor: CursorState,
|
||||
raw: RawMouse,
|
||||
rawKb: RawKeyboard,
|
||||
originals: any,
|
||||
stealth: any,
|
||||
): void {
|
||||
// Patch frame.$() — only if the method exists
|
||||
if (typeof frame.$ === 'function') {
|
||||
const origFrame$ = frame.$.bind(frame);
|
||||
(frame as any).$ = async (selector: string) => {
|
||||
const el = await origFrame$(selector);
|
||||
if (el) patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return el;
|
||||
};
|
||||
}
|
||||
|
||||
// Patch frame.$$()
|
||||
if (typeof frame.$$ === 'function') {
|
||||
const origFrame$$ = frame.$$.bind(frame);
|
||||
(frame as any).$$ = async (selector: string) => {
|
||||
const els = await origFrame$$(selector);
|
||||
for (const el of els) {
|
||||
patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
}
|
||||
return els;
|
||||
};
|
||||
}
|
||||
|
||||
// Patch frame.waitForSelector()
|
||||
if (typeof frame.waitForSelector === 'function') {
|
||||
const origFrameWaitForSelector = frame.waitForSelector.bind(frame);
|
||||
(frame as any).waitForSelector = async (selector: string, options?: {
|
||||
state?: 'attached' | 'detached' | 'visible' | 'hidden';
|
||||
strict?: boolean;
|
||||
timeout?: number;
|
||||
}) => {
|
||||
const el = await origFrameWaitForSelector(selector, options ?? {});
|
||||
if (el) patchSingleElementHandle(el, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
return el;
|
||||
};
|
||||
}
|
||||
}
|
||||
+221
-84
@@ -12,18 +12,28 @@
|
||||
* Patches all interaction methods:
|
||||
* click, dblclick, hover, type, fill, check, uncheck, selectOption,
|
||||
* press, pressSequentially, tap, dragTo, clear + Frame-level equivalents.
|
||||
*
|
||||
* ELEMENTHANDLE-LEVEL:
|
||||
* click, dblclick, hover, type, fill, press, selectOption,
|
||||
* check, uncheck, setChecked, tap, focus
|
||||
* + $, $$, waitForSelector (nested elements are also patched)
|
||||
*
|
||||
* page.$(), page.$$(), page.waitForSelector() and Frame equivalents
|
||||
* return patched ElementHandles automatically.
|
||||
*/
|
||||
|
||||
import type { Browser, BrowserContext, Page, Frame, CDPSession } from 'playwright-core';
|
||||
import { HumanConfig, resolveConfig, rand, randRange, sleep } from './config.js';
|
||||
import { HumanConfig, HumanActionOptions, resolveConfig, mergeConfig, rand, randRange, sleep } from './config.js';
|
||||
import { RawMouse, RawKeyboard, humanMove, humanClick, clickTarget, humanIdle } from './mouse.js';
|
||||
import { humanType } from './keyboard.js';
|
||||
import { scrollToElement } from './scroll.js';
|
||||
import { scrollToElement, humanScrollIntoView } from './scroll.js';
|
||||
import { patchPageElementHandles, patchFrameElementHandles, patchSingleElementHandle } from './elementhandle.js';
|
||||
|
||||
export { HumanConfig, resolveConfig } from './config.js';
|
||||
export { HumanConfig, resolveConfig, mergeConfig } from './config.js';
|
||||
export { humanMove, humanClick, clickTarget, humanIdle } from './mouse.js';
|
||||
export { humanType } from './keyboard.js';
|
||||
export { scrollToElement } from './scroll.js';
|
||||
export { scrollToElement, humanScrollIntoView } from './scroll.js';
|
||||
export { patchSingleElementHandle } from './elementhandle.js';
|
||||
|
||||
// --- Platform-aware select-all shortcut (macOS uses Meta, others use Control) ---
|
||||
const SELECT_ALL = process.platform === 'darwin' ? 'Meta+a' : 'Control+a';
|
||||
@@ -285,7 +295,11 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
}
|
||||
|
||||
// --- goto (invalidate isolated world on navigation) ---
|
||||
const humanGoto = async (url: string, options?: any) => {
|
||||
const humanGoto = async (url: string, options?: {
|
||||
referer?: string;
|
||||
timeout?: number;
|
||||
waitUntil?: 'load' | 'domcontentloaded' | 'networkidle' | 'commit';
|
||||
}) => {
|
||||
const response = await originals.goto(url, options);
|
||||
stealth.invalidate();
|
||||
patchFrames(page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
@@ -293,34 +307,37 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
};
|
||||
|
||||
// --- click ---
|
||||
const humanClickFn = async (selector: string, options?: any) => {
|
||||
const humanClickFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
await ensureCursorInit();
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, cfg);
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, callCfg, options?.timeout);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
const isInput = await isInputElement(stealth, page, selector);
|
||||
const target = clickTarget(box, isInput, cfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, cfg);
|
||||
const target = clickTarget(box, isInput, callCfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
await humanClick(raw, isInput, cfg);
|
||||
await humanClick(raw, isInput, callCfg);
|
||||
};
|
||||
|
||||
// --- dblclick ---
|
||||
const humanDblclickFn = async (selector: string, options?: any) => {
|
||||
const humanDblclickFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
await ensureCursorInit();
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, cfg);
|
||||
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, callCfg, options?.timeout);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
const isInput = await isInputElement(stealth, page, selector);
|
||||
const target = clickTarget(box, isInput, cfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, cfg);
|
||||
const target = clickTarget(box, isInput, callCfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
await raw.down({ clickCount: 2 });
|
||||
@@ -329,46 +346,49 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
};
|
||||
|
||||
// --- hover ---
|
||||
const humanHoverFn = async (selector: string, options?: any) => {
|
||||
const humanHoverFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
await ensureCursorInit();
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, cfg);
|
||||
const { box, cursorX, cursorY } = await scrollToElement(page, raw, selector, cursor.x, cursor.y, callCfg, options?.timeout);
|
||||
cursor.x = cursorX;
|
||||
cursor.y = cursorY;
|
||||
const target = clickTarget(box, false, cfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, cfg);
|
||||
const target = clickTarget(box, false, callCfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
};
|
||||
|
||||
// --- type ---
|
||||
const humanTypeFn = async (selector: string, text: string, options?: any) => {
|
||||
await sleep(randRange(cfg.field_switch_delay));
|
||||
await humanClickFn(selector);
|
||||
const humanTypeFn = async (selector: string, text: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
await sleep(randRange(callCfg.field_switch_delay));
|
||||
await humanClickFn(selector, options);
|
||||
await sleep(rand(100, 250));
|
||||
const cdp = await ensureCdp();
|
||||
await humanType(page, rawKb, text, cfg, cdp);
|
||||
await humanType(page, rawKb, text, callCfg, cdp);
|
||||
};
|
||||
|
||||
// --- fill (clears existing content first) ---
|
||||
const humanFillFn = async (selector: string, value: string, options?: any) => {
|
||||
await sleep(randRange(cfg.field_switch_delay));
|
||||
await humanClickFn(selector);
|
||||
const humanFillFn = async (selector: string, value: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
await sleep(randRange(callCfg.field_switch_delay));
|
||||
await humanClickFn(selector, options);
|
||||
await sleep(rand(100, 250));
|
||||
await originals.keyboardPress(SELECT_ALL);
|
||||
await sleep(rand(30, 80));
|
||||
await originals.keyboardPress('Backspace');
|
||||
await sleep(rand(50, 150));
|
||||
const cdp = await ensureCdp();
|
||||
await humanType(page, rawKb, value, cfg, cdp);
|
||||
await humanType(page, rawKb, value, callCfg, cdp);
|
||||
};
|
||||
|
||||
// --- clear ---
|
||||
const humanClearFn = async (selector: string, options?: any) => {
|
||||
const humanClearFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
if (!await isSelectorFocused(stealth, page, selector)) {
|
||||
await humanClickFn(selector);
|
||||
await humanClickFn(selector, options);
|
||||
}
|
||||
await sleep(rand(50, 150));
|
||||
await originals.keyboardPress(SELECT_ALL);
|
||||
@@ -377,55 +397,58 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
};
|
||||
|
||||
// --- check ---
|
||||
const humanCheckFn = async (selector: string, options?: any) => {
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
const humanCheckFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
const checked = await originals.isChecked(selector).catch(() => false);
|
||||
if (!checked) {
|
||||
await humanClickFn(selector);
|
||||
await humanClickFn(selector, options);
|
||||
}
|
||||
};
|
||||
|
||||
// --- uncheck ---
|
||||
const humanUncheckFn = async (selector: string, options?: any) => {
|
||||
if (cfg.idle_between_actions) {
|
||||
await humanIdle(raw, rand(cfg.idle_between_duration[0], cfg.idle_between_duration[1]), cursor.x, cursor.y, cfg);
|
||||
const humanUncheckFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
const checked = await originals.isChecked(selector).catch(() => true);
|
||||
if (checked) {
|
||||
await humanClickFn(selector);
|
||||
await humanClickFn(selector, options);
|
||||
}
|
||||
};
|
||||
|
||||
// --- selectOption ---
|
||||
const humanSelectOptionFn = async (selector: string, values: any, options?: any) => {
|
||||
await humanHoverFn(selector);
|
||||
const humanSelectOptionFn = async (selector: string, values: any, options?: HumanActionOptions) => {
|
||||
await humanHoverFn(selector, options);
|
||||
await sleep(rand(100, 300));
|
||||
return originals.selectOption(selector, values, options);
|
||||
};
|
||||
|
||||
// --- press (checks focus first — avoids redundant mouse moves) ---
|
||||
const humanPressFn = async (selector: string, key: string, options?: any) => {
|
||||
const humanPressFn = async (selector: string, key: string, options?: HumanActionOptions) => {
|
||||
if (!await isSelectorFocused(stealth, page, selector)) {
|
||||
await humanClickFn(selector);
|
||||
await humanClickFn(selector, options);
|
||||
}
|
||||
await sleep(rand(50, 150));
|
||||
await originals.keyboardPress(key);
|
||||
};
|
||||
|
||||
// --- pressSequentially ---
|
||||
const humanPressSequentiallyFn = async (selector: string, text: string, options?: any) => {
|
||||
const humanPressSequentiallyFn = async (selector: string, text: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (!await isSelectorFocused(stealth, page, selector)) {
|
||||
await humanClickFn(selector);
|
||||
await humanClickFn(selector, options);
|
||||
}
|
||||
await sleep(rand(100, 250));
|
||||
const cdp = await ensureCdp();
|
||||
await humanType(page, rawKb, text, cfg, cdp);
|
||||
await humanType(page, rawKb, text, callCfg, cdp);
|
||||
};
|
||||
|
||||
// --- tap ---
|
||||
const humanTapFn = async (selector: string, options?: any) => {
|
||||
const humanTapFn = async (selector: string, options?: HumanActionOptions) => {
|
||||
await humanClickFn(selector, options);
|
||||
};
|
||||
|
||||
@@ -445,14 +468,20 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
(page as any).clear = humanClearFn;
|
||||
|
||||
// --- mouse patches ---
|
||||
page.mouse.move = async (x: number, y: number, options?: any) => {
|
||||
page.mouse.move = async (x: number, y: number, options?: {
|
||||
steps?: number;
|
||||
}) => {
|
||||
await ensureCursorInit();
|
||||
await humanMove(raw, cursor.x, cursor.y, x, y, cfg);
|
||||
cursor.x = x;
|
||||
cursor.y = y;
|
||||
};
|
||||
|
||||
page.mouse.click = async (x: number, y: number, options?: any) => {
|
||||
page.mouse.click = async (x: number, y: number, options?: {
|
||||
button?: 'left' | 'right' | 'middle';
|
||||
clickCount?: number;
|
||||
delay?: number;
|
||||
}) => {
|
||||
await ensureCursorInit();
|
||||
await humanMove(raw, cursor.x, cursor.y, x, y, cfg);
|
||||
cursor.x = x;
|
||||
@@ -461,7 +490,7 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
};
|
||||
|
||||
// --- keyboard patches ---
|
||||
page.keyboard.type = async (text: string, options?: any) => {
|
||||
page.keyboard.type = async (text: string, options?: { delay?: number }) => {
|
||||
const cdp = await ensureCdp();
|
||||
await humanType(page, rawKb, text, cfg, cdp);
|
||||
};
|
||||
@@ -488,6 +517,9 @@ function patchPage(page: Page, cfg: HumanConfig, cursor: CursorState): void {
|
||||
|
||||
// --- Patch Frame-level methods (for sub-frames) ---
|
||||
patchFrames(page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
|
||||
// --- Patch ElementHandle selectors (page.$, page.$$, page.waitForSelector) ---
|
||||
patchPageElementHandles(page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
}
|
||||
|
||||
|
||||
@@ -510,14 +542,37 @@ function patchFrames(
|
||||
stealth: StealthEval,
|
||||
): void {
|
||||
for (const frame of iterFrames(page)) {
|
||||
patchSingleFrame(frame, page, cfg, originals, stealth);
|
||||
patchSingleFrame(frame, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
// Patch frame-level ElementHandle selectors ($, $$, waitForSelector)
|
||||
patchFrameElementHandles(frame, page, cfg, cursor, raw, rawKb, originals, stealth);
|
||||
}
|
||||
}
|
||||
|
||||
function firstFrameLocator(frame: Frame, selector: string): any {
|
||||
const locator = frame.locator(selector) as any;
|
||||
return typeof locator.first === 'function' ? locator.first() : locator;
|
||||
}
|
||||
|
||||
async function isFrameInputElement(frame: Frame, selector: string): Promise<boolean> {
|
||||
return firstFrameLocator(frame, selector).evaluate((el: Element) => {
|
||||
const tag = el.tagName.toLowerCase();
|
||||
return tag === 'input' || tag === 'textarea'
|
||||
|| el.getAttribute('contenteditable') === 'true';
|
||||
}).catch(() => false);
|
||||
}
|
||||
|
||||
async function isFrameSelectorFocused(frame: Frame, selector: string): Promise<boolean> {
|
||||
return firstFrameLocator(frame, selector).evaluate((el: Element) => el === document.activeElement)
|
||||
.catch(() => false);
|
||||
}
|
||||
|
||||
function patchSingleFrame(
|
||||
frame: Frame,
|
||||
page: Page,
|
||||
cfg: HumanConfig,
|
||||
cursor: CursorState,
|
||||
raw: RawMouse,
|
||||
rawKb: RawKeyboard,
|
||||
originals: any,
|
||||
stealth: StealthEval,
|
||||
): void {
|
||||
@@ -525,58 +580,132 @@ function patchSingleFrame(
|
||||
(frame as any)._humanPatched = true;
|
||||
|
||||
// Save originals for methods that need fallback
|
||||
const origFrameClick = frame.click.bind(frame);
|
||||
const origFrameDblclick = frame.dblclick.bind(frame);
|
||||
const origFrameHover = frame.hover.bind(frame);
|
||||
const origFrameType = frame.type.bind(frame);
|
||||
const origFrameFill = frame.fill.bind(frame);
|
||||
const origFrameCheck = frame.check.bind(frame);
|
||||
const origFrameUncheck = frame.uncheck.bind(frame);
|
||||
const origFrameSelectOption = frame.selectOption.bind(frame);
|
||||
const origFramePress = frame.press.bind(frame);
|
||||
const origFramePressSequentially = (frame as any).pressSequentially?.bind(frame);
|
||||
const origFrameTap = (frame as any).tap?.bind(frame);
|
||||
const origFrameDragAndDrop = frame.dragAndDrop.bind(frame);
|
||||
|
||||
(frame as any).click = async (selector: string, options?: any) => {
|
||||
await (page as any).click(selector, options);
|
||||
const moveToFrameSelector = async (selector: string, options?: HumanActionOptions, inputBias = false) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (callCfg.idle_between_actions) {
|
||||
await humanIdle(raw, cursor.x, cursor.y, callCfg);
|
||||
}
|
||||
|
||||
const locator = firstFrameLocator(frame, selector);
|
||||
if (typeof locator.scrollIntoViewIfNeeded === 'function') {
|
||||
await locator.scrollIntoViewIfNeeded({ timeout: options?.timeout }).catch(() => undefined);
|
||||
}
|
||||
const box = await locator.boundingBox({ timeout: options?.timeout ?? 30000 }).catch(() => null);
|
||||
if (!box) return null;
|
||||
|
||||
const isInput = inputBias || await isFrameInputElement(frame, selector);
|
||||
const target = clickTarget(box, isInput, callCfg);
|
||||
await humanMove(raw, cursor.x, cursor.y, target.x, target.y, callCfg);
|
||||
cursor.x = target.x;
|
||||
cursor.y = target.y;
|
||||
return { callCfg, isInput };
|
||||
};
|
||||
|
||||
(frame as any).dblclick = async (selector: string, options?: any) => {
|
||||
await (page as any).dblclick(selector, options);
|
||||
const frameClick = async (selector: string, options?: HumanActionOptions) => {
|
||||
const moved = await moveToFrameSelector(selector, options);
|
||||
if (!moved) return origFrameClick(selector, options);
|
||||
await humanClick(raw, moved.isInput, moved.callCfg);
|
||||
};
|
||||
|
||||
(frame as any).hover = async (selector: string, options?: any) => {
|
||||
await (page as any).hover(selector, options);
|
||||
const getFrameCdp = async () => stealth.getCdpSession().catch(() => null);
|
||||
|
||||
const frameHover = async (selector: string, options?: HumanActionOptions) => {
|
||||
const moved = await moveToFrameSelector(selector, options, false);
|
||||
if (!moved) return origFrameHover(selector, options);
|
||||
};
|
||||
|
||||
(frame as any).type = async (selector: string, text: string, options?: any) => {
|
||||
await (page as any).type(selector, text, options);
|
||||
(frame as any).click = frameClick;
|
||||
|
||||
(frame as any).dblclick = async (selector: string, options?: HumanActionOptions) => {
|
||||
const moved = await moveToFrameSelector(selector, options);
|
||||
if (!moved) return origFrameDblclick(selector, options);
|
||||
await raw.down({ clickCount: 2 });
|
||||
await sleep(rand(30, 60));
|
||||
await raw.up({ clickCount: 2 });
|
||||
};
|
||||
|
||||
(frame as any).fill = async (selector: string, value: string, options?: any) => {
|
||||
await (page as any).fill(selector, value, options);
|
||||
(frame as any).hover = frameHover;
|
||||
|
||||
(frame as any).type = async (selector: string, text: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
await sleep(randRange(callCfg.field_switch_delay));
|
||||
await frameClick(selector, options);
|
||||
await sleep(rand(100, 250));
|
||||
const cdp = await getFrameCdp();
|
||||
await humanType(page, rawKb, text, callCfg, cdp).catch(() => origFrameType(selector, text, options));
|
||||
};
|
||||
|
||||
(frame as any).check = async (selector: string, options?: any) => {
|
||||
await (page as any).check(selector, options);
|
||||
(frame as any).fill = async (selector: string, value: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
await sleep(randRange(callCfg.field_switch_delay));
|
||||
await frameClick(selector, options);
|
||||
await sleep(rand(100, 250));
|
||||
await originals.keyboardPress(SELECT_ALL);
|
||||
await sleep(rand(30, 80));
|
||||
await originals.keyboardPress('Backspace');
|
||||
await sleep(rand(50, 150));
|
||||
const cdp = await getFrameCdp();
|
||||
await humanType(page, rawKb, value, callCfg, cdp).catch(() => origFrameFill(selector, value, options));
|
||||
};
|
||||
|
||||
(frame as any).uncheck = async (selector: string, options?: any) => {
|
||||
await (page as any).uncheck(selector, options);
|
||||
(frame as any).check = async (selector: string, options?: HumanActionOptions) => {
|
||||
const locator = firstFrameLocator(frame, selector);
|
||||
if (typeof locator.isChecked !== 'function') return origFrameCheck(selector, options);
|
||||
const checked = await locator.isChecked();
|
||||
if (!checked) await frameClick(selector, options).catch(() => origFrameCheck(selector, options));
|
||||
};
|
||||
|
||||
(frame as any).selectOption = async (selector: string, values: any, options?: any) => {
|
||||
await (page as any).hover(selector);
|
||||
(frame as any).uncheck = async (selector: string, options?: HumanActionOptions) => {
|
||||
const locator = firstFrameLocator(frame, selector);
|
||||
if (typeof locator.isChecked !== 'function') return origFrameUncheck(selector, options);
|
||||
const checked = await locator.isChecked();
|
||||
if (checked) await frameClick(selector, options).catch(() => origFrameUncheck(selector, options));
|
||||
};
|
||||
|
||||
(frame as any).selectOption = async (selector: string, values: any, options?: HumanActionOptions) => {
|
||||
await frameHover(selector, options);
|
||||
await sleep(rand(100, 300));
|
||||
return origFrameSelectOption(selector, values, options);
|
||||
};
|
||||
|
||||
(frame as any).press = async (selector: string, key: string, options?: any) => {
|
||||
await (page as any).press(selector, key, options);
|
||||
(frame as any).press = async (selector: string, key: string, options?: HumanActionOptions) => {
|
||||
if (!await isFrameSelectorFocused(frame, selector)) {
|
||||
await frameClick(selector, options);
|
||||
}
|
||||
await sleep(rand(50, 150));
|
||||
await originals.keyboardPress(key);
|
||||
};
|
||||
|
||||
(frame as any).pressSequentially = async (selector: string, text: string, options?: any) => {
|
||||
await (page as any).pressSequentially(selector, text, options);
|
||||
(frame as any).pressSequentially = async (selector: string, text: string, options?: HumanActionOptions) => {
|
||||
const callCfg = mergeConfig(cfg, options?.human_config ?? options);
|
||||
if (!await isFrameSelectorFocused(frame, selector)) {
|
||||
await frameClick(selector, options);
|
||||
}
|
||||
await sleep(rand(100, 250));
|
||||
const cdp = await getFrameCdp();
|
||||
await humanType(page, rawKb, text, callCfg, cdp).catch(() => origFramePressSequentially?.(selector, text, options));
|
||||
};
|
||||
|
||||
(frame as any).tap = async (selector: string, options?: any) => {
|
||||
await (page as any).tap(selector, options);
|
||||
(frame as any).tap = async (selector: string, options?: HumanActionOptions) => {
|
||||
await frameClick(selector, options).catch(() => origFrameTap?.(selector, options));
|
||||
};
|
||||
|
||||
(frame as any).clear = async (selector: string, options?: any) => {
|
||||
if (!await isSelectorFocused(stealth, page, selector)) {
|
||||
await (page as any).click(selector);
|
||||
(frame as any).clear = async (selector: string, options?: HumanActionOptions) => {
|
||||
if (!await isFrameSelectorFocused(frame, selector)) {
|
||||
await frameClick(selector, options);
|
||||
}
|
||||
await sleep(rand(50, 150));
|
||||
await originals.keyboardPress(SELECT_ALL);
|
||||
@@ -584,9 +713,17 @@ function patchSingleFrame(
|
||||
await originals.keyboardPress('Backspace');
|
||||
};
|
||||
|
||||
(frame as any).dragAndDrop = async (source: string, target: string, options?: any) => {
|
||||
const srcBox = await frame.locator(source).boundingBox().catch(() => null);
|
||||
const tgtBox = await frame.locator(target).boundingBox().catch(() => null);
|
||||
(frame as any).dragAndDrop = async (source: string, target: string, options?: {
|
||||
force?: boolean;
|
||||
noWaitAfter?: boolean;
|
||||
sourcePosition?: { x: number; y: number };
|
||||
strict?: boolean;
|
||||
targetPosition?: { x: number; y: number };
|
||||
timeout?: number;
|
||||
trial?: boolean;
|
||||
}) => {
|
||||
const srcBox = await firstFrameLocator(frame, source).boundingBox({ timeout: options?.timeout ?? 30000 }).catch(() => null);
|
||||
const tgtBox = await firstFrameLocator(frame, target).boundingBox({ timeout: options?.timeout ?? 30000 }).catch(() => null);
|
||||
|
||||
if (srcBox && tgtBox) {
|
||||
const sx = srcBox.x + srcBox.width / 2;
|
||||
@@ -655,14 +792,14 @@ export function patchBrowser(browser: Browser, cfg: HumanConfig): void {
|
||||
}
|
||||
|
||||
const origNewContext = browser.newContext.bind(browser);
|
||||
(browser as any).newContext = async (options?: any) => {
|
||||
(browser as any).newContext = async (options?: Parameters<typeof origNewContext>[0]) => {
|
||||
const context = await origNewContext(options);
|
||||
patchContext(context, cfg);
|
||||
return context;
|
||||
};
|
||||
|
||||
const origNewPage = browser.newPage.bind(browser);
|
||||
(browser as any).newPage = async (options?: any) => {
|
||||
(browser as any).newPage = async (options?: Parameters<typeof origNewPage>[0]) => {
|
||||
const page = await origNewPage(options);
|
||||
if (!(page as any)._original) {
|
||||
const ctx = page.context();
|
||||
|
||||
+21
-1
@@ -172,13 +172,33 @@ export async function humanClick(
|
||||
// Human idle / drift
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function humanIdle(
|
||||
export function humanIdle(
|
||||
raw: RawMouse,
|
||||
cx: number,
|
||||
cy: number,
|
||||
cfg: HumanConfig,
|
||||
): Promise<void>;
|
||||
export function humanIdle(
|
||||
raw: RawMouse,
|
||||
seconds: number,
|
||||
cx: number,
|
||||
cy: number,
|
||||
cfg: HumanConfig,
|
||||
): Promise<void>;
|
||||
export async function humanIdle(
|
||||
raw: RawMouse,
|
||||
secondsOrCx: number,
|
||||
cxOrCy: number,
|
||||
cyOrCfg: number | HumanConfig,
|
||||
maybeCfg?: HumanConfig,
|
||||
): Promise<void> {
|
||||
const hasExplicitSeconds = maybeCfg !== undefined;
|
||||
const seconds = hasExplicitSeconds
|
||||
? secondsOrCx
|
||||
: rand((cyOrCfg as HumanConfig).idle_between_duration[0], (cyOrCfg as HumanConfig).idle_between_duration[1]);
|
||||
const cx = hasExplicitSeconds ? cxOrCy : secondsOrCx;
|
||||
const cy = hasExplicitSeconds ? (cyOrCfg as number) : cxOrCy;
|
||||
const cfg = hasExplicitSeconds ? maybeCfg! : (cyOrCfg as HumanConfig);
|
||||
const endTime = Date.now() + seconds * 1000;
|
||||
let x = cx;
|
||||
let y = cy;
|
||||
|
||||
+43
-13
@@ -38,10 +38,17 @@ async function smoothWheel(raw: RawMouse, delta: number, cfg: HumanConfig): Prom
|
||||
}
|
||||
}
|
||||
|
||||
export async function scrollToElement(
|
||||
/**
|
||||
* Humanized scrolling that takes an arbitrary ``getBox`` callable.
|
||||
*
|
||||
* Used by both ``scrollToElement`` (selector-based) and the ElementHandle
|
||||
* ``scrollIntoViewIfNeeded`` patch so the same accelerate → cruise →
|
||||
* decelerate → overshoot behavior runs everywhere.
|
||||
*/
|
||||
export async function humanScrollIntoView(
|
||||
page: Page,
|
||||
raw: RawMouse,
|
||||
selector: string,
|
||||
getBox: () => Promise<ElementBounds | null>,
|
||||
cursorX: number,
|
||||
cursorY: number,
|
||||
cfg: HumanConfig,
|
||||
@@ -49,12 +56,8 @@ export async function scrollToElement(
|
||||
const viewport = page.viewportSize();
|
||||
if (!viewport) throw new Error('Viewport size not available');
|
||||
|
||||
let box = await getElementBox(page, selector);
|
||||
if (!box) {
|
||||
await sleep(200);
|
||||
box = await getElementBox(page, selector);
|
||||
if (!box) throw new Error(`Element not found: ${selector}`);
|
||||
}
|
||||
let box = await getBox();
|
||||
if (!box) throw new Error('Element not found while scrolling into view');
|
||||
|
||||
if (isInViewport(box, viewport.height, cfg)) {
|
||||
return { box, cursorX, cursorY };
|
||||
@@ -107,7 +110,7 @@ export async function scrollToElement(
|
||||
|
||||
// Check visibility every 3 steps
|
||||
if (i % 3 === 2 || i === totalClicks - 1) {
|
||||
box = await getElementBox(page, selector);
|
||||
box = await getBox();
|
||||
if (box && isInViewport(box, viewport.height, cfg)) {
|
||||
break;
|
||||
}
|
||||
@@ -133,16 +136,43 @@ export async function scrollToElement(
|
||||
// Settle
|
||||
await sleep(randRange(cfg.scroll_settle_delay));
|
||||
|
||||
box = await getElementBox(page, selector);
|
||||
if (!box) throw new Error(`Element lost after scrolling: ${selector}`);
|
||||
box = await getBox();
|
||||
if (!box) throw new Error('Element lost after scrolling into view');
|
||||
|
||||
return { box, cursorX, cursorY };
|
||||
}
|
||||
|
||||
async function getElementBox(page: Page, selector: string): Promise<ElementBounds | null> {
|
||||
/**
|
||||
* Selector-based humanized scroll.
|
||||
*
|
||||
* ``timeout`` is forwarded to Playwright's ``boundingBox({ timeout })`` so
|
||||
* callers like ``page.click('#x', { timeout: 5000 })`` can wait longer for
|
||||
* slow-loading elements (#172). Default matches Playwright's 30000ms when not specified.
|
||||
*/
|
||||
export async function scrollToElement(
|
||||
page: Page,
|
||||
raw: RawMouse,
|
||||
selector: string,
|
||||
cursorX: number,
|
||||
cursorY: number,
|
||||
cfg: HumanConfig,
|
||||
timeout?: number,
|
||||
): Promise<{ box: ElementBounds; cursorX: number; cursorY: number }> {
|
||||
return humanScrollIntoView(
|
||||
page, raw,
|
||||
() => getElementBox(page, selector, timeout),
|
||||
cursorX, cursorY, cfg,
|
||||
);
|
||||
}
|
||||
|
||||
async function getElementBox(
|
||||
page: Page,
|
||||
selector: string,
|
||||
timeout: number = 30000,
|
||||
): Promise<ElementBounds | null> {
|
||||
const el = page.locator(selector).first();
|
||||
try {
|
||||
const box = await el.boundingBox({ timeout: 2000 });
|
||||
const box = await el.boundingBox({ timeout });
|
||||
return box;
|
||||
} catch {
|
||||
return null;
|
||||
|
||||
+43
-16
@@ -3,12 +3,12 @@
|
||||
* Mirrors Python cloakbrowser/browser.py.
|
||||
*/
|
||||
|
||||
import type { Browser, BrowserContext } from "playwright-core";
|
||||
import type { Browser, BrowserContext, BrowserContextOptions } from "playwright-core";
|
||||
import type { LaunchOptions, LaunchContextOptions, LaunchPersistentContextOptions } from "./types.js";
|
||||
import { DEFAULT_VIEWPORT, IGNORE_DEFAULT_ARGS } from "./config.js";
|
||||
import { buildArgs } from "./args.js";
|
||||
import { ensureBinary } from "./download.js";
|
||||
import { parseProxyUrl } from "./proxy.js";
|
||||
import { resolveProxyConfig } from "./proxy.js";
|
||||
import { maybeResolveGeoip, resolveWebrtcArgs } from "./geoip.js";
|
||||
|
||||
/** @internal Accept both timezone and timezoneId — either works, no warning. Exported for testing. */
|
||||
@@ -21,6 +21,29 @@ export function resolveTimezone<T extends { timezone?: string; timezoneId?: stri
|
||||
return options;
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip `locale` and `timezoneId` from user-provided contextOptions — both route
|
||||
* through detectable CDP emulation. The wrapper's top-level `locale`/`timezone`
|
||||
* fields use binary flags instead (undetectable). Warn so users notice.
|
||||
*/
|
||||
function filterStealthCtxOptions(ctx?: BrowserContextOptions): Partial<BrowserContextOptions> {
|
||||
if (!ctx) return {};
|
||||
const { locale, timezoneId, ...rest } = ctx;
|
||||
if (locale !== undefined) {
|
||||
console.warn(
|
||||
"[cloakbrowser] contextOptions.locale ignored — use top-level `locale` " +
|
||||
"instead (routes through binary flag, avoids detectable CDP emulation)."
|
||||
);
|
||||
}
|
||||
if (timezoneId !== undefined) {
|
||||
console.warn(
|
||||
"[cloakbrowser] contextOptions.timezoneId ignored — use top-level `timezone` " +
|
||||
"instead (routes through binary flag, avoids detectable CDP emulation)."
|
||||
);
|
||||
}
|
||||
return rest;
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch stealth Chromium browser via Playwright.
|
||||
*
|
||||
@@ -39,20 +62,19 @@ export async function launch(options: LaunchOptions = {}): Promise<Browser> {
|
||||
|
||||
const binaryPath = process.env.CLOAKBROWSER_BINARY_PATH || (await ensureBinary());
|
||||
const { exitIp, ...resolved } = await maybeResolveGeoip(options);
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig(options.proxy);
|
||||
let resolvedArgs = await resolveWebrtcArgs(options);
|
||||
if (exitIp && !(resolvedArgs ?? []).some(a => a.startsWith("--fingerprint-webrtc-ip"))) {
|
||||
resolvedArgs = [...(resolvedArgs ?? []), `--fingerprint-webrtc-ip=${exitIp}`];
|
||||
}
|
||||
const args = buildArgs({ ...options, ...resolved, args: resolvedArgs });
|
||||
const args = buildArgs({ ...options, ...resolved, args: [...(resolvedArgs ?? []), ...proxyArgs] });
|
||||
|
||||
const browser = await chromium.launch({
|
||||
executablePath: binaryPath,
|
||||
headless: options.headless ?? true,
|
||||
args,
|
||||
ignoreDefaultArgs: IGNORE_DEFAULT_ARGS,
|
||||
...(options.proxy
|
||||
? { proxy: typeof options.proxy === "string" ? parseProxyUrl(options.proxy) : options.proxy }
|
||||
: {}),
|
||||
...(proxyOption ? { proxy: proxyOption } : {}),
|
||||
...options.launchOptions,
|
||||
});
|
||||
|
||||
@@ -61,8 +83,8 @@ export async function launch(options: LaunchOptions = {}): Promise<Browser> {
|
||||
const { patchBrowser } = await import('./human/index.js');
|
||||
const { resolveConfig } = await import('./human/config.js');
|
||||
const cfg = resolveConfig(
|
||||
(options.humanPreset as any) ?? 'default',
|
||||
options.humanConfig as any,
|
||||
options.humanPreset ?? 'default',
|
||||
options.humanConfig,
|
||||
);
|
||||
patchBrowser(browser, cfg);
|
||||
}
|
||||
@@ -105,6 +127,9 @@ export async function launchContext(
|
||||
let context: BrowserContext;
|
||||
try {
|
||||
context = await browser.newContext({
|
||||
// contextOptions first — explicit wrapper fields below override it.
|
||||
// filterStealthCtxOptions strips locale/timezoneId to prevent CDP detection.
|
||||
...filterStealthCtxOptions(options.contextOptions),
|
||||
...(options.userAgent ? { userAgent: options.userAgent } : {}),
|
||||
viewport: options.viewport === undefined ? DEFAULT_VIEWPORT : options.viewport,
|
||||
...(options.colorScheme ? { colorScheme: options.colorScheme } : {}),
|
||||
@@ -126,8 +151,8 @@ export async function launchContext(
|
||||
const { patchContext } = await import('./human/index.js');
|
||||
const { resolveConfig } = await import('./human/config.js');
|
||||
const cfg = resolveConfig(
|
||||
(options.humanPreset as any) ?? 'default',
|
||||
options.humanConfig as any,
|
||||
options.humanPreset ?? 'default',
|
||||
options.humanConfig,
|
||||
);
|
||||
patchContext(context, cfg);
|
||||
}
|
||||
@@ -164,11 +189,12 @@ export async function launchPersistentContext(
|
||||
|
||||
const binaryPath = process.env.CLOAKBROWSER_BINARY_PATH || (await ensureBinary());
|
||||
const { exitIp, ...resolved } = await maybeResolveGeoip(options);
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig(options.proxy);
|
||||
let resolvedArgs = await resolveWebrtcArgs(options);
|
||||
if (exitIp && !(resolvedArgs ?? []).some(a => a.startsWith("--fingerprint-webrtc-ip"))) {
|
||||
resolvedArgs = [...(resolvedArgs ?? []), `--fingerprint-webrtc-ip=${exitIp}`];
|
||||
}
|
||||
const args = buildArgs({ ...options, ...resolved, args: resolvedArgs });
|
||||
const args = buildArgs({ ...options, ...resolved, args: [...(resolvedArgs ?? []), ...proxyArgs] });
|
||||
|
||||
// locale and timezone are set via binary flags (--lang, --fingerprint-timezone)
|
||||
// — NOT via Playwright context kwargs which use detectable CDP emulation.
|
||||
@@ -177,9 +203,10 @@ export async function launchPersistentContext(
|
||||
headless: options.headless ?? true,
|
||||
args,
|
||||
ignoreDefaultArgs: IGNORE_DEFAULT_ARGS,
|
||||
...(options.proxy
|
||||
? { proxy: typeof options.proxy === "string" ? parseProxyUrl(options.proxy) : options.proxy }
|
||||
: {}),
|
||||
...(proxyOption ? { proxy: proxyOption } : {}),
|
||||
// contextOptions before explicit wrapper fields so explicit wins.
|
||||
// filterStealthCtxOptions strips locale/timezoneId to prevent CDP detection.
|
||||
...filterStealthCtxOptions(options.contextOptions),
|
||||
...(options.userAgent ? { userAgent: options.userAgent } : {}),
|
||||
viewport: options.viewport === undefined ? DEFAULT_VIEWPORT : options.viewport,
|
||||
...(options.colorScheme ? { colorScheme: options.colorScheme } : {}),
|
||||
@@ -191,8 +218,8 @@ export async function launchPersistentContext(
|
||||
const { patchContext } = await import('./human/index.js');
|
||||
const { resolveConfig } = await import('./human/config.js');
|
||||
const cfg = resolveConfig(
|
||||
(options.humanPreset as any) ?? 'default',
|
||||
options.humanConfig as any,
|
||||
options.humanPreset ?? 'default',
|
||||
options.humanConfig,
|
||||
);
|
||||
patchContext(context, cfg);
|
||||
}
|
||||
|
||||
+161
@@ -23,6 +23,167 @@ export function ensureProxyScheme(proxyUrl: string): string {
|
||||
* Also handles: no credentials, URL-encoded special chars, socks5://, missing port,
|
||||
* and bare proxy strings without a scheme (e.g. "user:pass@host:port" -> treated as http).
|
||||
*/
|
||||
/** Proxy dict shape accepted by Playwright/Puppeteer wrappers. */
|
||||
export type ProxyDict = { server: string; bypass?: string; username?: string; password?: string };
|
||||
|
||||
/** Result of resolveProxyConfig — either Playwright dict OR Chrome arg, never both. */
|
||||
export interface ProxyConfig {
|
||||
/** Playwright proxy option (for HTTP proxies). */
|
||||
proxyOption?: ParsedProxy;
|
||||
/** Chrome CLI args (for SOCKS5 proxies, e.g. ["--proxy-server=socks5://..."]). */
|
||||
proxyArgs: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a proxy uses the SOCKS5 protocol.
|
||||
*/
|
||||
export function isSocksProxy(proxy: string | ProxyDict | undefined | null): boolean {
|
||||
if (!proxy) return false;
|
||||
const url = typeof proxy === "string" ? proxy : proxy.server;
|
||||
return /^socks5h?:\/\//i.test(url);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a SOCKS URL from already-percent-encoded credentials and a host suffix.
|
||||
*
|
||||
* `encPass === null` means no password (no colon in userinfo). Empty string
|
||||
* means present-but-empty (colon preserved).
|
||||
*/
|
||||
function assembleSocksUrl(
|
||||
scheme: string,
|
||||
encUser: string,
|
||||
encPass: string | null,
|
||||
hostAndRest: string,
|
||||
): string {
|
||||
let userinfo: string;
|
||||
if (encPass !== null) {
|
||||
userinfo = `${encUser}:${encPass}@`;
|
||||
} else if (encUser) {
|
||||
userinfo = `${encUser}@`;
|
||||
} else {
|
||||
userinfo = "";
|
||||
}
|
||||
return `${scheme}://${userinfo}${hostAndRest}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lenient percent-decode that handles malformed escapes gracefully, matching
|
||||
* Python's ``urllib.parse.unquote``: valid ``%XX`` sequences are decoded,
|
||||
* bare ``%`` not followed by two hex digits is left as a literal ``%``.
|
||||
*/
|
||||
function lenientDecodeURIComponent(s: string): string {
|
||||
return s.replace(/%([0-9A-Fa-f]{2})|%/g, (match, hex) =>
|
||||
hex ? String.fromCharCode(parseInt(hex, 16)) : "%",
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconstruct a SOCKS5 URL with inline credentials from a proxy dict.
|
||||
*/
|
||||
export function reconstructSocksUrl(proxy: ProxyDict): string {
|
||||
const url = new URL(proxy.server);
|
||||
if (proxy.username) {
|
||||
url.username = encodeURIComponent(proxy.username);
|
||||
if (proxy.password) url.password = encodeURIComponent(proxy.password);
|
||||
}
|
||||
return url.href.replace(/\/$/, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-encode credentials in a SOCKS5 URL string so Chromium's parser doesn't
|
||||
* truncate them at special chars like '='. Idempotent: pre-encoded input stays
|
||||
* the same (decoded then re-encoded).
|
||||
*
|
||||
* Parsing is done manually rather than via `new URL` + setters, because WHATWG
|
||||
* URL's username/password setters re-encode `%` on assignment, causing
|
||||
* double-encoding when we round-trip decode-then-encode.
|
||||
*
|
||||
* On any unexpected failure, logs a warning and returns the original string
|
||||
* so Chromium's own error handling can surface the real problem.
|
||||
*/
|
||||
export function normalizeSocksStringUrl(urlStr: string): string {
|
||||
// Split userinfo from host at the LAST '@' (RFC 3986), so a raw '@' inside
|
||||
// a password like `socks5://user:p@ss@host:1080` parses correctly. Matches
|
||||
// Python urlparse's rpartition('@') behavior.
|
||||
const schemeMatch = urlStr.match(/^([a-z][a-z0-9+\-.]*):\/\/(.*)$/i);
|
||||
if (!schemeMatch) return urlStr;
|
||||
const [, scheme, rest] = schemeMatch;
|
||||
const hostStart = rest.search(/[/?#]/);
|
||||
const authority = hostStart === -1 ? rest : rest.slice(0, hostStart);
|
||||
const suffix = hostStart === -1 ? "" : rest.slice(hostStart);
|
||||
const atIdx = authority.lastIndexOf("@");
|
||||
if (atIdx === -1) return urlStr; // no creds
|
||||
const userinfo = authority.slice(0, atIdx);
|
||||
const hostPart = authority.slice(atIdx + 1);
|
||||
// Validate port (matches Python's urlparse().port ValueError guard).
|
||||
// Extract port after last ':' — but skip IPv6 brackets (e.g. [::1]:1080).
|
||||
const bracketEnd = hostPart.lastIndexOf("]");
|
||||
const portColonIdx = hostPart.indexOf(":", Math.max(bracketEnd, 0));
|
||||
if (portColonIdx !== -1) {
|
||||
const portStr = hostPart.slice(portColonIdx + 1);
|
||||
if (portStr && !/^\d+$/.test(portStr)) {
|
||||
console.warn(`[cloakbrowser] Malformed SOCKS5 proxy URL, passing through unchanged: invalid port`);
|
||||
return urlStr;
|
||||
}
|
||||
}
|
||||
const hostAndRest = hostPart + suffix;
|
||||
const colonIdx = userinfo.indexOf(":");
|
||||
const rawUserEnc = colonIdx === -1 ? userinfo : userinfo.slice(0, colonIdx);
|
||||
const hasPassword = colonIdx !== -1;
|
||||
const rawPassEnc = hasPassword ? userinfo.slice(colonIdx + 1) : "";
|
||||
try {
|
||||
const encUser = rawUserEnc ? encodeURIComponent(lenientDecodeURIComponent(rawUserEnc)) : "";
|
||||
const encPass = hasPassword
|
||||
? (rawPassEnc ? encodeURIComponent(lenientDecodeURIComponent(rawPassEnc)) : "")
|
||||
: null;
|
||||
const normalized = assembleSocksUrl(scheme, encUser, encPass, hostAndRest);
|
||||
// Compare credentials, not the full URL: keeps the log condition focused
|
||||
// on real encoding work, not cosmetic differences (parity with the Python
|
||||
// implementation, which has to skip urlparse's hostname lowercasing).
|
||||
const credsChanged = encUser !== rawUserEnc
|
||||
|| (hasPassword ? encPass !== rawPassEnc : false);
|
||||
if (credsChanged) {
|
||||
console.info(
|
||||
"[cloakbrowser] Auto URL-encoded SOCKS5 proxy credentials (special " +
|
||||
"characters detected). Pre-encode the URL to suppress this notice.",
|
||||
);
|
||||
}
|
||||
return normalized;
|
||||
} catch (e) {
|
||||
console.warn(`[cloakbrowser] Could not normalize SOCKS5 proxy URL, passing through unchanged: ${(e as Error).message}`);
|
||||
return urlStr;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve proxy into Playwright option and/or Chrome args.
|
||||
*
|
||||
* Playwright rejects SOCKS5 proxies with credentials in its proxy dict,
|
||||
* so SOCKS5 is passed via --proxy-server Chrome arg instead.
|
||||
*/
|
||||
export function resolveProxyConfig(proxy: string | ProxyDict | undefined): ProxyConfig {
|
||||
if (!proxy) return { proxyArgs: [] };
|
||||
|
||||
if (isSocksProxy(proxy)) {
|
||||
// SOCKS5: bypass Playwright, pass directly to Chrome via --proxy-server.
|
||||
if (typeof proxy === "string") {
|
||||
// Re-encode creds to work around Chromium parser truncating passwords
|
||||
// at '=' and other special chars (#157).
|
||||
return { proxyArgs: [`--proxy-server=${normalizeSocksStringUrl(proxy)}`] };
|
||||
}
|
||||
const socksUrl = reconstructSocksUrl(proxy);
|
||||
const args = [`--proxy-server=${socksUrl}`];
|
||||
if (proxy.bypass) args.push(`--proxy-bypass-list=${proxy.bypass}`);
|
||||
return { proxyArgs: args };
|
||||
}
|
||||
|
||||
// HTTP/HTTPS: use Playwright's proxy dict
|
||||
if (typeof proxy === "string") {
|
||||
return { proxyOption: parseProxyUrl(proxy), proxyArgs: [] };
|
||||
}
|
||||
return { proxyOption: proxy as ParsedProxy, proxyArgs: [] };
|
||||
}
|
||||
|
||||
export function parseProxyUrl(proxy: string): ParsedProxy {
|
||||
let url: URL;
|
||||
// Bare format: "user:pass@host:port" — new URL() throws without a scheme.
|
||||
|
||||
+11
-9
@@ -9,7 +9,7 @@ import type { LaunchOptions } from "./types.js";
|
||||
import { IGNORE_DEFAULT_ARGS } from "./config.js";
|
||||
import { buildArgs } from "./args.js";
|
||||
import { ensureBinary } from "./download.js";
|
||||
import { parseProxyUrl } from "./proxy.js";
|
||||
import { isSocksProxy, parseProxyUrl, resolveProxyConfig } from "./proxy.js";
|
||||
import { maybeResolveGeoip, resolveWebrtcArgs } from "./geoip.js";
|
||||
|
||||
/**
|
||||
@@ -39,25 +39,27 @@ export async function launch(options: LaunchOptions = {}): Promise<Browser> {
|
||||
const args = buildArgs({ ...options, ...resolved, args: resolvedArgs });
|
||||
|
||||
// Puppeteer handles proxy via CLI args, not a separate option.
|
||||
// Chromium's --proxy-server does NOT support inline credentials,
|
||||
// so we strip them and use page.authenticate() instead.
|
||||
// SOCKS5: Chrome supports inline credentials natively (RFC 1929 auth).
|
||||
// HTTP: Chrome does NOT support inline credentials — strip them and
|
||||
// use page.authenticate() for Proxy-Authorization headers instead.
|
||||
let proxyAuth: { username: string; password: string } | undefined;
|
||||
if (options.proxy) {
|
||||
if (typeof options.proxy === "string") {
|
||||
if (isSocksProxy(options.proxy)) {
|
||||
// SOCKS5: pass full URL with credentials to Chrome directly
|
||||
const { proxyArgs } = resolveProxyConfig(options.proxy);
|
||||
args.push(...proxyArgs);
|
||||
} else if (typeof options.proxy === "string") {
|
||||
const { server, username, password } = parseProxyUrl(options.proxy);
|
||||
args.push(`--proxy-server=${server}`);
|
||||
if (username) {
|
||||
proxyAuth = { username, password: password ?? "" };
|
||||
}
|
||||
} else {
|
||||
// Strip any inline credentials from the server URL — Chromium's
|
||||
// --proxy-server doesn't support them; use page.authenticate() instead.
|
||||
const parsed = parseProxyUrl(options.proxy.server);
|
||||
args.push(`--proxy-server=${parsed.server}`);
|
||||
if (options.proxy.bypass) {
|
||||
args.push(`--proxy-bypass-list=${options.proxy.bypass}`);
|
||||
}
|
||||
// Explicit username/password fields take precedence over inline creds
|
||||
const username = options.proxy.username ?? parsed.username;
|
||||
const password = options.proxy.password ?? parsed.password;
|
||||
if (username) {
|
||||
@@ -92,8 +94,8 @@ export async function launch(options: LaunchOptions = {}): Promise<Browser> {
|
||||
const { patchBrowser } = await import('./human-puppeteer/index.js');
|
||||
const { resolveConfig } = await import('./human/config.js');
|
||||
const cfg = resolveConfig(
|
||||
(options.humanPreset as any) ?? 'default',
|
||||
options.humanConfig as any,
|
||||
options.humanPreset ?? 'default',
|
||||
options.humanConfig,
|
||||
);
|
||||
patchBrowser(browser, cfg);
|
||||
}
|
||||
|
||||
+14
-2
@@ -2,6 +2,9 @@
|
||||
* Shared types for cloakbrowser launch wrappers.
|
||||
*/
|
||||
|
||||
import type { BrowserContextOptions } from "playwright-core";
|
||||
import type { HumanConfig, HumanPreset } from "./human/config.js";
|
||||
|
||||
export interface LaunchOptions {
|
||||
/** Run in headless mode (default: true). */
|
||||
headless?: boolean;
|
||||
@@ -27,9 +30,9 @@ export interface LaunchOptions {
|
||||
/** Enable human-like mouse, keyboard, and scroll behavior. */
|
||||
humanize?: boolean;
|
||||
/** Human behavior preset: 'default' or 'careful'. */
|
||||
humanPreset?: 'default' | 'careful';
|
||||
humanPreset?: HumanPreset;
|
||||
/** Override individual human behavior parameters. */
|
||||
humanConfig?: Record<string, unknown>;
|
||||
humanConfig?: Partial<HumanConfig>;
|
||||
}
|
||||
|
||||
export interface LaunchContextOptions extends LaunchOptions {
|
||||
@@ -43,6 +46,15 @@ export interface LaunchContextOptions extends LaunchOptions {
|
||||
timezoneId?: string;
|
||||
/** Color scheme preference — 'light', 'dark', or 'no-preference'. */
|
||||
colorScheme?: "light" | "dark" | "no-preference";
|
||||
/**
|
||||
* Extra options forwarded directly to Playwright's `browser.newContext()` —
|
||||
* e.g. `storageState`, `permissions`, `geolocation`, `extraHTTPHeaders`,
|
||||
* `httpCredentials`. Use this for context-level options not surfaced as
|
||||
* top-level fields. `locale` and `timezoneId` are stripped here to avoid
|
||||
* detectable CDP emulation — use the top-level `locale` and `timezone`
|
||||
* wrapper fields instead (they route through undetectable binary flags).
|
||||
*/
|
||||
contextOptions?: BrowserContextOptions;
|
||||
}
|
||||
|
||||
export interface LaunchPersistentContextOptions extends LaunchContextOptions {
|
||||
|
||||
+59
-2
@@ -1,5 +1,17 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { COUNTRY_LOCALE_MAP, resolveProxyIp } from "../src/geoip.js";
|
||||
import { describe, it, expect, afterEach, vi } from "vitest";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { COUNTRY_LOCALE_MAP, maybeResolveGeoip, resolveProxyGeo, resolveProxyIp } from "../src/geoip.js";
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
delete process.env.CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS;
|
||||
delete process.env.CLOAKBROWSER_CACHE_DIR;
|
||||
for (const dir of tempDirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe("resolveProxyIp", () => {
|
||||
it("returns literal IPv4 from proxy URL", async () => {
|
||||
@@ -38,6 +50,51 @@ describe("resolveProxyIp", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("maybeResolveGeoip", () => {
|
||||
it("does not apply the GeoIP resolution timeout to first-use database download", async () => {
|
||||
const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cloak-geoip-download-"));
|
||||
tempDirs.push(cacheDir);
|
||||
process.env.CLOAKBROWSER_CACHE_DIR = cacheDir;
|
||||
process.env.CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS = "0.001";
|
||||
|
||||
const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue({
|
||||
ok: true,
|
||||
body: new ReadableStream({
|
||||
start(controller) {
|
||||
controller.enqueue(new Uint8Array([1, 2, 3]));
|
||||
controller.close();
|
||||
},
|
||||
}),
|
||||
} as Response);
|
||||
|
||||
const result = await resolveProxyGeo("http://203.0.113.10:8080");
|
||||
|
||||
expect(result).toEqual({ timezone: null, locale: null, exitIp: null });
|
||||
expect(fetchSpy).toHaveBeenCalledOnce();
|
||||
expect(fetchSpy.mock.calls[0][1]).toEqual({ redirect: "follow" });
|
||||
});
|
||||
|
||||
it("returns quickly when GeoIP resolution times out", async () => {
|
||||
const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cloak-geoip-timeout-"));
|
||||
tempDirs.push(cacheDir);
|
||||
process.env.CLOAKBROWSER_CACHE_DIR = cacheDir;
|
||||
process.env.CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS = "0.025";
|
||||
|
||||
const start = performance.now();
|
||||
const result = await maybeResolveGeoip({
|
||||
geoip: true,
|
||||
proxy: "http://203.0.113.10:8080",
|
||||
timezone: "Europe/Paris",
|
||||
locale: "fr-FR",
|
||||
});
|
||||
const elapsed = performance.now() - start;
|
||||
|
||||
expect(result).toEqual({ timezone: "Europe/Paris", locale: "fr-FR", exitIp: undefined });
|
||||
expect(elapsed).toBeLessThan(500);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
describe("COUNTRY_LOCALE_MAP", () => {
|
||||
it("contains common countries", () => {
|
||||
for (const code of ["US", "GB", "DE", "FR", "JP", "BR", "IL", "RU"]) {
|
||||
|
||||
+809
-71
File diff suppressed because it is too large
Load Diff
+116
-7
@@ -4,14 +4,20 @@ import { DEFAULT_VIEWPORT, getChromiumVersion } from "../src/config.js";
|
||||
|
||||
describe("binaryInfo", () => {
|
||||
it("returns correct structure", () => {
|
||||
const info = binaryInfo();
|
||||
const orig = process.env.CLOAKBROWSER_CACHE_DIR;
|
||||
process.env.CLOAKBROWSER_CACHE_DIR = `/tmp/cloakbrowser-test-${Date.now()}`;
|
||||
try {
|
||||
const info = binaryInfo();
|
||||
|
||||
expect(info.version).toBe(getChromiumVersion());
|
||||
expect(info.platform).toMatch(/^(linux|darwin|windows)-(x64|arm64)$/);
|
||||
expect(info.binaryPath).toBeTruthy();
|
||||
expect(typeof info.installed).toBe("boolean");
|
||||
expect(info.cacheDir).toContain("cloakbrowser");
|
||||
expect(info.downloadUrl).toContain(".tar.gz");
|
||||
expect(info.version).toBe(getChromiumVersion());
|
||||
expect(info.platform).toMatch(/^(linux|darwin|windows)-(x64|arm64)$/);
|
||||
expect(info.binaryPath).toBeTruthy();
|
||||
expect(typeof info.installed).toBe("boolean");
|
||||
expect(info.cacheDir).toContain("cloakbrowser");
|
||||
} finally {
|
||||
if (orig) process.env.CLOAKBROWSER_CACHE_DIR = orig;
|
||||
else delete process.env.CLOAKBROWSER_CACHE_DIR;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -130,6 +136,60 @@ describe("launchContext (unit)", () => {
|
||||
// Browser also closed
|
||||
expect(mockBrowser.close).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("forwards contextOptions to newContext (storageState, etc.)", async () => {
|
||||
const { launchContext } = await import("../src/playwright.js");
|
||||
await launchContext({
|
||||
contextOptions: {
|
||||
storageState: "state.json",
|
||||
permissions: ["geolocation"],
|
||||
},
|
||||
});
|
||||
|
||||
const ctxArgs = mockBrowser.newContext.mock.calls[0][0];
|
||||
expect(ctxArgs.storageState).toBe("state.json");
|
||||
expect(ctxArgs.permissions).toEqual(["geolocation"]);
|
||||
});
|
||||
|
||||
it("explicit top-level fields win over contextOptions on collision", async () => {
|
||||
const { launchContext } = await import("../src/playwright.js");
|
||||
await launchContext({
|
||||
userAgent: "Explicit/1.0",
|
||||
viewport: { width: 1280, height: 720 },
|
||||
colorScheme: "dark",
|
||||
contextOptions: {
|
||||
userAgent: "ShouldBeOverridden/9.9",
|
||||
viewport: { width: 9999, height: 9999 },
|
||||
colorScheme: "light",
|
||||
},
|
||||
});
|
||||
|
||||
const ctxArgs = mockBrowser.newContext.mock.calls[0][0];
|
||||
expect(ctxArgs.userAgent).toBe("Explicit/1.0");
|
||||
expect(ctxArgs.viewport).toEqual({ width: 1280, height: 720 });
|
||||
expect(ctxArgs.colorScheme).toBe("dark");
|
||||
});
|
||||
|
||||
it("strips locale and timezoneId from contextOptions (stealth-sensitive)", async () => {
|
||||
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||
const { launchContext } = await import("../src/playwright.js");
|
||||
await launchContext({
|
||||
contextOptions: {
|
||||
storageState: "state.json",
|
||||
locale: "de-DE",
|
||||
timezoneId: "Europe/Berlin",
|
||||
},
|
||||
});
|
||||
|
||||
const ctxArgs = mockBrowser.newContext.mock.calls[0][0];
|
||||
// Stealth-sensitive keys stripped — they would reintroduce detectable CDP emulation.
|
||||
expect(ctxArgs.locale).toBeUndefined();
|
||||
expect(ctxArgs.timezoneId).toBeUndefined();
|
||||
// Benign keys preserved
|
||||
expect(ctxArgs.storageState).toBe("state.json");
|
||||
// Warning was logged for both stripped keys
|
||||
expect(warnSpy).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("launchPersistentContext (unit)", () => {
|
||||
@@ -207,4 +267,53 @@ describe("launchPersistentContext (unit)", () => {
|
||||
expect(args.userAgent).toBe("Custom/1.0");
|
||||
expect(args.colorScheme).toBe("dark");
|
||||
});
|
||||
|
||||
it("forwards contextOptions to launchPersistentContext", async () => {
|
||||
const { launchPersistentContext } = await import("../src/playwright.js");
|
||||
await launchPersistentContext({
|
||||
userDataDir: "/tmp/profile",
|
||||
contextOptions: {
|
||||
permissions: ["geolocation"],
|
||||
extraHTTPHeaders: { "X-Custom": "1" },
|
||||
},
|
||||
});
|
||||
|
||||
const args = mockChromium.launchPersistentContext.mock.calls[0][1];
|
||||
expect(args.permissions).toEqual(["geolocation"]);
|
||||
expect(args.extraHTTPHeaders).toEqual({ "X-Custom": "1" });
|
||||
});
|
||||
|
||||
it("explicit top-level fields win over contextOptions in persistent context", async () => {
|
||||
const { launchPersistentContext } = await import("../src/playwright.js");
|
||||
await launchPersistentContext({
|
||||
userDataDir: "/tmp/profile",
|
||||
userAgent: "Explicit/1.0",
|
||||
viewport: { width: 1280, height: 720 },
|
||||
contextOptions: {
|
||||
userAgent: "ShouldBeOverridden/9.9",
|
||||
viewport: { width: 9999, height: 9999 },
|
||||
},
|
||||
});
|
||||
|
||||
const args = mockChromium.launchPersistentContext.mock.calls[0][1];
|
||||
expect(args.userAgent).toBe("Explicit/1.0");
|
||||
expect(args.viewport).toEqual({ width: 1280, height: 720 });
|
||||
});
|
||||
|
||||
it("strips locale and timezoneId from contextOptions (persistent context)", async () => {
|
||||
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||
const { launchPersistentContext } = await import("../src/playwright.js");
|
||||
await launchPersistentContext({
|
||||
userDataDir: "/tmp/profile",
|
||||
contextOptions: {
|
||||
locale: "de-DE",
|
||||
timezoneId: "Europe/Berlin",
|
||||
},
|
||||
});
|
||||
|
||||
const args = mockChromium.launchPersistentContext.mock.calls[0][1];
|
||||
expect(args.locale).toBeUndefined();
|
||||
expect(args.timezoneId).toBeUndefined();
|
||||
expect(warnSpy).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
|
||||
+209
-2
@@ -1,5 +1,5 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { parseProxyUrl } from "../src/proxy.js";
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { parseProxyUrl, isSocksProxy, resolveProxyConfig } from "../src/proxy.js";
|
||||
import type { LaunchOptions } from "../src/types.js";
|
||||
|
||||
describe("parseProxyUrl", () => {
|
||||
@@ -115,3 +115,210 @@ describe("bare proxy format (user:pass@host:port)", () => {
|
||||
expect(parseProxyUrl("proxy:8080")).toEqual({ server: "proxy:8080" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("isSocksProxy", () => {
|
||||
it("detects socks5 string", () => {
|
||||
expect(isSocksProxy("socks5://user:pass@host:1080")).toBe(true);
|
||||
});
|
||||
|
||||
it("detects socks5h string", () => {
|
||||
expect(isSocksProxy("socks5h://host:1080")).toBe(true);
|
||||
});
|
||||
|
||||
it("case insensitive", () => {
|
||||
expect(isSocksProxy("SOCKS5://host:1080")).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects http", () => {
|
||||
expect(isSocksProxy("http://host:8080")).toBe(false);
|
||||
});
|
||||
|
||||
it("detects socks5 dict", () => {
|
||||
expect(isSocksProxy({ server: "socks5://host:1080" })).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects http dict", () => {
|
||||
expect(isSocksProxy({ server: "http://host:8080" })).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for undefined", () => {
|
||||
expect(isSocksProxy(undefined)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveProxyConfig", () => {
|
||||
it("returns empty for undefined", () => {
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig(undefined);
|
||||
expect(proxyOption).toBeUndefined();
|
||||
expect(proxyArgs).toEqual([]);
|
||||
});
|
||||
|
||||
it("returns playwright dict for http string", () => {
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig("http://user:pass@proxy:8080");
|
||||
expect(proxyOption).toEqual({ server: "http://proxy:8080", username: "user", password: "pass" });
|
||||
expect(proxyArgs).toEqual([]);
|
||||
});
|
||||
|
||||
it("returns playwright dict for http dict", () => {
|
||||
const proxy = { server: "http://proxy:8080", bypass: ".example.com" };
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig(proxy);
|
||||
expect(proxyOption).toEqual(proxy);
|
||||
expect(proxyArgs).toEqual([]);
|
||||
});
|
||||
|
||||
it("returns chrome arg for socks5 string", () => {
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig("socks5://user:pass@host:1080");
|
||||
expect(proxyOption).toBeUndefined();
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:pass@host:1080"]);
|
||||
});
|
||||
|
||||
it("returns chrome arg for socks5 no auth", () => {
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig("socks5://host:1080");
|
||||
expect(proxyOption).toBeUndefined();
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://host:1080"]);
|
||||
});
|
||||
|
||||
it("returns chrome arg for socks5h string", () => {
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig("socks5h://user:pass@host:1080");
|
||||
expect(proxyOption).toBeUndefined();
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5h://user:pass@host:1080"]);
|
||||
});
|
||||
|
||||
it("reconstructs URL from socks5 dict with auth", () => {
|
||||
const { proxyOption, proxyArgs } = resolveProxyConfig({
|
||||
server: "socks5://host:1080",
|
||||
username: "user",
|
||||
password: "p@ss",
|
||||
});
|
||||
expect(proxyOption).toBeUndefined();
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:p%40ss@host:1080"]);
|
||||
});
|
||||
|
||||
it("includes bypass for socks5 dict", () => {
|
||||
const { proxyArgs } = resolveProxyConfig({
|
||||
server: "socks5://host:1080",
|
||||
bypass: ".example.com",
|
||||
});
|
||||
expect(proxyArgs).toContain("--proxy-server=socks5://host:1080");
|
||||
expect(proxyArgs).toContain("--proxy-bypass-list=.example.com");
|
||||
});
|
||||
|
||||
// Chromium's --proxy-server parser truncates passwords at '=' (#157).
|
||||
// Wrapper must auto URL-encode before passing to Chrome.
|
||||
it("encodes '=' in socks5 string password", () => {
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://user:pass=123@host:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:pass%3D123@host:1080"]);
|
||||
});
|
||||
|
||||
it("encoding is idempotent for already-encoded socks5 string", () => {
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://user:pass%3D123@host:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:pass%3D123@host:1080"]);
|
||||
});
|
||||
|
||||
it("leaves socks5 string without creds unchanged", () => {
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://host:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://host:1080"]);
|
||||
});
|
||||
|
||||
it("encodes password even with empty username (password-only userinfo)", () => {
|
||||
// Regression: empty-username bypass would skip encoding, leaving the
|
||||
// Chromium truncation bug alive for this userinfo shape.
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://:pass=123@host:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://:pass%3D123@host:1080"]);
|
||||
});
|
||||
|
||||
it("handles literal '%' in password without throwing (malformed escape)", () => {
|
||||
// JS's decodeURIComponent throws on '%sure' (% not followed by 2 hex digits).
|
||||
// Must fall back to treating '%' as literal and percent-encoding it.
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://user:100%sure@host:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:100%25sure@host:1080"]);
|
||||
});
|
||||
|
||||
it("passes malformed SOCKS5 URLs through unchanged (no throw)", () => {
|
||||
// Broken IPv6 bracket — wrapper must not throw;
|
||||
// Chromium will surface its own error.
|
||||
const { proxyArgs: a1 } = resolveProxyConfig("socks5://user:pass@[::1");
|
||||
expect(a1).toEqual(["--proxy-server=socks5://user:pass@[::1"]);
|
||||
});
|
||||
|
||||
it("passes non-numeric port through unchanged", () => {
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://user:pass@host:abc");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:pass@host:abc"]);
|
||||
});
|
||||
|
||||
it("encodes special chars in IPv6 SOCKS5 string password", () => {
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://user:pass=eq@[::1]:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:pass%3Deq@[::1]:1080"]);
|
||||
});
|
||||
|
||||
// Regression #157: userinfo must be split at the LAST '@' (RFC 3986),
|
||||
// not the first, so raw '@' in a password parses correctly.
|
||||
it("encodes raw '@' in socks5 string password (last-@ split)", () => {
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://user:p@ss@host:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:p%40ss@host:1080"]);
|
||||
});
|
||||
|
||||
it("handles multiple raw '@' in password (splits at last)", () => {
|
||||
const { proxyArgs } = resolveProxyConfig("socks5://user:a@b@c@host:1080");
|
||||
expect(proxyArgs).toEqual(["--proxy-server=socks5://user:a%40b%40c@host:1080"]);
|
||||
});
|
||||
|
||||
// Visibility for #157: when wrapper actually rewrites the URL, surface an
|
||||
// info log so users debugging silent SOCKS5 fallback can see what happened.
|
||||
it("logs info message when SOCKS5 credentials get re-encoded", () => {
|
||||
const debugSpy = vi.spyOn(console, "info").mockImplementation(() => {});
|
||||
try {
|
||||
resolveProxyConfig("socks5://user:pass=123@host:1080");
|
||||
expect(debugSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining("Auto URL-encoded SOCKS5"),
|
||||
);
|
||||
// Credentials must not leak into the log.
|
||||
const calls = debugSpy.mock.calls.flat().join(" ");
|
||||
expect(calls).not.toContain("pass=123");
|
||||
expect(calls).not.toContain("pass%3D123");
|
||||
} finally {
|
||||
debugSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("stays silent when SOCKS5 URL is already encoded (no log spam)", () => {
|
||||
const debugSpy = vi.spyOn(console, "info").mockImplementation(() => {});
|
||||
try {
|
||||
resolveProxyConfig("socks5://user:pass%3D123@host:1080");
|
||||
const reencodedCalls = debugSpy.mock.calls
|
||||
.flat()
|
||||
.filter((arg) => typeof arg === "string" && arg.includes("Auto URL-encoded SOCKS5"));
|
||||
expect(reencodedCalls).toHaveLength(0);
|
||||
} finally {
|
||||
debugSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("stays silent when SOCKS5 URL has no credentials", () => {
|
||||
const debugSpy = vi.spyOn(console, "info").mockImplementation(() => {});
|
||||
try {
|
||||
resolveProxyConfig("socks5://host:1080");
|
||||
const reencodedCalls = debugSpy.mock.calls
|
||||
.flat()
|
||||
.filter((arg) => typeof arg === "string" && arg.includes("Auto URL-encoded SOCKS5"));
|
||||
expect(reencodedCalls).toHaveLength(0);
|
||||
} finally {
|
||||
debugSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("stays silent when only host case differs (no credential rewrite)", () => {
|
||||
// Parity with Python: log condition must track credential changes, not
|
||||
// cosmetic URL-string differences (regression for Copilot's PR #209 review).
|
||||
const debugSpy = vi.spyOn(console, "info").mockImplementation(() => {});
|
||||
try {
|
||||
resolveProxyConfig("socks5://USER:pass@HOST.com:1080");
|
||||
const reencodedCalls = debugSpy.mock.calls
|
||||
.flat()
|
||||
.filter((arg) => typeof arg === "string" && arg.includes("Auto URL-encoded SOCKS5"));
|
||||
expect(reencodedCalls).toHaveLength(0);
|
||||
} finally {
|
||||
debugSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -113,4 +113,29 @@ describe("puppeteer launch", () => {
|
||||
expect(callArgs.args).toContain("--disable-gpu");
|
||||
expect(callArgs.args).toContain("--no-first-run");
|
||||
});
|
||||
|
||||
it("keeps SOCKS5 credentials in --proxy-server URL", async () => {
|
||||
const { launch } = await import("../src/puppeteer.js");
|
||||
const browser = await launch({ proxy: "socks5://user:pass@proxy:1080" });
|
||||
|
||||
const callArgs = vi.mocked(puppeteerMock.default.launch).mock.calls[0][0];
|
||||
expect(callArgs.args).toContain("--proxy-server=socks5://user:pass@proxy:1080");
|
||||
|
||||
// Should NOT set up page.authenticate for SOCKS5
|
||||
const page = await browser.newPage();
|
||||
expect(page.authenticate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reconstructs SOCKS5 dict with auth into --proxy-server URL", async () => {
|
||||
const { launch } = await import("../src/puppeteer.js");
|
||||
const browser = await launch({
|
||||
proxy: { server: "socks5://proxy:1080", username: "user", password: "p@ss" },
|
||||
});
|
||||
|
||||
const callArgs = vi.mocked(puppeteerMock.default.launch).mock.calls[0][0];
|
||||
expect(callArgs.args).toContain("--proxy-server=socks5://user:p%40ss@proxy:1080");
|
||||
|
||||
const page = await browser.newPage();
|
||||
expect(page.authenticate).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -532,6 +532,52 @@ describe("Puppeteer: non-ASCII text avoids CDP shift path", () => {
|
||||
});
|
||||
|
||||
|
||||
// =========================================================================
|
||||
// Per-call human config override (Puppeteer page-level)
|
||||
// =========================================================================
|
||||
describe("Puppeteer: page.type accepts per-call human config override", () => {
|
||||
it("page.type forwards merged config to humanType", async () => {
|
||||
const keyboardMod = await import("../src/human-puppeteer/keyboard.js");
|
||||
const scrollMod = await import("../src/human-puppeteer/scroll.js");
|
||||
|
||||
const cfg = resolveConfig("default", {
|
||||
idle_between_actions: false,
|
||||
field_switch_delay: [0, 1],
|
||||
});
|
||||
expect(cfg.typing_delay).toBe(70);
|
||||
|
||||
let captured: any = null;
|
||||
const typeSpy = vi.spyOn(keyboardMod, "humanType").mockImplementation(
|
||||
async (_page, _raw, _text, callCfg) => { captured = callCfg; },
|
||||
);
|
||||
const scrollSpy = vi.spyOn(scrollMod, "scrollToElement").mockImplementation(
|
||||
async (_page, _raw, _sel, cx, cy) => ({
|
||||
box: { x: 100, y: 100, width: 50, height: 30 },
|
||||
cursorX: cx,
|
||||
cursorY: cy,
|
||||
}),
|
||||
);
|
||||
|
||||
const { patchPage } = await import("../src/human-puppeteer/index.js");
|
||||
const page = buildMockPage();
|
||||
const cursor = { x: 100, y: 100, initialized: true };
|
||||
patchPage(page as any, cfg, cursor as any);
|
||||
|
||||
await (page as any).type("#email", "hi", {
|
||||
typing_delay: 30,
|
||||
mistype_chance: 0,
|
||||
});
|
||||
|
||||
expect(captured.typing_delay).toBe(30);
|
||||
expect(captured.mistype_chance).toBe(0);
|
||||
expect(cfg.typing_delay).toBe(70);
|
||||
|
||||
typeSpy.mockRestore();
|
||||
scrollSpy.mockRestore();
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
// =========================================================================
|
||||
// patchPage stealth infrastructure (Puppeteer)
|
||||
// =========================================================================
|
||||
@@ -1671,11 +1717,13 @@ describe("Puppeteer: isInputElement stealth integration via patchPage", () => {
|
||||
}),
|
||||
});
|
||||
|
||||
const mockEl = buildMockElementHandle();
|
||||
const page = buildMockPage({
|
||||
evaluate: vi.fn(async (...args: any[]) => {
|
||||
evaluateCalls.push(args);
|
||||
return false;
|
||||
}),
|
||||
$: vi.fn(async () => mockEl),
|
||||
});
|
||||
page.createCDPSession = vi.fn(async () => mockCdp);
|
||||
|
||||
|
||||
@@ -320,8 +320,14 @@ describe("download fallback", () => {
|
||||
|
||||
describe("effective version", () => {
|
||||
it("returns platform version when no marker exists", () => {
|
||||
// Default behavior — no marker file in test environment
|
||||
expect(getEffectiveVersion()).toBe(getChromiumVersion());
|
||||
const orig = process.env.CLOAKBROWSER_CACHE_DIR;
|
||||
process.env.CLOAKBROWSER_CACHE_DIR = `/tmp/cloakbrowser-test-${Date.now()}`;
|
||||
try {
|
||||
expect(getEffectiveVersion()).toBe(getChromiumVersion());
|
||||
} finally {
|
||||
if (orig) process.env.CLOAKBROWSER_CACHE_DIR = orig;
|
||||
else delete process.env.CLOAKBROWSER_CACHE_DIR;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
+1
-1
@@ -54,7 +54,7 @@ dependencies = [
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
geoip = ["geoip2>=4.0"]
|
||||
geoip = ["geoip2>=4.0", "socksio>=1.0"] # socksio: SOCKS5 transport for httpx
|
||||
patchright = ["patchright>=1.40"]
|
||||
serve = ["aiohttp>=3.9", "websockets>=12.0"]
|
||||
dev = ["pytest>=7.0", "pytest-asyncio>=0.23"]
|
||||
|
||||
@@ -22,6 +22,8 @@ parse_connection_params = _mod.parse_connection_params
|
||||
parse_cli_args = _mod.parse_cli_args
|
||||
ChromePool = _mod.ChromePool
|
||||
_default_data_dir = _mod._default_data_dir
|
||||
SAFE_SEED_RE = _mod.SAFE_SEED_RE
|
||||
RESERVED_SEEDS = _mod.RESERVED_SEEDS
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -105,8 +107,10 @@ class TestParseCliArgs:
|
||||
|
||||
def test_passthrough_args(self):
|
||||
args = ["--no-sandbox", "--disable-gpu", "--fingerprint=999"]
|
||||
_, passthrough = parse_cli_args(args)
|
||||
assert passthrough == args
|
||||
config, passthrough = parse_cli_args(args)
|
||||
# --fingerprint=999 is consumed into config["default_seed"], not passed through
|
||||
assert passthrough == ["--no-sandbox", "--disable-gpu"]
|
||||
assert config["default_seed"] == "999"
|
||||
|
||||
def test_port_not_in_passthrough(self):
|
||||
_, passthrough = parse_cli_args(["--port=9222", "--no-sandbox"])
|
||||
@@ -242,3 +246,95 @@ class TestConnectionTracking:
|
||||
pool.disconnect("a")
|
||||
assert pool._connections["a"] == 1
|
||||
assert pool._connections["b"] == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Seed validation (CVE fix — path traversal via fingerprint param)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSeedValidation:
|
||||
"""Verify SAFE_SEED_RE rejects path traversal and reserved names."""
|
||||
|
||||
@pytest.mark.parametrize("seed", [
|
||||
"../foo", "../../etc", "/etc/passwd", "..", ".", "foo/bar",
|
||||
"foo\\bar", "\x00evil", "", "a" * 129,
|
||||
])
|
||||
def test_malicious_seeds_rejected(self, seed):
|
||||
assert not SAFE_SEED_RE.match(seed)
|
||||
|
||||
@pytest.mark.parametrize("seed", [
|
||||
"__default__",
|
||||
])
|
||||
def test_reserved_seeds_rejected(self, seed):
|
||||
assert seed in RESERVED_SEEDS
|
||||
|
||||
@pytest.mark.parametrize("seed", [
|
||||
"12345", "my-seed_01", "ABC", "a" * 128, "0", "test-seed",
|
||||
])
|
||||
def test_valid_seeds_accepted(self, seed):
|
||||
assert SAFE_SEED_RE.match(seed)
|
||||
assert seed not in RESERVED_SEEDS
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Path containment (_safe_rmtree)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSafeRmtree:
|
||||
"""Verify _safe_rmtree refuses to delete outside data_dir."""
|
||||
|
||||
def _make_pool(self, data_dir: str):
|
||||
return ChromePool(
|
||||
binary="/fake/chrome",
|
||||
global_args=[],
|
||||
headless=True,
|
||||
data_dir=data_dir,
|
||||
)
|
||||
|
||||
def test_refuses_path_outside_data_dir(self, tmp_path):
|
||||
data_dir = tmp_path / "profiles"
|
||||
data_dir.mkdir()
|
||||
victim = tmp_path / "victim"
|
||||
victim.mkdir()
|
||||
(victim / "sentinel").touch()
|
||||
|
||||
pool = self._make_pool(str(data_dir))
|
||||
pool._safe_rmtree(str(victim))
|
||||
|
||||
assert victim.exists(), "Directory outside data_dir must not be deleted"
|
||||
|
||||
def test_refuses_data_dir_itself(self, tmp_path):
|
||||
data_dir = tmp_path / "profiles"
|
||||
data_dir.mkdir()
|
||||
(data_dir / "sentinel").touch()
|
||||
|
||||
pool = self._make_pool(str(data_dir))
|
||||
pool._safe_rmtree(str(data_dir))
|
||||
|
||||
assert data_dir.exists(), "data_dir itself must not be deleted"
|
||||
|
||||
def test_deletes_valid_subdirectory(self, tmp_path):
|
||||
data_dir = tmp_path / "profiles"
|
||||
data_dir.mkdir()
|
||||
subdir = data_dir / "seed-12345"
|
||||
subdir.mkdir()
|
||||
(subdir / "data").touch()
|
||||
|
||||
pool = self._make_pool(str(data_dir))
|
||||
pool._safe_rmtree(str(subdir))
|
||||
|
||||
assert not subdir.exists(), "Valid subdirectory should be deleted"
|
||||
|
||||
def test_refuses_traversal_path(self, tmp_path):
|
||||
data_dir = tmp_path / "profiles"
|
||||
data_dir.mkdir()
|
||||
victim = tmp_path / "victim"
|
||||
victim.mkdir()
|
||||
|
||||
traversal = str(data_dir / ".." / "victim")
|
||||
pool = self._make_pool(str(data_dir))
|
||||
pool._safe_rmtree(traversal)
|
||||
|
||||
assert victim.exists(), "Traversal path must not be deleted"
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Unit tests for GeoIP-based timezone/locale detection."""
|
||||
|
||||
from unittest.mock import patch
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -144,6 +145,20 @@ def test_maybe_resolve_fills_both():
|
||||
assert ip == "5.6.7.8"
|
||||
|
||||
|
||||
def test_maybe_resolve_geoip_timeout_returns_existing_values(monkeypatch):
|
||||
"""A stalled proxy lookup should not block launch indefinitely."""
|
||||
mock_geoip2 = type("module", (), {"database": type("db", (), {"Reader": None})})()
|
||||
monkeypatch.setenv("CLOAKBROWSER_GEOIP_TIMEOUT_SECONDS", "0.05")
|
||||
with patch.dict("sys.modules", {"geoip2": mock_geoip2, "geoip2.database": mock_geoip2.database}):
|
||||
with patch("cloakbrowser.geoip._ensure_geoip_db", return_value=object()):
|
||||
start = time.monotonic()
|
||||
tz, loc, ip = maybe_resolve_geoip(True, "http://203.0.113.10:8080", None, "fr-FR")
|
||||
elapsed = time.monotonic() - start
|
||||
|
||||
assert (tz, loc, ip) == (None, "fr-FR", None)
|
||||
assert elapsed < 0.5
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _is_private_ip
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -279,6 +279,67 @@ if __name__ == "__main__":
|
||||
check("keyboard.type", kb_ms > 500, f"{kb_ms} ms")
|
||||
time.sleep(1)
|
||||
|
||||
# ============================================================
|
||||
# SCENARIO 7: ElementHandle — query_selector interactions
|
||||
# ============================================================
|
||||
step("ElementHandle — query_selector click, type, fill, hover")
|
||||
page.goto('https://www.wikipedia.org', wait_until='domcontentloaded')
|
||||
time.sleep(2)
|
||||
inject(page)
|
||||
time.sleep(1)
|
||||
|
||||
print(" Watch: get element via query_selector, cursor moves smoothly")
|
||||
el = page.query_selector('#searchInput')
|
||||
assert el is not None, "query_selector returned None"
|
||||
assert getattr(el, '_human_patched', False), "ElementHandle not patched!"
|
||||
|
||||
t0 = time.time()
|
||||
el.click()
|
||||
eh_click_ms = int((time.time() - t0) * 1000)
|
||||
check("ElementHandle click", eh_click_ms > 100, f"{eh_click_ms} ms")
|
||||
time.sleep(0.5)
|
||||
|
||||
print(" Watch: ElementHandle type — characters appear one by one")
|
||||
t0 = time.time()
|
||||
el.type('ElementHandle typing')
|
||||
eh_type_ms = int((time.time() - t0) * 1000)
|
||||
val = page.locator('#searchInput').input_value()
|
||||
check("ElementHandle type", val == 'ElementHandle typing' and eh_type_ms > 1500, f"{eh_type_ms} ms, value='{val}'")
|
||||
time.sleep(0.5)
|
||||
|
||||
print(" Watch: ElementHandle fill — clears then types")
|
||||
t0 = time.time()
|
||||
el.fill('Filled via EH')
|
||||
eh_fill_ms = int((time.time() - t0) * 1000)
|
||||
val = page.locator('#searchInput').input_value()
|
||||
check("ElementHandle fill", val == 'Filled via EH' and eh_fill_ms > 1000, f"{eh_fill_ms} ms, value='{val}'")
|
||||
time.sleep(0.5)
|
||||
|
||||
print(" Watch: ElementHandle hover — cursor moves without clicking")
|
||||
btn_el = page.query_selector('button[type="submit"]')
|
||||
t0 = time.time()
|
||||
btn_el.hover()
|
||||
eh_hover_ms = int((time.time() - t0) * 1000)
|
||||
check("ElementHandle hover", eh_hover_ms > 50, f"{eh_hover_ms} ms")
|
||||
time.sleep(0.5)
|
||||
|
||||
print(" Watch: query_selector_all returns patched handles")
|
||||
page.goto('https://the-internet.herokuapp.com/checkboxes', wait_until='domcontentloaded')
|
||||
time.sleep(2)
|
||||
inject(page)
|
||||
time.sleep(1)
|
||||
els = page.query_selector_all('input[type="checkbox"]')
|
||||
all_patched = all(getattr(e, '_human_patched', False) for e in els)
|
||||
check("query_selector_all all patched", all_patched and len(els) >= 2, f"{len(els)} elements, all_patched={all_patched}")
|
||||
|
||||
if els:
|
||||
print(" Watch: click checkbox via ElementHandle")
|
||||
t0 = time.time()
|
||||
els[0].click()
|
||||
cb_click_ms = int((time.time() - t0) * 1000)
|
||||
check("ElementHandle checkbox click", cb_click_ms > 100, f"{cb_click_ms} ms")
|
||||
time.sleep(1)
|
||||
|
||||
# ============================================================
|
||||
# SUMMARY
|
||||
# ============================================================
|
||||
|
||||
+1087
-26
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,6 @@
|
||||
"""Unit tests for launch_context() — context kwargs, viewport defaults, close cleanup."""
|
||||
|
||||
from unittest.mock import MagicMock, call, patch
|
||||
from unittest.mock import AsyncMock, MagicMock, call, patch
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -207,3 +207,125 @@ def test_kwargs_passthrough(mock_launch, _mock_bin):
|
||||
# Verify kwarg did NOT leak to launch()
|
||||
launch_kwargs = mock_launch.call_args[1]
|
||||
assert "record_video_dir" not in launch_kwargs
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Async: launch_context_async()
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_mock_async_browser():
|
||||
"""Create a mock async browser whose new_context() returns a mock context."""
|
||||
browser = AsyncMock()
|
||||
context = AsyncMock()
|
||||
browser.new_context.return_value = context
|
||||
return browser, context
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("cloakbrowser.browser.ensure_binary", return_value="/fake/chrome")
|
||||
@patch("cloakbrowser.browser.launch_async")
|
||||
async def test_async_storage_state_forwarded(mock_launch_async, _mock_bin):
|
||||
"""storage_state kwarg forwarded to browser.new_context() in async path.
|
||||
|
||||
This is the motivating use case from issue #141.
|
||||
"""
|
||||
browser, context = _make_mock_async_browser()
|
||||
mock_launch_async.return_value = browser
|
||||
|
||||
from cloakbrowser.browser import launch_context_async
|
||||
await launch_context_async(storage_state="state.json")
|
||||
|
||||
ctx_kwargs = browser.new_context.call_args
|
||||
assert ctx_kwargs[1]["storage_state"] == "state.json"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("cloakbrowser.browser.ensure_binary", return_value="/fake/chrome")
|
||||
@patch("cloakbrowser.browser.launch_async")
|
||||
async def test_async_default_viewport(mock_launch_async, _mock_bin):
|
||||
"""DEFAULT_VIEWPORT applied when no viewport given (async)."""
|
||||
browser, context = _make_mock_async_browser()
|
||||
mock_launch_async.return_value = browser
|
||||
|
||||
from cloakbrowser.browser import launch_context_async
|
||||
await launch_context_async()
|
||||
|
||||
ctx_kwargs = browser.new_context.call_args
|
||||
assert ctx_kwargs[1]["viewport"] == DEFAULT_VIEWPORT
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("cloakbrowser.browser.ensure_binary", return_value="/fake/chrome")
|
||||
@patch("cloakbrowser.browser.launch_async")
|
||||
async def test_async_locale_flows_to_binary_not_cdp(mock_launch_async, _mock_bin):
|
||||
"""locale flows to launch_async() for --lang flag, NOT to new_context() CDP."""
|
||||
browser, context = _make_mock_async_browser()
|
||||
mock_launch_async.return_value = browser
|
||||
|
||||
from cloakbrowser.browser import launch_context_async
|
||||
await launch_context_async(locale="de-DE", timezone="Europe/Berlin")
|
||||
|
||||
# Binary flags
|
||||
assert mock_launch_async.call_args[1]["locale"] == "de-DE"
|
||||
assert mock_launch_async.call_args[1]["timezone"] == "Europe/Berlin"
|
||||
# Not in context — would trigger detectable CDP emulation
|
||||
ctx_kwargs = browser.new_context.call_args
|
||||
assert "locale" not in ctx_kwargs[1]
|
||||
assert "timezone_id" not in ctx_kwargs[1]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("cloakbrowser.browser.ensure_binary", return_value="/fake/chrome")
|
||||
@patch("cloakbrowser.browser.launch_async")
|
||||
async def test_async_close_closes_browser(mock_launch_async, _mock_bin):
|
||||
"""await ctx.close() also closes the underlying browser."""
|
||||
browser, context = _make_mock_async_browser()
|
||||
original_ctx_close = context.close
|
||||
mock_launch_async.return_value = browser
|
||||
|
||||
from cloakbrowser.browser import launch_context_async
|
||||
ctx = await launch_context_async()
|
||||
|
||||
await ctx.close()
|
||||
original_ctx_close.assert_called_once()
|
||||
browser.close.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("cloakbrowser.browser.ensure_binary", return_value="/fake/chrome")
|
||||
@patch("cloakbrowser.browser.launch_async")
|
||||
async def test_async_error_closes_browser(mock_launch_async, _mock_bin):
|
||||
"""If new_context() raises in async path, browser is still closed."""
|
||||
browser = AsyncMock()
|
||||
browser.new_context.side_effect = RuntimeError("context creation failed")
|
||||
mock_launch_async.return_value = browser
|
||||
|
||||
from cloakbrowser.browser import launch_context_async
|
||||
with pytest.raises(RuntimeError, match="context creation failed"):
|
||||
await launch_context_async()
|
||||
|
||||
browser.close.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("cloakbrowser.browser.ensure_binary", return_value="/fake/chrome")
|
||||
@patch("cloakbrowser.browser.launch_async")
|
||||
async def test_async_cancellation_closes_browser(mock_launch_async, _mock_bin):
|
||||
"""asyncio.CancelledError during new_context() still closes browser.
|
||||
|
||||
CancelledError derives from BaseException (not Exception) in Python 3.8+,
|
||||
so the cleanup must catch BaseException to prevent browser process leaks
|
||||
when the awaiting task is cancelled.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
browser = AsyncMock()
|
||||
browser.new_context.side_effect = asyncio.CancelledError()
|
||||
mock_launch_async.return_value = browser
|
||||
|
||||
from cloakbrowser.browser import launch_context_async
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await launch_context_async()
|
||||
|
||||
browser.close.assert_called_once()
|
||||
|
||||
+235
-15
@@ -2,7 +2,12 @@
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
from cloakbrowser.browser import _build_proxy_kwargs, maybe_resolve_geoip, _parse_proxy_url
|
||||
from cloakbrowser.browser import (
|
||||
_is_socks_proxy,
|
||||
_parse_proxy_url,
|
||||
_resolve_proxy_config,
|
||||
maybe_resolve_geoip,
|
||||
)
|
||||
|
||||
|
||||
class TestParseProxyUrl:
|
||||
@@ -38,23 +43,30 @@ class TestParseProxyUrl:
|
||||
|
||||
|
||||
class TestBuildProxyKwargs:
|
||||
"""Tests for _resolve_proxy_config (formerly _build_proxy_kwargs) HTTP path."""
|
||||
|
||||
def test_none(self):
|
||||
assert _build_proxy_kwargs(None) == {}
|
||||
kwargs, args = _resolve_proxy_config(None)
|
||||
assert kwargs == {}
|
||||
assert args == []
|
||||
|
||||
def test_simple_proxy(self):
|
||||
result = _build_proxy_kwargs("http://proxy:8080")
|
||||
assert result == {"proxy": {"server": "http://proxy:8080"}}
|
||||
kwargs, args = _resolve_proxy_config("http://proxy:8080")
|
||||
assert kwargs == {"proxy": {"server": "http://proxy:8080"}}
|
||||
assert args == []
|
||||
|
||||
def test_proxy_with_auth(self):
|
||||
result = _build_proxy_kwargs("http://user:pass@proxy:8080")
|
||||
assert result == {
|
||||
kwargs, args = _resolve_proxy_config("http://user:pass@proxy:8080")
|
||||
assert kwargs == {
|
||||
"proxy": {"server": "http://proxy:8080", "username": "user", "password": "pass"}
|
||||
}
|
||||
assert args == []
|
||||
|
||||
def test_proxy_dict_passthrough(self):
|
||||
proxy_dict = {"server": "http://proxy:8080", "bypass": ".google.com,localhost"}
|
||||
result = _build_proxy_kwargs(proxy_dict)
|
||||
assert result == {"proxy": proxy_dict}
|
||||
kwargs, args = _resolve_proxy_config(proxy_dict)
|
||||
assert kwargs == {"proxy": proxy_dict}
|
||||
assert args == []
|
||||
|
||||
def test_proxy_dict_with_auth(self):
|
||||
proxy_dict = {
|
||||
@@ -63,8 +75,9 @@ class TestBuildProxyKwargs:
|
||||
"password": "pass",
|
||||
"bypass": ".example.com",
|
||||
}
|
||||
result = _build_proxy_kwargs(proxy_dict)
|
||||
assert result == {"proxy": proxy_dict}
|
||||
kwargs, args = _resolve_proxy_config(proxy_dict)
|
||||
assert kwargs == {"proxy": proxy_dict}
|
||||
assert args == []
|
||||
|
||||
|
||||
class TestMaybeResolveGeoip:
|
||||
@@ -117,6 +130,27 @@ class TestMaybeResolveGeoip:
|
||||
mock_geo.assert_called_once_with("http://proxy:8080")
|
||||
assert tz == "America/New_York"
|
||||
|
||||
@patch("cloakbrowser.geoip.resolve_proxy_geo_with_ip", return_value=("Europe/Berlin", "de-DE", "5.6.7.8"))
|
||||
def test_geoip_socks5_dict_reconstructs_credentials(self, mock_geo):
|
||||
proxy_dict = {"server": "socks5://proxy:1080", "username": "user", "password": "pass"}
|
||||
tz, locale, ip = maybe_resolve_geoip(True, proxy_dict, None, None)
|
||||
mock_geo.assert_called_once_with("socks5://user:pass@proxy:1080")
|
||||
assert tz == "Europe/Berlin"
|
||||
assert locale == "de-DE"
|
||||
|
||||
@patch("cloakbrowser.geoip.resolve_proxy_geo_with_ip", return_value=("Europe/Berlin", "de-DE", "5.6.7.8"))
|
||||
def test_geoip_socks5_dict_no_auth_uses_server(self, mock_geo):
|
||||
proxy_dict = {"server": "socks5://proxy:1080"}
|
||||
tz, locale, ip = maybe_resolve_geoip(True, proxy_dict, None, None)
|
||||
mock_geo.assert_called_once_with("socks5://proxy:1080")
|
||||
|
||||
@patch("cloakbrowser.geoip.resolve_proxy_geo_with_ip", return_value=("Europe/London", "en-GB", "1.1.1.1"))
|
||||
def test_geoip_http_dict_does_not_inline_creds(self, mock_geo):
|
||||
# HTTP dict: credentials stay separate, only server URL passed
|
||||
proxy_dict = {"server": "http://proxy:8080", "username": "user", "password": "pass"}
|
||||
tz, locale, ip = maybe_resolve_geoip(True, proxy_dict, None, None)
|
||||
mock_geo.assert_called_once_with("http://proxy:8080")
|
||||
|
||||
|
||||
class TestBareProxyFormat:
|
||||
"""_parse_proxy_url must handle bare 'user:pass@host:port' strings (no scheme)."""
|
||||
@@ -149,8 +183,194 @@ class TestBareProxyFormat:
|
||||
r = _parse_proxy_url("proxy:8080")
|
||||
assert r == {"server": "proxy:8080"}
|
||||
|
||||
def test_build_proxy_kwargs_bare(self):
|
||||
r = _build_proxy_kwargs("user:pass@proxy:8080")
|
||||
assert r["proxy"]["username"] == "user"
|
||||
assert r["proxy"]["password"] == "pass"
|
||||
assert "user" not in r["proxy"]["server"]
|
||||
def test_resolve_proxy_config_bare(self):
|
||||
kwargs, args = _resolve_proxy_config("user:pass@proxy:8080")
|
||||
assert kwargs["proxy"]["username"] == "user"
|
||||
assert kwargs["proxy"]["password"] == "pass"
|
||||
assert "user" not in kwargs["proxy"]["server"]
|
||||
|
||||
|
||||
class TestIsSocksProxy:
|
||||
def test_socks5_string(self):
|
||||
assert _is_socks_proxy("socks5://user:pass@host:1080") is True
|
||||
|
||||
def test_socks5h_string(self):
|
||||
assert _is_socks_proxy("socks5h://host:1080") is True
|
||||
|
||||
def test_socks5_uppercase(self):
|
||||
assert _is_socks_proxy("SOCKS5://host:1080") is True
|
||||
|
||||
def test_http_string(self):
|
||||
assert _is_socks_proxy("http://host:8080") is False
|
||||
|
||||
def test_dict_socks5(self):
|
||||
assert _is_socks_proxy({"server": "socks5://host:1080"}) is True
|
||||
|
||||
def test_dict_http(self):
|
||||
assert _is_socks_proxy({"server": "http://host:8080"}) is False
|
||||
|
||||
def test_none(self):
|
||||
assert _is_socks_proxy(None) is False
|
||||
|
||||
|
||||
class TestResolveProxyConfig:
|
||||
def test_none(self):
|
||||
kwargs, args = _resolve_proxy_config(None)
|
||||
assert kwargs == {}
|
||||
assert args == []
|
||||
|
||||
def test_http_string_returns_playwright_dict(self):
|
||||
kwargs, args = _resolve_proxy_config("http://user:pass@proxy:8080")
|
||||
assert "proxy" in kwargs
|
||||
assert kwargs["proxy"]["server"] == "http://proxy:8080"
|
||||
assert kwargs["proxy"]["username"] == "user"
|
||||
assert args == []
|
||||
|
||||
def test_http_dict_passthrough(self):
|
||||
proxy = {"server": "http://proxy:8080", "bypass": ".example.com"}
|
||||
kwargs, args = _resolve_proxy_config(proxy)
|
||||
assert kwargs == {"proxy": proxy}
|
||||
assert args == []
|
||||
|
||||
def test_socks5_string_returns_chrome_arg(self):
|
||||
kwargs, args = _resolve_proxy_config("socks5://user:pass@host:1080")
|
||||
assert kwargs == {}
|
||||
assert args == ["--proxy-server=socks5://user:pass@host:1080"]
|
||||
|
||||
def test_socks5_no_auth_returns_chrome_arg(self):
|
||||
kwargs, args = _resolve_proxy_config("socks5://host:1080")
|
||||
assert kwargs == {}
|
||||
assert args == ["--proxy-server=socks5://host:1080"]
|
||||
|
||||
def test_socks5h_returns_chrome_arg(self):
|
||||
kwargs, args = _resolve_proxy_config("socks5h://user:pass@host:1080")
|
||||
assert kwargs == {}
|
||||
assert args == ["--proxy-server=socks5h://user:pass@host:1080"]
|
||||
|
||||
def test_socks5_dict_reconstructs_url(self):
|
||||
proxy = {"server": "socks5://host:1080", "username": "user", "password": "p@ss"}
|
||||
kwargs, args = _resolve_proxy_config(proxy)
|
||||
assert kwargs == {}
|
||||
assert len(args) == 1
|
||||
assert args[0].startswith("--proxy-server=socks5://user:p%40ss@host:1080")
|
||||
|
||||
def test_socks5_dict_ipv6_preserves_brackets(self):
|
||||
proxy = {"server": "socks5://[::1]:1080", "username": "user", "password": "pass"}
|
||||
kwargs, args = _resolve_proxy_config(proxy)
|
||||
assert kwargs == {}
|
||||
assert "[::1]" in args[0]
|
||||
|
||||
def test_socks5_dict_with_bypass(self):
|
||||
proxy = {"server": "socks5://host:1080", "bypass": ".example.com"}
|
||||
kwargs, args = _resolve_proxy_config(proxy)
|
||||
assert kwargs == {}
|
||||
assert "--proxy-server=socks5://host:1080" in args
|
||||
assert "--proxy-bypass-list=.example.com" in args
|
||||
|
||||
def test_socks5_string_encodes_equals_in_password(self):
|
||||
# Chromium's --proxy-server parser truncates passwords at '=' (#157).
|
||||
# Wrapper must auto URL-encode before passing to Chrome.
|
||||
_, args = _resolve_proxy_config("socks5://user:pass=123@host:1080")
|
||||
assert args == ["--proxy-server=socks5://user:pass%3D123@host:1080"]
|
||||
|
||||
def test_socks5_string_encodes_at_in_password(self):
|
||||
_, args = _resolve_proxy_config("socks5://user:p@ss@host:1080")
|
||||
# Note: parsing "user:p@ss@host" — urlparse takes everything up to LAST @
|
||||
# as userinfo, so password = "p@ss".
|
||||
assert args == ["--proxy-server=socks5://user:p%40ss@host:1080"]
|
||||
|
||||
def test_socks5_string_encoding_idempotent(self):
|
||||
# Already-encoded input should remain encoded (not double-encoded).
|
||||
_, args = _resolve_proxy_config("socks5://user:pass%3D123@host:1080")
|
||||
assert args == ["--proxy-server=socks5://user:pass%3D123@host:1080"]
|
||||
|
||||
def test_socks5_string_logs_info_when_reencoding(self, caplog):
|
||||
# When wrapper actually rewrites the URL (e.g. unencoded '=' in pwd),
|
||||
# surface an INFO log so users debugging SOCKS5 connectivity (#157)
|
||||
# can see what the wrapper did instead of being silently surprised.
|
||||
import logging
|
||||
with caplog.at_level(logging.INFO, logger="cloakbrowser"):
|
||||
_resolve_proxy_config("socks5://user:pass=123@host:1080")
|
||||
assert any("Auto URL-encoded SOCKS5" in r.message for r in caplog.records)
|
||||
# Credentials must not leak into the log.
|
||||
for r in caplog.records:
|
||||
assert "pass=123" not in r.message
|
||||
assert "pass%3D123" not in r.message
|
||||
|
||||
def test_socks5_string_silent_when_already_encoded(self, caplog):
|
||||
# Idempotent path: pre-encoded URL produces no log noise.
|
||||
import logging
|
||||
with caplog.at_level(logging.INFO, logger="cloakbrowser"):
|
||||
_resolve_proxy_config("socks5://user:pass%3D123@host:1080")
|
||||
assert not any("Auto URL-encoded SOCKS5" in r.message for r in caplog.records)
|
||||
|
||||
def test_socks5_string_silent_when_no_credentials(self, caplog):
|
||||
# No userinfo at all → no encoding work → no log.
|
||||
import logging
|
||||
with caplog.at_level(logging.INFO, logger="cloakbrowser"):
|
||||
_resolve_proxy_config("socks5://host:1080")
|
||||
assert not any("Auto URL-encoded SOCKS5" in r.message for r in caplog.records)
|
||||
|
||||
def test_socks5_string_silent_when_only_cosmetic_change(self, caplog):
|
||||
# urlparse lowercases scheme and hostname, but credentials are
|
||||
# untouched. The log must NOT fire for these cosmetic-only rewrites
|
||||
# (regression for Copilot's review on PR #209).
|
||||
import logging
|
||||
with caplog.at_level(logging.INFO, logger="cloakbrowser"):
|
||||
_resolve_proxy_config("socks5://USER:pass@HOST.com:1080")
|
||||
assert not any("Auto URL-encoded SOCKS5" in r.message for r in caplog.records)
|
||||
|
||||
def test_socks5_string_no_creds_unchanged(self):
|
||||
_, args = _resolve_proxy_config("socks5://host:1080")
|
||||
assert args == ["--proxy-server=socks5://host:1080"]
|
||||
|
||||
def test_socks5_string_password_only_still_encoded(self):
|
||||
# Empty username with password: fix must still re-encode the password
|
||||
# (regression test for empty-username bypass).
|
||||
_, args = _resolve_proxy_config("socks5://:pass=123@host:1080")
|
||||
assert args == ["--proxy-server=socks5://:pass%3D123@host:1080"]
|
||||
|
||||
def test_socks5_string_empty_password_preserves_colon(self):
|
||||
# `user:@host` (empty password) must NOT collapse to `user@host` —
|
||||
# semantics differ between the two forms.
|
||||
_, args = _resolve_proxy_config("socks5://user:@host:1080")
|
||||
assert args == ["--proxy-server=socks5://user:@host:1080"]
|
||||
|
||||
def test_socks5_string_literal_percent_in_password(self):
|
||||
# Literal '%' not followed by 2 hex digits must be encoded as '%25'
|
||||
# so Chrome decodes it back to '%'. Must not crash.
|
||||
_, args = _resolve_proxy_config("socks5://user:100%sure@host:1080")
|
||||
assert args == ["--proxy-server=socks5://user:100%25sure@host:1080"]
|
||||
|
||||
def test_socks5_string_malformed_port_passes_through(self, caplog):
|
||||
# Invalid port (non-numeric) raises in urlparse.port. Wrapper should
|
||||
# log a warning and pass original through to Chromium.
|
||||
import logging
|
||||
with caplog.at_level(logging.WARNING, logger="cloakbrowser"):
|
||||
_, args = _resolve_proxy_config("socks5://user:pass@host:abc")
|
||||
assert args == ["--proxy-server=socks5://user:pass@host:abc"]
|
||||
assert any("Malformed SOCKS5" in r.message for r in caplog.records)
|
||||
|
||||
def test_socks5_string_malformed_ipv6_passes_through(self, caplog):
|
||||
# Broken IPv6 bracket — must not crash, and must reach Chromium
|
||||
# verbatim so its own error surfaces instead of a silent rewrite.
|
||||
import logging
|
||||
with caplog.at_level(logging.WARNING, logger="cloakbrowser"):
|
||||
_, args = _resolve_proxy_config("socks5://user:pass@[::1")
|
||||
assert args == ["--proxy-server=socks5://user:pass@[::1"]
|
||||
|
||||
def test_socks5_string_preserves_path_and_query(self):
|
||||
# Nonstandard for SOCKS5, but don't silently drop user-supplied suffixes.
|
||||
# Matches JS behavior.
|
||||
_, args = _resolve_proxy_config("socks5://user:pass@host:1080/p?x=1#f")
|
||||
assert args[0] == "--proxy-server=socks5://user:pass@host:1080/p?x=1#f"
|
||||
|
||||
def test_socks5_string_ipv6_with_special_char_password(self):
|
||||
# IPv6 host + special char in password — both must be handled.
|
||||
_, args = _resolve_proxy_config("socks5://user:pass=eq@[::1]:1080")
|
||||
assert args[0] == "--proxy-server=socks5://user:pass%3Deq@[::1]:1080"
|
||||
|
||||
def test_socks5_string_port_zero_preserved(self):
|
||||
# Port 0 is an unusual but valid URL component; don't silently strip it.
|
||||
_, args = _resolve_proxy_config("socks5://user:pass=1@host:0")
|
||||
assert args[0] == "--proxy-server=socks5://user:pass%3D1@host:0"
|
||||
|
||||
Reference in New Issue
Block a user