diff --git a/CHANGELOG.md b/CHANGELOG.md index 944493d..0f8b6bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Cold-start performance milestone:** TRAWL's complete first request, including browser launch, is now nearly **4x faster** in like-for-like Docker benchmarks. Redis validation and browser warmup now run concurrently, Tier 0 becomes available immediately, and browser capacity is published progressively. Warm-request timings vary with browser state, session caching, and challenge behavior and are not included in this cold-start comparison. ### Fixed +- Support explicit non-root Docker users by baking the pinned uBlock Origin addon into both API image variants and using a writable temporary home directory. Document CA volume ownership and read-only container requirements (#60). - Reduce cold-start latency by warming Redis alongside the browser pool, publishing the first browser immediately, warming the remaining browsers concurrently, and accepting Tier 0 proxy traffic during warmup. Unavailable Redis now disables Tier 2 promptly instead of delaying the first request. Tier 0 also handles informational HTTP responses correctly and escalates authoritative `cf-mitigated: challenge` headers immediately. - Keep browser-tier status, headers, content type, and raw body aligned with the latest main-frame navigation response across redirects, and prevent persistent Cloudflare challenges from being returned as successful rendered pages (#53). - Translate Prowlarr's serialized `headers.contentType` metadata at the FlareSolverr `/v1` compatibility boundary and discard `contentLength`, allowing form POST requests to enter the scraper pipeline (#50). diff --git a/apps/api/Dockerfile b/apps/api/Dockerfile index 3c98e50..4dee112 100644 --- a/apps/api/Dockerfile +++ b/apps/api/Dockerfile @@ -42,6 +42,10 @@ RUN bun install --frozen-lockfile --production --omit=dev --linker=hoisted \ FROM oven/bun:1.3.14 AS camoufox ENV CAMOUFOX_INSTALL_DIR=/opt/camoufox +ARG UBO_VERSION=1.73.0 +ARG UBO_AMO_FILE_ID=4940584 +ARG UBO_SHA256=bccc51a773150af4af6e1fd62c7bfdeb7238b79ff2381b998fa9f2e38f64786a + RUN apt-get update && apt-get install -y --no-install-recommends \ curl unzip ca-certificates && rm -rf /var/lib/apt/lists/* @@ -82,6 +86,19 @@ RUN curl -fsSL \ [ "$(stat -c%s /opt/camoufox/GeoLite2-City.mmdb)" -gt 10000000 ] || \ { echo "GeoLite2-City.mmdb download too small / failed"; exit 1; } +# Bake uBlock Origin into Camoufox's expected addon directory. camoufox-js still +# performs its normal addon registration at startup, but no runtime download or +# extraction is needed (including when /opt/camoufox is read-only). +RUN curl -fsSL \ + "https://addons.mozilla.org/firefox/downloads/file/${UBO_AMO_FILE_ID}/ublock_origin-${UBO_VERSION}.xpi" \ + -o /tmp/ubo.xpi && \ + echo "${UBO_SHA256} /tmp/ubo.xpi" | sha256sum --check --strict - && \ + mkdir -p /opt/camoufox/addons/UBO && \ + unzip -q /tmp/ubo.xpi -d /opt/camoufox/addons/UBO && \ + test -f /opt/camoufox/addons/UBO/manifest.json && \ + test "$(bun -e 'console.log(JSON.parse(await Bun.file("/opt/camoufox/addons/UBO/manifest.json").text()).version)')" = "${UBO_VERSION}" && \ + rm /tmp/ubo.xpi + # ── Stage 3: lean runtime (only API-required files) ──────────────────────────── # debian:bookworm-slim replaces ubuntu:22.04 — same glibc family, ~50 MB smaller base. # Camoufox/Firefox require glibc; Alpine's musl is incompatible. @@ -130,6 +147,7 @@ COPY apps/api/ /app/apps/api/ COPY package.json /app/ ENV CAMOUFOX_INSTALL_DIR=/opt/camoufox \ + HOME=/tmp \ # Safe Bun runtime knobs — all tested runtime-neutral (no behavior change). BUN_DISABLE_CJS=1 \ BUN_DEBUG=0 \ diff --git a/apps/api/Dockerfile.baseline b/apps/api/Dockerfile.baseline index e717c1d..128530f 100644 --- a/apps/api/Dockerfile.baseline +++ b/apps/api/Dockerfile.baseline @@ -42,6 +42,10 @@ RUN bun install --frozen-lockfile --production --omit=dev --linker=hoisted \ FROM oven/bun:1.3.14 AS camoufox ENV CAMOUFOX_INSTALL_DIR=/opt/camoufox +ARG UBO_VERSION=1.73.0 +ARG UBO_AMO_FILE_ID=4940584 +ARG UBO_SHA256=bccc51a773150af4af6e1fd62c7bfdeb7238b79ff2381b998fa9f2e38f64786a + RUN apt-get update && apt-get install -y --no-install-recommends \ curl unzip ca-certificates && rm -rf /var/lib/apt/lists/* @@ -82,6 +86,19 @@ RUN curl -fsSL \ [ "$(stat -c%s /opt/camoufox/GeoLite2-City.mmdb)" -gt 10000000 ] || \ { echo "GeoLite2-City.mmdb download too small / failed"; exit 1; } +# Bake uBlock Origin into Camoufox's expected addon directory. camoufox-js still +# performs its normal addon registration at startup, but no runtime download or +# extraction is needed (including when /opt/camoufox is read-only). +RUN curl -fsSL \ + "https://addons.mozilla.org/firefox/downloads/file/${UBO_AMO_FILE_ID}/ublock_origin-${UBO_VERSION}.xpi" \ + -o /tmp/ubo.xpi && \ + echo "${UBO_SHA256} /tmp/ubo.xpi" | sha256sum --check --strict - && \ + mkdir -p /opt/camoufox/addons/UBO && \ + unzip -q /tmp/ubo.xpi -d /opt/camoufox/addons/UBO && \ + test -f /opt/camoufox/addons/UBO/manifest.json && \ + test "$(bun -e 'console.log(JSON.parse(await Bun.file("/opt/camoufox/addons/UBO/manifest.json").text()).version)')" = "${UBO_VERSION}" && \ + rm /tmp/ubo.xpi + # ── Stage 3: lean runtime (only API-required files) ──────────────────────────── # debian:bookworm-slim replaces ubuntu:22.04 — same glibc family, ~50 MB smaller base. # Camoufox/Firefox require glibc; Alpine's musl is incompatible. @@ -148,6 +165,7 @@ COPY apps/api/ /app/apps/api/ COPY package.json /app/ ENV CAMOUFOX_INSTALL_DIR=/opt/camoufox \ + HOME=/tmp \ # Safe Bun runtime knobs — all tested runtime-neutral (no behavior change). BUN_DISABLE_CJS=1 \ BUN_DEBUG=0 \ diff --git a/apps/docs/deployment/docker-compose.md b/apps/docs/deployment/docker-compose.md index db30efb..3ed218c 100644 --- a/apps/docs/deployment/docker-compose.md +++ b/apps/docs/deployment/docker-compose.md @@ -66,6 +66,51 @@ services: See [Standalone Containers → Older CPUs & Synology NAS](/deployment/standalone#older-cpus-synology-nas) for how to tell if you need this, and the [README](https://github.com/germondai/trawl#docker-images-one-ghcr-package-two-tags) for the full tag comparison. ::: +## Run as a non-root user + +The image defaults to root for backwards compatibility, but supports an explicit numeric UID and +GID. Camoufox and uBlock Origin are included under the read-only `/opt/camoufox` tree, while browser +profiles and other temporary files use the writable `/tmp` directory. + +```yaml +services: + trawl: + image: ghcr.io/germondai/trawl:latest + user: "1001:1001" + volumes: + - trawl_proxy_ca:/data/proxy-ca +``` + +The user must be able to write to the persistent CA volume. For a new named volume, initialize its +ownership once before starting TRAWL: + +```bash +docker run --rm -v trawl_proxy_ca:/data/proxy-ca alpine \ + chown -R 1001:1001 /data/proxy-ca +``` + +Use the actual Compose-prefixed volume name shown by `docker volume ls` if it differs from +`trawl_proxy_ca`. Bind mounts must likewise be owned by the configured UID/GID. Without write +access, the MITM proxy cannot create or reuse its CA certificate and private key. + +For a read-only container filesystem, provide a writable tmpfs for browser profiles and keep the CA +volume writable: + +```yaml +services: + trawl: + image: ghcr.io/germondai/trawl:latest + user: "1001:1001" + read_only: true + tmpfs: + - /tmp + volumes: + - trawl_proxy_ca:/data/proxy-ca +``` + +No supported persistent application data is stored in `/tmp`; sessions and cookies are managed by +TRAWL and Redis. + ## Environment variables | Variable | Default | Description |