upgrading dependencies

* fixing listing filtering by applying the correct id

Dockerfile

@@ -1,16 +1,15 @@
 FROM node:22-slim
 
-ARG TARGETARCH
-
-# System deps for Chrome for Testing + build tools for native modules (better-sqlite3)
-# On ARM64 we also install system Chromium (Chrome for Testing has no ARM64 binary)
+# System deps for CloakBrowser + build tools for native modules (better-sqlite3)
+# fonts-noto-color-emoji and fonts-freefont-ttf are required so canvas fingerprint
+# hashes match real browsers; missing emoji fonts cause bot detection on Kasada/Akamai.
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl ca-certificates fonts-liberation libasound2 \
     libatk-bridge2.0-0 libatk1.0-0 libcups2 libdbus-1-3 \
     libdrm2 libgbm1 libgtk-3-0 libnspr4 libnss3 \
     libx11-xcb1 libxcomposite1 libxdamage1 libxrandr2 xdg-utils \
+    fonts-noto-color-emoji fonts-freefont-ttf \
     python3 make g++ \
-  && if [ "$TARGETARCH" = "arm64" ]; then apt-get install -y --no-install-recommends chromium; fi \
   && rm -rf /var/lib/apt/lists/* \
   && mkdir -p /db /conf /fredy
 
@@ -26,8 +25,8 @@ RUN yarn config set network-timeout 600000 \
   && yarn --frozen-lockfile \
   && yarn cache clean
 
-# on arm64 use the system Chromium installed above
-RUN if [ "$TARGETARCH" != "arm64" ]; then npx puppeteer browsers install chrome; fi
+# Pre-download the CloakBrowser stealth Chromium binary (supports x86_64 and arm64)
+RUN node -e "import('cloakbrowser').then(({ensureBinary}) => ensureBinary())"
 
 # Purge build tools now that native modules are compiled
 RUN apt-get purge -y python3 make g++ \

README.md

@@ -167,6 +167,40 @@ For more information on how to set it up and use it, please refer to the [MCP Re
 
 Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
 
+## 🛡️ Bot Detection & Proxies
+
+Most browser-based providers (immowelt, immonet, kleinanzeigen, ...) are scraped through a hardened headless browser ([CloakBrowser](https://www.npmjs.com/package/cloakbrowser)). It makes the **browser fingerprint** indistinguishable from a real Chrome, which is enough when you run Fredy on a normal home connection.
+
+On a **server / VPS the requests usually originate from a datacenter IP**, and providers behind anti-bot systems (e.g. AWS CloudFront/WAF) block those based on **IP reputation alone**, no matter how perfect the fingerprint is. The typical symptom: it works locally but you get `We have been detected as a bot :-/` on the server.
+
+### The fix: a residential proxy
+
+A **residential proxy** routes Fredy's browser through the internet connection of a real household, so the provider sees a "normal user" IP instead of a datacenter. For German portals, use a **German (DE) residential** (or mobile/4G) proxy. Plain VPNs and **datacenter proxies do not help** here, they share the same bad reputation as your server.
+
+**Configure it** under **Settings → Execution → Proxy URL**. Supported formats:
+
+```
+http://user:pass@host:port
+socks5://user:pass@host:port
+```
+
+Leave the field empty to disable. The proxy applies to all headless-browser providers and takes effect on the next job run (no restart needed). Immoscout uses a separate mobile API and is not affected.
+
+### Where to get a residential proxy
+
+Residential proxies are a paid service (usually billed per GB, Fredy's traffic is small). Well-known providers offering German residential IPs include:
+
+| Provider | Notes |
+|---|---|
+| [IPRoyal](https://iproyal.com) | Pay-as-you-go, no monthly minimum, good for low volume |
+| [Webshare](https://www.webshare.io) | Cheap entry tier, has a small free plan to test with |
+| [Decodo (formerly Smartproxy)](https://decodo.com) | Easy setup, country/city targeting |
+| [SOAX](https://soax.com) | Residential + mobile, fine-grained geo-targeting |
+| [Bright Data](https://brightdata.com) | Largest pool, most features, higher complexity/price |
+| [Oxylabs](https://oxylabs.io) | Enterprise-grade, larger plans |
+
+This is not an endorsement, pick whatever fits your budget. For low-volume use like Fredy, a pay-as-you-go plan (e.g. IPRoyal) or a cheap entry tier (e.g. Webshare) is usually plenty. Make sure to select **Germany** as the proxy location and keep the search interval reasonable (the higher the interval, the less you look like a bot).
+
 ## Analytics
 
 Fredy is completely free (and will always remain free). However, it would be a huge help if you’d allow me to collect some analytical data.

index.js

@@ -15,6 +15,15 @@ import { initGeocodingCron } from './lib/services/crons/geocoding-cron.js';
 import { getSettings } from './lib/services/storage/settingsStorage.js';
 import SqliteConnection, { computeDbPath } from './lib/services/storage/SqliteConnection.js';
 import { initJobExecutionService } from './lib/services/jobs/jobExecutionService.js';
+import { ensureValidBinary } from './lib/services/ensureValidBinary.js';
+
+// Ensure the CloakBrowser stealth Chromium binary is present and complete before
+// jobs run.  ensureValidBinary() also detects and auto-heals partial extractions
+// (e.g. a newer version that was downloaded but only the chrome executable was
+// written) so Chrome never crashes with "Invalid file descriptor to ICU data".
+logger.info('Checking CloakBrowser binary...');
+await ensureValidBinary();
+logger.info('CloakBrowser binary ready.');
 
 //in the config, we store the path of the sqlite file, thus we must check if it is available
 const isConfigAccessible = await checkIfConfigIsAccessible();

lib/FredyPipelineExecutioner.js

@@ -5,9 +5,10 @@
 
 import { NoNewListingsWarning } from './errors.js';
 import {
-  storeListings,
-  getKnownListingHashesForJobAndProvider,
   deleteListingsById,
+  getKnownListingHashesForJobAndProvider,
+  storeListings,
+  updateListingDistance,
 } from './services/storage/listingsStorage.js';
 import { getJob } from './services/storage/jobStorage.js';
 import * as notify from './notification/notify.js';
@@ -16,8 +17,7 @@ import urlModifier from './services/queryStringMutator.js';
 import logger from './services/logger.js';
 import { geocodeAddress } from './services/geocoding/geoCodingService.js';
 import { distanceMeters } from './services/listings/distanceCalculator.js';
-import { getUserSettings, getSettings } from './services/storage/settingsStorage.js';
-import { updateListingDistance } from './services/storage/listingsStorage.js';
+import { getSettings, getUserSettings } from './services/storage/settingsStorage.js';
 import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
 import { formatListing } from './utils/formatListing.js';
 
@@ -97,9 +97,9 @@ class FredyPipelineExecutioner {
   }
 
   /**
-   * Optionally enrich new listings with data from their detail pages.
+   * Optionally, enrich new listings with data from their detail pages.
    * Only called when the provider config defines a `fetchDetails` function.
-   * Runs all fetches in parallel. Each individual fetch must handle its own errors
+   * Runs all fetches in parallel. Each fetch must handle its own errors
    * and always resolve (never reject) to avoid aborting other listings.
    *
    * @param {Listing[]} newListings New listings to enrich.
@@ -132,7 +132,7 @@ class FredyPipelineExecutioner {
     for (const listing of newListings) {
       if (listing.address) {
         const coords = await geocodeAddress(listing.address);
-        if (coords) {
+        if (coords && coords.lat !== -1 && coords.lng !== -1) {
           listing.latitude = coords.lat;
           listing.longitude = coords.lng;
         }
@@ -227,7 +227,7 @@ class FredyPipelineExecutioner {
     const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
     return new Promise((resolve, reject) => {
       extractor
-        .execute(url, this._providerConfig.waitForSelector)
+        .execute(url, this._providerConfig.waitForSelector, this._providerId)
         .then(() => {
           const listings = extractor.parseResponseText(
             this._providerConfig.crawlContainer,
@@ -264,15 +264,15 @@ class FredyPipelineExecutioner {
     const requiredKeys = this._providerConfig.requiredFieldNames;
     const requireValues = ['id', 'link', 'title'];
 
-    const filteredListings = listings
-      // this should never filter some listings out, because the normalize function should always extract all fields.
-      .filter((item) => requiredKeys.every((key) => key in item))
-      // TODO: move blacklist filter to this file, so it will handle for all providers in same way.
-      .filter(this._providerConfig.filter)
-      // filter out listings that are missing required fields
-      .filter((item) => requireValues.every((key) => item[key] != null));
-
-    return filteredListings;
+    return (
+      listings
+        // this should never filter some listings out, because the normalize function should always extract all fields.
+        .filter((item) => requiredKeys.every((key) => key in item))
+        // TODO: move blacklist filter to this file, so it will handle for all providers in same way.
+        .filter(this._providerConfig.filter)
+        // filter out listings that are missing required fields
+        .filter((item) => requireValues.every((key) => item[key] != null))
+    );
   }
 
   /**

lib/TRACKING_POIS.js

@@ -7,4 +7,9 @@ export const TRACKING_POIS = {
   DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
   WELCOME_FINISHED: 'WELCOME_FINISHED',
   WELCOME_SKIPPED: 'WELCOME_SKIPPED',
+  JOBS_TABLE_VIEW: 'JOBS_TABLE_VIEW',
+  LISTING_TABLE_VIEW: 'LISTING_TABLE_VIEW',
+  BASE_URL_SETTING: 'BASE_URL_SETTING',
+  SET_PROXY_SETTING: 'SET_PROXY_SETTING',
+  DETECTED_AS_BOT: 'DETECTED_AS_BOT',
 };

lib/api/api.js

@@ -76,13 +76,13 @@ fastify.register(async (app) => {
   app.register(dashboardPlugin, { prefix: '/api/dashboard' });
   app.register(userSettingsPlugin, { prefix: '/api/user/settings' });
   app.register(trackingPlugin, { prefix: '/api/tracking' });
+  app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
 });
 
 // Admin-only routes
 fastify.register(async (app) => {
   app.addHook('preHandler', authHook);
   app.addHook('preHandler', adminHook);
-  app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
   app.register(backupPlugin, { prefix: '/api/admin/backup' });
   app.register(userPlugin, { prefix: '/api/admin/users' });
 });

lib/api/routes/backupRouter.js

@@ -9,6 +9,10 @@ import {
   precheckRestore,
   restoreFromZip,
 } from '../../services/storage/backupRestoreService.js';
+import { getSettings } from '../../services/storage/settingsStorage.js';
+import { isAdmin } from '../security.js';
+
+const DEMO_MODE_ERROR = 'Backup and restore are not available in demo mode.';
 
 /**
  * @param {import('fastify').FastifyInstance} fastify
@@ -21,7 +25,11 @@ export default async function backupPlugin(fastify) {
     (req, body, done) => done(null, body),
   );
 
-  fastify.get('/', async (_request, reply) => {
+  fastify.get('/', async (request, reply) => {
+    const settings = await getSettings();
+    if (settings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: DEMO_MODE_ERROR });
+    }
     const zipBuffer = await createBackupZip();
     const fileName = await buildBackupFileName();
     reply.header('Content-Type', 'application/zip');
@@ -30,6 +38,10 @@ export default async function backupPlugin(fastify) {
   });
 
   fastify.post('/restore', async (request, reply) => {
+    const settings = await getSettings();
+    if (settings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: DEMO_MODE_ERROR });
+    }
     const { dryRun = 'false', force = 'false' } = request.query || {};
     const doDryRun = String(dryRun) === 'true';
     const doForce = String(force) === 'true';

lib/api/routes/generalSettingsRoute.js

@@ -9,6 +9,8 @@ import { ensureDemoUserExists } from '../../services/storage/userStorage.js';
 import logger from '../../services/logger.js';
 import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
 import { isAdmin } from '../security.js';
+import { trackPoi } from '../../services/tracking/Tracker.js';
+import { TRACKING_POIS } from '../../TRACKING_POIS.js';
 
 /**
  * @param {import('fastify').FastifyInstance} fastify
@@ -25,16 +27,26 @@ export default async function generalSettingsPlugin(fastify) {
     }
     const localSettings = await getSettings();
 
-    if (localSettings.demoMode && !isAdmin(request)) {
-      return reply.code(403).send({ error: 'In demo mode, it is not allowed to change these settings.' });
+    if (!isAdmin(request)) {
+      const reason = localSettings.demoMode
+        ? 'In demo mode, it is not allowed to change these settings.'
+        : 'Only admins can change these settings.';
+      return reply.code(403).send({ error: reason });
     }
 
     try {
       if (typeof sqlitepath !== 'undefined') {
         fs.writeFileSync(`${getDirName()}/../conf/config.json`, JSON.stringify({ sqlitepath }));
       }
+
       upsertSettings(appSettings);
       ensureDemoUserExists();
+      if (appSettings.baseUrl != null) {
+        await trackPoi(TRACKING_POIS.BASE_URL_SETTING);
+      }
+      if (appSettings.proxyUrl != null) {
+        await trackPoi(TRACKING_POIS.SET_PROXY_SETTING);
+      }
     } catch (err) {
       logger.error(err);
       return reply.code(500).send({ error: 'Error while trying to write settings.' });

lib/api/routes/userSettingsRoute.js

@@ -118,6 +118,10 @@ export default async function userSettingsPlugin(fastify) {
       return reply.code(400).send({ error: 'listings_view_mode must be "grid" or "table".' });
     }
 
+    if (listings_view_mode === 'table') {
+      await trackPoi(TRACKING_POIS.LISTING_TABLE_VIEW);
+    }
+
     try {
       upsertSettings({ listings_view_mode }, userId);
       return { success: true };
@@ -135,6 +139,10 @@ export default async function userSettingsPlugin(fastify) {
       return reply.code(400).send({ error: 'jobs_view_mode must be "grid" or "table".' });
     }
 
+    if (jobs_view_mode === 'table') {
+      await trackPoi(TRACKING_POIS.JOBS_TABLE_VIEW);
+    }
+
     try {
       upsertSettings({ jobs_view_mode }, userId);
       return { success: true };

lib/provider/immobilienDe.js

@@ -26,7 +26,7 @@ function parseId(shortenedLink) {
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immobilienDe_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);

lib/provider/immowelt.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immowelt_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);
@@ -87,7 +87,19 @@ const config = {
   crawlContainer:
     'div[data-testid="serp-core-scrollablelistview-testid"]:not(div[data-testid="serp-enlargementlist-testid"] div[data-testid="serp-card-testid"]) div[data-testid="serp-core-classified-card-testid"]',
   sortByDateParam: 'order=DateDesc',
-  waitForSelector: 'div[data-testid="serp-gridcontainer-testid"]',
+  // waitForSelector is null: extract the full page via page.content() so the
+  // Cheerio crawler can search anywhere in the rendered document.
+  // preNavigateUrl visits the homepage first to establish a trusted session
+  // before hitting the search URL; this prevents CDN-level bot challenges that
+  // fire on cold sessions. waitForNetworkIdle (phase 2) then catches React's
+  // listing API round-trip that fires well after domcontentloaded.
+  waitForSelector: null,
+  puppeteerOptions: {
+    puppeteerTimeout: 60_000,
+    preNavigateUrl: 'https://www.immowelt.de/',
+    waitForNetworkIdle: true,
+    waitForNetworkIdleTimeout: 60_000,
+  },
   crawlFields: {
     id: 'a@href',
     price: 'div[data-testid="cardmfe-price-testid"] | removeNewline | trim',

lib/provider/kleinanzeigen.js

@@ -128,7 +128,7 @@ async function enrichListingFromDetails(listing, browser) {
   if (!absoluteLink) return listing;
 
   try {
-    const html = await puppeteerExtractor(absoluteLink, null, { browser });
+    const html = await puppeteerExtractor(absoluteLink, null, { browser, name: 'kleinanzeigen_details' });
     if (!html) return { ...listing, link: absoluteLink };
 
     const { detailAddress, detailDescription } = extractDetailFromHtml(html);
@@ -196,8 +196,8 @@ const config = {
     id: '.aditem@data-adid',
     price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
     tags: '.aditem-main--middle--tags | removeNewline | trim',
-    title: '.aditem-main .text-module-begin a | removeNewline | trim',
-    link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
+    title: '.aditem-main .text-module-begin | removeNewline | trim',
+    link: '.aditem@data-href',
     description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
     address: '.aditem-main--top--left | trim | removeNewline',
     image: 'img@src',

lib/provider/sparkasse.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, 'body', { browser });
+    const html = await puppeteerExtractor(listing.link, 'body', { browser, name: 'sparkasse_details' });
 
     const $ = cheerio.load(html);
     const nextDataRaw = $('#__NEXT_DATA__').text;

lib/provider/wgGesucht.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'wgGesucht_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);

lib/services/ensureValidBinary.js

@@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { ensureBinary } from 'cloakbrowser';
+import fs from 'fs';
+import path from 'path';
+import os from 'os';
+
+/**
+ * Resource files required on Linux/Windows — they must live next to the chrome binary.
+ * macOS packages these inside the .app bundle's Frameworks directory so a different
+ * check is used there (see isBinaryComplete).
+ */
+const LINUX_WIN_REQUIRED_FILES = ['icudtl.dat', 'resources.pak'];
+
+/**
+ * Return the top-level versioned installation directory for any platform.
+ *
+ * - Linux/Windows: binaryPath is   ~/.cloakbrowser/chromium-X.Y.Z/chrome
+ *                  → dirname        ~/.cloakbrowser/chromium-X.Y.Z/
+ * - macOS:         binaryPath is   ~/.cloakbrowser/chromium-X.Y.Z/Chromium.app/Contents/MacOS/Chromium
+ *                  → 4 levels up    ~/.cloakbrowser/chromium-X.Y.Z/
+ *
+ * @param {string} binaryPath
+ * @returns {string}
+ */
+function getVersionedDir(binaryPath) {
+  if (process.platform === 'darwin') {
+    return path.resolve(path.dirname(binaryPath), '../../..');
+  }
+  return path.dirname(binaryPath);
+}
+
+/**
+ * Return true when the binary at binaryPath belongs to a complete installation.
+ *
+ * On macOS the binary lives inside an .app bundle:
+ *   Chromium.app/Contents/MacOS/Chromium
+ * Resource files (icudtl.dat etc.) are deep inside
+ *   Chromium.app/Contents/Frameworks/…
+ * so checking for them next to the binary is wrong. Instead we verify the two
+ * structural markers that are only present after a full extraction: Info.plist
+ * and the Frameworks directory inside Contents/.
+ *
+ * On Linux/Windows the binary and all resource files are siblings in the same
+ * directory.
+ *
+ * @param {string} binaryPath
+ * @returns {boolean}
+ */
+function isBinaryComplete(binaryPath) {
+  if (process.platform === 'darwin') {
+    const contentsDir = path.resolve(path.dirname(binaryPath), '..');
+    return fs.existsSync(path.join(contentsDir, 'Info.plist')) && fs.existsSync(path.join(contentsDir, 'Frameworks'));
+  }
+  const dir = path.dirname(binaryPath);
+  return LINUX_WIN_REQUIRED_FILES.every((f) => fs.existsSync(path.join(dir, f)));
+}
+
+/**
+ * Return a human-readable description of which required files/dirs are missing.
+ *
+ * @param {string} binaryPath
+ * @returns {string}
+ */
+function missingDescription(binaryPath) {
+  if (process.platform === 'darwin') {
+    const contentsDir = path.resolve(path.dirname(binaryPath), '..');
+    return ['Info.plist', 'Frameworks'].filter((f) => !fs.existsSync(path.join(contentsDir, f))).join(', ');
+  }
+  const dir = path.dirname(binaryPath);
+  return LINUX_WIN_REQUIRED_FILES.filter((f) => !fs.existsSync(path.join(dir, f))).join(', ');
+}
+
+/**
+ * Remove a corrupt binary installation and all `latest_version*` markers from
+ * the CloakBrowser cache so the next `ensureBinary()` call falls back to the
+ * package-bundled version.
+ *
+ * Removes the full versioned directory (e.g. chromium-X.Y.Z/) on all platforms,
+ * not just the subdirectory that contains the binary.
+ *
+ * @param {string} binaryPath - Path to the (corrupt) chrome/Chromium binary.
+ */
+function removeCorruptInstallation(binaryPath) {
+  const versionedDir = getVersionedDir(binaryPath);
+  const cacheDir = process.env.CLOAKBROWSER_CACHE_DIR || path.join(os.homedir(), '.cloakbrowser');
+
+  fs.rmSync(versionedDir, { recursive: true, force: true });
+
+  try {
+    for (const entry of fs.readdirSync(cacheDir)) {
+      if (entry.startsWith('latest_version')) {
+        fs.rmSync(path.join(cacheDir, entry), { force: true });
+      }
+    }
+  } catch {
+    // Cache dir may not exist if versionedDir was the only entry — ignore.
+  }
+}
+
+/**
+ * Ensure the CloakBrowser stealth Chromium binary is present **and** complete.
+ *
+ * `cloakbrowser`'s own `ensureBinary()` only checks that the chrome/Chromium
+ * file exists. An incomplete extraction (e.g. interrupted download, disk full)
+ * can leave a directory that contains the executable but is missing essential
+ * resource files. Chrome then crashes immediately on launch.
+ *
+ * This wrapper validates the path returned by `ensureBinary()`. If the
+ * installation is incomplete it removes the corrupt directory, clears the
+ * version marker files, and calls `ensureBinary()` again so it falls back to
+ * (or re-downloads) a complete build.
+ *
+ * The validated path is also pinned via `CLOAKBROWSER_BINARY_PATH` so that
+ * CloakBrowser's own internal `ensureBinary()` call inside `launch()` always
+ * picks up the same, verified binary.
+ *
+ * @returns {Promise<string>} Absolute path to the validated binary.
+ * @throws {Error} When even the fallback binary is incomplete.
+ */
+export async function ensureValidBinary() {
+  const binaryPath = await ensureBinary();
+
+  if (isBinaryComplete(binaryPath)) {
+    process.env.CLOAKBROWSER_BINARY_PATH = binaryPath;
+    return binaryPath;
+  }
+
+  console.warn(
+    `[fredy] CloakBrowser installation at ${getVersionedDir(binaryPath)} is missing: ${missingDescription(binaryPath)}. Removing and retrying.`,
+  );
+
+  removeCorruptInstallation(binaryPath);
+
+  const fallbackPath = await ensureBinary();
+  if (!isBinaryComplete(fallbackPath)) {
+    throw new Error(
+      `CloakBrowser binary at ${getVersionedDir(fallbackPath)} is still missing required files after re-download: ${missingDescription(fallbackPath)}`,
+    );
+  }
+
+  process.env.CLOAKBROWSER_BINARY_PATH = fallbackPath;
+  return fallbackPath;
+}

lib/services/extractor/extractor.js

@@ -29,11 +29,12 @@ export default class Extractor {
    * your response will never contain what you are really looking for
    * @param url
    * @param waitForSelector
+   * @param jobKey
    */
-  execute = async (url, waitForSelector = null) => {
+  execute = async (url, waitForSelector = null, jobKey = null) => {
     this.responseText = null;
     try {
-      this.responseText = await puppeteerExtractor(url, waitForSelector, this.options);
+      this.responseText = await puppeteerExtractor(url, waitForSelector, { ...this.options, name: jobKey });
       if (this.responseText != null) {
         loadParser(this.responseText);
       }

lib/services/extractor/puppeteerExtractor.js

@@ -3,121 +3,133 @@
  * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
  */
 
-import puppeteer from 'puppeteer-extra';
-import StealthPlugin from 'puppeteer-extra-plugin-stealth';
-import { debug, botDetected } from './utils.js';
-import {
-  getPreLaunchConfig,
-  applyBotPreventionToPage,
-  applyLanguagePersistence,
-  applyPostNavigationHumanSignals,
-} from './botPrevention.js';
+import { launch } from 'cloakbrowser/puppeteer';
+import { botDetected, debug } from './utils.js';
+import { getPreLaunchConfig } from './botPrevention.js';
 import logger from '../logger.js';
-import fs from 'fs';
-import os from 'os';
-import path from 'path';
-
-puppeteer.use(StealthPlugin());
+import { trackPoi } from '../tracking/Tracker.js';
+import { TRACKING_POIS } from '../../TRACKING_POIS.js';
 
+/**
+ * Launch a CloakBrowser/Puppeteer browser instance with stealth and humanizer enabled.
+ *
+ * CloakBrowser applies 49 C++ source-level patches (canvas, WebGL, audio, WebRTC,
+ * navigator.*, automation signals) that are indistinguishable from a real browser.
+ * All fingerprinting and human-behaviour simulation is handled natively; no CDP
+ * overrides (setUserAgent, setExtraHTTPHeaders, evaluateOnNewDocument) are applied
+ * here because they would create detectable inconsistencies on top of the C++ patches.
+ *
+ * @param {string} url - Initial URL (used to derive locale/timezone hints).
+ * @param {object} [options]
+ * @param {boolean} [options.puppeteerHeadless]
+ * @param {number}  [options.puppeteerTimeout]
+ * @param {string}  [options.proxyUrl]
+ * @param {string}  [options.timezone]
+ * @param {string}  [options.acceptLanguage]
+ * @param {object}  [options.viewport]
+ * @returns {Promise<import('puppeteer-core').Browser>}
+ */
 export async function launchBrowser(url, options) {
   const preCfg = getPreLaunchConfig(url, options || {});
-  const launchArgs = [
+
+  // Docker requires --no-sandbox; CloakBrowser handles all stealth args internally.
+  // --ignore-certificate-errors is needed because CloakBrowser ships its own Chromium
+  // binary with an independent CA bundle that may not trust proxies or interceptors
+  // present in the host environment.
+  const args = [
     '--no-sandbox',
-    '--disable-gpu',
     '--disable-setuid-sandbox',
     '--disable-dev-shm-usage',
-    '--disable-crash-reporter',
     '--no-first-run',
     '--no-default-browser-check',
-    preCfg.langArg,
+    '--ignore-certificate-errors',
+    // Disables the zygote process model. Required in some container environments
+    // (e.g. limited kernel namespaces) where the zygote cannot acquire the
+    // locks it needs and exits with "Invalid file descriptor to ICU data received".
+    '--no-zygote',
     preCfg.windowSizeArg,
-    ...preCfg.extraArgs,
   ];
-  if (options?.proxyUrl) {
-    launchArgs.push(`--proxy-server=${options.proxyUrl}`);
-  }
 
-  let userDataDir;
-  let removeUserDataDir = false;
-  if (options && options.userDataDir) {
-    userDataDir = options.userDataDir;
-  } else {
-    const prefix = path.join(os.tmpdir(), 'puppeteer-fredy-');
-    userDataDir = fs.mkdtempSync(prefix);
-    removeUserDataDir = true;
-  }
-
-  // On ARM64 Docker, Chrome for Testing has no native binary - use system Chromium instead.
-  const executablePath =
-    options?.executablePath ||
-    (process.arch === 'arm64' && process.env.IS_DOCKER === 'true' ? '/usr/bin/chromium' : undefined);
-
-  const browser = await puppeteer.launch({
+  return await launch({
     headless: options?.puppeteerHeadless ?? true,
-    args: launchArgs,
-    timeout: options?.puppeteerTimeout || 45_000,
-    userDataDir,
-    executablePath,
+    humanize: true,
+    args,
+    // locale sets Accept-Language headers and JS navigator.language consistently
+    locale: preCfg.langForFlag,
+    ...(options?.proxyUrl ? { proxy: options.proxyUrl } : {}),
+    ...(preCfg.timezone ? { timezone: preCfg.timezone } : {}),
   });
-
-  browser.__fredy_userDataDir = userDataDir;
-  browser.__fredy_removeUserDataDir = removeUserDataDir;
-
-  return browser;
 }
 
+/**
+ * Close a browser instance returned by {@link launchBrowser}.
+ *
+ * @param {import('puppeteer-core').Browser | null} browser
+ */
 export async function closeBrowser(browser) {
   if (!browser) return;
-  const userDataDir = browser.__fredy_userDataDir;
-  const removeUserDataDir = browser.__fredy_removeUserDataDir;
   try {
     await browser.close();
   } catch {
     // ignore
   }
-  if (removeUserDataDir && userDataDir) {
-    try {
-      await fs.promises.rm(userDataDir, { recursive: true, force: true });
-    } catch {
-      // ignore
-    }
-  }
 }
 
+/**
+ * Open a page in a (possibly reused) browser, navigate to `url`, and return the HTML source.
+ * Returns `null` when a bot-detection page is encountered or on timeout.
+ *
+ * @param {string} url
+ * @param {string | null} waitForSelector
+ * @param {object} [options]
+ * @returns {Promise<string | null>}
+ */
 export default async function execute(url, waitForSelector, options) {
   let browser = options?.browser;
   let isExternalBrowser = !!browser;
   let page;
   let result;
   try {
-    debug(`Sending request to ${url} using Puppeteer.`);
+    debug(`Sending request to ${url} using CloakBrowser.`);
 
     if (!isExternalBrowser) {
       browser = await launchBrowser(url, options);
     }
 
     page = await browser.newPage();
-    const preCfg = getPreLaunchConfig(url, options || {});
-    await applyBotPreventionToPage(page, preCfg);
-    // Provide languages value before navigation
-    await applyLanguagePersistence(page, preCfg);
 
-    // Optional cookies
     if (Array.isArray(options?.cookies) && options.cookies.length > 0) {
       await page.setCookie(...options.cookies);
     }
 
-    // Navigation
+    // Warm-up navigation: visit a trusted page first so the site sees an
+    // established session before the actual target URL. Silently ignored on
+    // failure so it never blocks the main request.
+    if (options?.preNavigateUrl) {
+      try {
+        await page.goto(options.preNavigateUrl, { waitUntil: 'domcontentloaded', timeout: 30_000 });
+        await new Promise((r) => setTimeout(r, 1500 + Math.random() * 2000));
+      } catch {
+        // ignore
+      }
+    }
+
     const response = await page.goto(url, {
       waitUntil: options?.waitUntil || 'domcontentloaded',
       timeout: options?.puppeteerTimeout || 60000,
     });
 
-    // Optionally wait and add subtle human-like interactions
-    await applyPostNavigationHumanSignals(page, preCfg);
+    // Optional second idle wait: useful for React SPAs that trigger API calls
+    // after domcontentloaded. Times out silently so we use whatever is rendered.
+    if (options?.waitForNetworkIdle) {
+      try {
+        await page.waitForNetworkIdle({ timeout: options?.waitForNetworkIdleTimeout ?? 60_000 });
+      } catch {
+        // ignore — we proceed with whatever the DOM contains at this point
+      }
+    }
 
     let pageSource;
-    // if we're extracting data from a SPA, we must wait for the selector
     if (waitForSelector != null) {
       const selectorTimeout = options?.puppeteerSelectorTimeout ?? options?.puppeteerTimeout ?? 30_000;
       await page.waitForSelector(waitForSelector, { timeout: selectorTimeout });
@@ -133,15 +145,22 @@ export default async function execute(url, waitForSelector, options) {
 
     if (botDetected(pageSource, statusCode)) {
       logger.warn('We have been detected as a bot :-/ Tried url: => ', url);
+
+      if (options != null && options.name != null) {
+        await trackPoi(TRACKING_POIS.DETECTED_AS_BOT + '_' + options.name);
+      } else {
+        await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
+      }
+
       result = null;
     } else {
       result = pageSource || (await page.content());
     }
   } catch (error) {
     if (error?.name?.includes('Timeout')) {
-      logger.debug('Error executing with puppeteer executor', error);
+      logger.debug('Error executing with CloakBrowser executor', error);
     } else {
-      logger.warn('Error executing with puppeteer executor', error);
+      logger.warn('Error executing with CloakBrowser executor', error);
     }
     result = null;
   } finally {

lib/services/immoscout/immoscout-web-translator.js

@@ -168,10 +168,6 @@ export function convertWebToMobile(webUrl) {
     }
   }
 
-  if (segments.includes('shape')) {
-    throw new Error('Shape is currently not supported using Immoscout');
-  }
-
   const { query: rawParams } = queryString.parseUrl(webUrl, { arrayFormat: 'comma' });
   const webParams = Object.fromEntries(
     Object.entries(rawParams).filter(([key]) => key !== 'enteredFrom' && PARAM_NAME_MAP[key]),
@@ -179,18 +175,31 @@ export function convertWebToMobile(webUrl) {
 
   const geocodes = `/${segments.slice(2, segments.length - 1).join('/')}`;
   const isRadius = segments.includes('radius');
+  const isShape = segments.includes('shape');
   const mobileParams = {
-    searchType: isRadius ? 'radius' : 'region',
+    searchType: isRadius ? 'radius' : isShape ? 'shape' : 'region',
     realestatetype: realType,
-    ...(isRadius ? {} : { geocodes }),
+    ...(isRadius || isShape ? {} : { geocodes }),
     ...additionalParamsFromWebPath,
   };
 
+  if (isShape && !webParams.shape) {
+    throw new Error('Shape search URL is missing the required "shape" query parameter');
+  }
+
+  if (isShape && webParams.shape) {
+    const browserShape = webParams.shape;
+    const normalized = browserShape.replace(/\.\./g, '==').replace(/\./g, '=');
+    const polyline = Buffer.from(normalized, 'base64').toString('utf-8');
+    mobileParams.shape = polyline;
+  }
+
   if (webParams.geocoordinates) {
     mobileParams.geocoordinates = webParams.geocoordinates;
   }
 
   for (const [key, val] of Object.entries(webParams)) {
+    if (key === 'shape') continue;
     if (key === 'equipment') {
       const items = [].concat(val).flatMap((v) => `${v}`.split(','));
       const currentEquipmentParams = mobileParams[PARAM_NAME_MAP[key]];

lib/services/jobs/jobExecutionService.js

@@ -14,6 +14,7 @@ import * as similarityCache from '../similarity-check/similarityCache.js';
 import { isRunning, markFinished, markRunning } from './run-state.js';
 import { sendToUsers } from '../sse/sse-broker.js';
 import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
+import { getSettings } from '../storage/settingsStorage.js';
 
 /**
  * Initializes the job execution service.
@@ -160,6 +161,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
     }
     let browser;
     try {
+      // Read the proxy live (not from the startup snapshot) so changing it in the
+      // UI takes effect on the next run without a backend restart. An empty value
+      // disables the proxy. Routing the headless browser through a (German
+      // residential) proxy avoids datacenter-IP based bot detection on the
+      // Puppeteer-based providers (immowelt, immonet, kleinanzeigen, ...).
+      const liveSettings = await getSettings();
+      const proxyUrl = typeof liveSettings?.proxyUrl === 'string' ? liveSettings.proxyUrl.trim() : '';
+
       const jobProviders = job.provider.filter(
         (p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
       );
@@ -168,14 +177,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
           const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
           matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
 
-          if (browser && !browser.isConnected()) {
+          if (browser && !browser.connected) {
             logger.debug('Browser is disconnected, nullifying to launch a new one.');
             await puppeteerExtractor.closeBrowser(browser);
             browser = null;
           }
 
           if (!browser && matchedProvider.config.getListings == null) {
-            browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
+            browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, proxyUrl ? { proxyUrl } : {});
           }
 
           await new FredyPipelineExecutioner(matchedProvider.config, job, prov.id, similarityCache, browser).execute();

lib/services/storage/listingsStorage.js

@@ -214,6 +214,8 @@ export const storeListings = (jobId, providerId, listings) => {
         longitude: item.longitude || null,
       };
       stmt.run(params);
+      // Propagate the DB primary key back so downstream pipeline steps use the correct id
+      item.id = params.id;
     }
   });
 
@@ -417,9 +419,10 @@ export const deleteListingsByJobId = (jobId, hardDelete = false) => {
 };
 
 /**
- * Delete listings by a list of listing IDs.
+ * Delete listings by a list of listing IDs (the nanoid primary key stored in the `id` column).
+ * Used by API routes that receive row IDs from the client.
  *
- * @param {string[]} ids - Array of listing IDs to delete.
+ * @param {string[]} ids - Array of DB row IDs to delete.
  * @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
  * @returns {any} The result from SqliteConnection.execute.
  */

package.json

@@ -1,6 +1,6 @@
 {
   "name": "fredy",
-  "version": "21.3.0",
+  "version": "22.2.0",
   "description": "[F]ind [R]eal [E]states [d]amn eas[y].",
   "scripts": {
     "prepare": "husky",
@@ -62,9 +62,9 @@
     "Firefox ESR"
   ],
   "dependencies": {
-    "@douyinfe/semi-icons": "^2.96.1",
-    "@douyinfe/semi-ui": "2.96.1",
-    "@douyinfe/semi-ui-19": "^2.96.1",
+    "@douyinfe/semi-icons": "^2.99.3",
+    "@douyinfe/semi-ui": "2.99.3",
+    "@douyinfe/semi-ui-19": "^2.99.3",
     "@fastify/cookie": "^11.0.2",
     "@fastify/helmet": "^13.0.2",
     "@fastify/session": "^11.1.1",
@@ -73,11 +73,12 @@
     "@modelcontextprotocol/sdk": "^1.29.0",
     "@sendgrid/mail": "8.1.6",
     "@turf/boolean-point-in-polygon": "^7.3.5",
-    "@vitejs/plugin-react": "6.0.1",
+    "@vitejs/plugin-react": "6.0.2",
     "adm-zip": "^0.5.17",
-    "better-sqlite3": "^12.9.0",
+    "better-sqlite3": "^12.10.0",
     "chart.js": "^4.5.1",
     "cheerio": "^1.2.0",
+    "cloakbrowser": "^0.3.31",
     "fastify": "^5.8.5",
     "handlebars": "4.7.9",
     "maplibre-gl": "^5.24.0",
@@ -85,43 +86,41 @@
     "node-cron": "^4.2.1",
     "node-fetch": "3.3.2",
     "node-mailjet": "6.0.11",
-    "nodemailer": "^8.0.7",
+    "nodemailer": "^8.0.10",
     "p-throttle": "^8.1.0",
     "package-up": "^5.0.0",
-    "puppeteer": "^24.43.0",
-    "puppeteer-extra": "^3.3.6",
-    "puppeteer-extra-plugin-stealth": "^2.11.2",
-    "query-string": "9.3.1",
-    "react": "19.2.6",
+    "puppeteer-core": "^25.1.0",
+    "query-string": "9.4.0",
+    "react": "19.2.7",
     "react-chartjs-2": "^5.3.1",
-    "react-dom": "19.2.6",
+    "react-dom": "19.2.7",
     "react-range-slider-input": "^3.3.5",
-    "react-router": "7.15.0",
-    "react-router-dom": "7.15.0",
-    "resend": "^6.12.3",
-    "semver": "^7.7.4",
+    "react-router": "7.16.0",
+    "react-router-dom": "7.16.0",
+    "resend": "^6.12.4",
+    "semver": "^7.8.1",
     "slack": "11.0.2",
-    "vite": "8.0.11",
+    "vite": "8.0.16",
     "x-var": "^3.0.1",
-    "zustand": "^5.0.13"
+    "zustand": "^5.0.14"
   },
   "devDependencies": {
-    "@babel/core": "7.29.0",
-    "@babel/eslint-parser": "7.28.6",
-    "@babel/preset-env": "7.29.5",
-    "@babel/preset-react": "7.28.5",
+    "@babel/core": "7.29.7",
+    "@babel/eslint-parser": "7.29.7",
+    "@babel/preset-env": "7.29.7",
+    "@babel/preset-react": "7.29.7",
     "@eslint/js": "^10.0.1",
     "chalk": "^5.6.2",
-    "eslint": "10.3.0",
+    "eslint": "10.4.1",
     "eslint-config-prettier": "10.1.8",
     "eslint-plugin-react": "7.37.5",
     "globals": "^17.6.0",
     "history": "5.3.0",
     "husky": "9.1.7",
     "less": "4.6.4",
-    "lint-staged": "16.4.0",
+    "lint-staged": "17.0.7",
     "nodemon": "^3.1.14",
     "prettier": "3.8.3",
-    "vitest": "^4.1.5"
+    "vitest": "^4.1.8"
   }
 }

test/globalSetup.js

@@ -0,0 +1,18 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { ensureValidBinary } from '../lib/services/ensureValidBinary.js';
+
+/**
+ * Vitest global setup — runs once in the main process before any workers start.
+ * Downloads and validates the CloakBrowser stealth Chromium binary.
+ * ensureValidBinary() also removes and re-downloads partial/corrupt installations
+ * so tests never fail with "Invalid file descriptor to ICU data received".
+ * Skipped in offline mode because the browser is fully mocked there.
+ */
+export async function setup() {
+  if (process.env.TEST_MODE === 'offline') return;
+  await ensureValidBinary();
+}

test/mocks/mockStore.js

@@ -32,4 +32,7 @@ export const deletedIds = [];
 export const deleteListingsById = (ids) => {
   deletedIds.push(...ids);
 };
+export const deleteListingsByHash = (hashes) => {
+  deletedIds.push(...hashes);
+};
 /* eslint-enable no-unused-vars */

test/provider/immobilienDe.test.js

@@ -6,83 +6,89 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { providerConfig, mockFredy } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/immobilienDe.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session, avoiding double cold-start bot detection.
+const TEST_TIMEOUT = 120_000;
 
 describe('#immobilien.de testsuite()', () => {
   provider.init(providerConfig.immobilienDe, [], []);
-  it('should test immobilien.de provider', async () => {
-    const mockedJob = {
-      id: 'test1',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
 
-    const Fredy = await mockFredy();
-    const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
-    const listing = await fredy.execute();
+  let browser;
+  let liveListings;
 
-    if (listing == null || listing.length === 0) {
-      throw new Error('Listings is empty!');
-    }
+  beforeAll(async () => {
+    browser = await launchBrowser(providerConfig.immobilienDe.url);
+  }, TEST_TIMEOUT);
 
-    expect(listing).toBeInstanceOf(Array);
-    const notificationObj = get();
-    expect(notificationObj).toBeTypeOf('object');
-    expect(notificationObj.serviceName).toBe('immobilienDe');
-    notificationObj.payload.forEach((notify) => {
-      /** check the actual structure **/
-      expect(notify.id).toBeTypeOf('string');
-      expect(notify.price).toBeTypeOf('string');
-      expect(notify.size).toBeTypeOf('string');
-      expect(notify.title).toBeTypeOf('string');
-      expect(notify.link).toBeTypeOf('string');
-      expect(notify.address).toBeTypeOf('string');
-      /** check the values if possible **/
-      expect(notify.price).toContain('€');
-      expect(notify.size).toContain('m²');
-      expect(notify.title).not.toBe('');
-      expect(notify.link).toContain('https://www.immobilien.de');
-      expect(notify.address).not.toBe('');
-    });
+  afterAll(async () => {
+    await closeBrowser(browser);
   });
 
+  it(
+    'should test immobilien.de provider',
+    async () => {
+      const mockedJob = {
+        id: 'test1',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+
+      const Fredy = await mockFredy();
+      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+      liveListings = await fredy.execute();
+
+      if (liveListings == null || liveListings.length === 0) {
+        throw new Error('Listings is empty!');
+      }
+
+      expect(liveListings).toBeInstanceOf(Array);
+      const notificationObj = get();
+      expect(notificationObj).toBeTypeOf('object');
+      expect(notificationObj.serviceName).toBe('immobilienDe');
+      notificationObj.payload.forEach((notify) => {
+        /** check the actual structure **/
+        expect(notify.id).toBeTypeOf('string');
+        expect(notify.price).toBeTypeOf('string');
+        expect(notify.size).toBeTypeOf('string');
+        expect(notify.title).toBeTypeOf('string');
+        expect(notify.link).toBeTypeOf('string');
+        expect(notify.address).toBeTypeOf('string');
+        /** check the values if possible **/
+        expect(notify.price).toContain('€');
+        expect(notify.size).toContain('m²');
+        expect(notify.title).not.toBe('');
+        expect(notify.link).toContain('https://www.immobilien.de');
+        expect(notify.address).not.toBe('');
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
-    });
+    it(
+      'should enrich listings with details',
+      async () => {
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
-      vi.restoreAllMocks();
-    });
+        // Call fetchDetails directly on the first live listing — no need to
+        // re-scrape the search page. The shared browser keeps the session warm.
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
-      const Fredy = await mockFredy();
-      provider.init(providerConfig.immobilienDe, [], []);
-      const mockedJob = { id: 'test1', notificationAdapter: null, specFilter: null, spatialFilter: null };
-
-      const fredy = new Fredy(
-        provider.config,
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      if (listings == null) return;
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.immobilien.de');
-        expect(listing.address).toBeTypeOf('string');
-        expect(listing.address).not.toBe('');
-        // description may be null if selectors don't match yet - falls back gracefully
-        if (listing.description != null) {
-          expect(listing.description).toBeTypeOf('string');
+        if (enriched == null) return;
+        expect(enriched.link).toContain('https://www.immobilien.de');
+        expect(enriched.address).toBeTypeOf('string');
+        expect(enriched.address).not.toBe('');
+        // description may be null if selectors don't match yet — falls back gracefully
+        if (enriched.description != null) {
+          expect(enriched.description).toBeTypeOf('string');
         }
-      });
-    });
+      },
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/immoscout.test.js

@@ -3,85 +3,85 @@
  * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
  */
 
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { mockFredy, providerConfig } from '../utils.js';
 import { get } from '../mocks/mockNotification.js';
 import * as provider from '../../lib/provider/immoscout.js';
-import * as mockStore from '../mocks/mockStore.js';
+
+// immoscout uses the mobile REST API (fetch-based, no browser). Both tests share
+// the same module-level listings so the API is only queried once, avoiding
+// duplicate requests that could trigger rate-limiting.
+const TEST_TIMEOUT = 120_000;
 
 describe('#immoscout provider testsuite()', () => {
   provider.init(providerConfig.immoscout, [], []);
-  it('should test immoscout provider', async () => {
-    const Fredy = await mockFredy();
-    const mockedJob = {
-      id: '',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
 
-    return await new Promise((resolve, reject) => {
-      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
-      fredy.execute().then((listings) => {
-        if (listings == null || listings.length === 0) {
-          reject('Listings is empty!');
-          return;
-        }
+  let liveListings;
 
-        expect(listings).toBeInstanceOf(Array);
-        const notificationObj = get();
-        expect(notificationObj).toBeTypeOf('object');
+  it(
+    'should test immoscout provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: '',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
 
-        // check if there is at least one valid notification
-        const hasValidNotification = notificationObj.payload.some((notify) => {
-          return (
-            typeof notify.id === 'string' &&
-            typeof notify.price === 'string' &&
-            notify.price.includes('€') &&
-            typeof notify.size === 'string' &&
-            notify.size.includes('m²') &&
-            typeof notify.title === 'string' &&
-            notify.title !== '' &&
-            typeof notify.link === 'string' &&
-            notify.link.includes('https://www.immobilienscout24.de/') &&
-            typeof notify.address === 'string'
-          );
+      return await new Promise((resolve, reject) => {
+        const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+        fredy.execute().then((listings) => {
+          if (listings == null || listings.length === 0) {
+            reject('Listings is empty!');
+            return;
+          }
+
+          liveListings = listings;
+          expect(listings).toBeInstanceOf(Array);
+          const notificationObj = get();
+          expect(notificationObj).toBeTypeOf('object');
+
+          // check if there is at least one valid notification
+          const hasValidNotification = notificationObj.payload.some((notify) => {
+            return (
+              typeof notify.id === 'string' &&
+              typeof notify.price === 'string' &&
+              notify.price.includes('€') &&
+              typeof notify.size === 'string' &&
+              notify.size.includes('m²') &&
+              typeof notify.title === 'string' &&
+              notify.title !== '' &&
+              typeof notify.link === 'string' &&
+              notify.link.includes('https://www.immobilienscout24.de/') &&
+              typeof notify.address === 'string'
+            );
+          });
+
+          expect(hasValidNotification).toBe(true);
+          resolve();
         });
-
-        expect(hasValidNotification).toBe(true);
-        resolve();
       });
-    });
-  });
+    },
+    TEST_TIMEOUT,
+  );
 
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
-    });
+    it(
+      'should enrich listings with details',
+      async () => {
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
-      vi.restoreAllMocks();
-    });
+        // Call fetchDetails directly on the first live listing — no need to
+        // re-query the search API. immoscout uses fetch (no browser).
+        const enriched = await provider.config.fetchDetails(liveListings[0]);
 
-    it('should enrich listings with details', async () => {
-      const Fredy = await mockFredy();
-      provider.init(providerConfig.immoscout, [], []);
-      const mockedJob = { id: '', notificationAdapter: null, specFilter: null, spatialFilter: null };
-      const fredy = new Fredy(
-        provider.config,
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.description).toBeTypeOf('string');
-        expect(listing.description).not.toBe('');
-      });
-    });
+        expect(enriched).toBeTruthy();
+        expect(enriched.description).toBeTypeOf('string');
+        expect(enriched.description).not.toBe('');
+      },
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/immowelt.test.js

@@ -6,87 +6,95 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { mockFredy, providerConfig } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/immowelt.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session. Immowelt's CDN challenges cold sessions
+// aggressively; a shared warm browser prevents the second request from being
+// blocked as a bot hit.
+const TEST_TIMEOUT = 180_000;
 
 describe('#immowelt testsuite()', () => {
-  it('should test immowelt provider', async () => {
-    const Fredy = await mockFredy();
-    const mockedJob = {
-      id: 'immowelt',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
-    provider.init(providerConfig.immowelt, [], []);
+  let browser;
+  let liveListings;
 
-    const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  beforeAll(async () => {
+    browser = await launchBrowser(providerConfig.immowelt.url);
+  }, TEST_TIMEOUT);
 
-    const listing = await fredy.execute();
-
-    if (listing == null || listing.length === 0) {
-      throw new Error('Listings is empty!');
-    }
-
-    expect(listing).toBeInstanceOf(Array);
-    const notificationObj = get();
-    expect(notificationObj).toBeTypeOf('object');
-    expect(notificationObj.serviceName).toBe('immowelt');
-    notificationObj.payload.forEach((notify) => {
-      /** check the actual structure **/
-      expect(notify.id).toBeTypeOf('string');
-      if (notify.price != null) {
-        expect(notify.price).toBeTypeOf('string');
-        expect(notify.price).toContain('€');
-      }
-      expect(notify.title).toBeTypeOf('string');
-      expect(notify.link).toBeTypeOf('string');
-      expect(notify.address).toBeTypeOf('string');
-      /** check the values if possible **/
-      if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
-        expect(notify.size).toBeTypeOf('string');
-        expect(notify.size).toContain('m²');
-      }
-      expect(notify.title).not.toBe('');
-      expect(notify.link).toContain('https://www.immowelt.de');
-      expect(notify.address).not.toBe('');
-    });
+  afterAll(async () => {
+    await closeBrowser(browser);
   });
 
+  it(
+    'should test immowelt provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'immowelt',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+      provider.init(providerConfig.immowelt, [], []);
+
+      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+      liveListings = await fredy.execute();
+
+      if (liveListings == null || liveListings.length === 0) {
+        throw new Error('Listings is empty!');
+      }
+
+      expect(liveListings).toBeInstanceOf(Array);
+      const notificationObj = get();
+      expect(notificationObj).toBeTypeOf('object');
+      expect(notificationObj.serviceName).toBe('immowelt');
+      notificationObj.payload.forEach((notify) => {
+        /** check the actual structure **/
+        expect(notify.id).toBeTypeOf('string');
+        if (notify.price != null) {
+          expect(notify.price).toBeTypeOf('string');
+          expect(notify.price).toContain('€');
+        }
+        expect(notify.title).toBeTypeOf('string');
+        expect(notify.link).toBeTypeOf('string');
+        expect(notify.address).toBeTypeOf('string');
+        /** check the values if possible **/
+        if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
+          expect(notify.size).toBeTypeOf('string');
+          expect(notify.size).toContain('m²');
+        }
+        expect(notify.title).not.toBe('');
+        expect(notify.link).toContain('https://www.immowelt.de');
+        expect(notify.address).not.toBe('');
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
-    });
+    it(
+      'should enrich listings with details',
+      async () => {
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
-      vi.restoreAllMocks();
-    });
+        // Call fetchDetails directly on the first live listing — no need to
+        // re-scrape the search page. The shared browser keeps the session warm.
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
-      const Fredy = await mockFredy();
-      provider.init(providerConfig.immowelt, [], []);
-      const mockedJob = { id: 'immowelt', notificationAdapter: null, specFilter: null, spatialFilter: null };
-
-      const fredy = new Fredy(
-        provider.config,
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.immowelt.de');
-        expect(listing.address).toBeTypeOf('string');
-        expect(listing.address).not.toBe('');
+        expect(enriched).toBeTruthy();
+        expect(enriched.link).toContain('https://www.immowelt.de');
+        expect(enriched.address).toBeTypeOf('string');
+        expect(enriched.address).not.toBe('');
         // description is enriched from the detail page; falls back gracefully if blocked
-        if (listing.description != null) {
-          expect(listing.description).toBeTypeOf('string');
+        if (enriched.description != null) {
+          expect(enriched.description).toBeTypeOf('string');
         }
-      });
-    });
+      },
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/kleinanzeigen.test.js

@@ -6,80 +6,88 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { mockFredy, providerConfig } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/kleinanzeigen.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session. Kleinanzeigen rate-limits cold browser
+// sessions; a shared warm browser prevents the second request from being blocked.
+const TEST_TIMEOUT = 180_000;
 
 describe('#kleinanzeigen testsuite()', () => {
-  it('should test kleinanzeigen provider', async () => {
-    const Fredy = await mockFredy();
-    const mockedJob = {
-      id: 'kleinanzeigen',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
-    provider.init(providerConfig.kleinanzeigen, [], []);
-    return await new Promise((resolve, reject) => {
-      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  let browser;
+  let liveListings;
 
-      fredy.execute().then((listing) => {
-        if (listing == null || listing.length === 0) {
-          reject('Listings is empty!');
-          return;
-        }
+  beforeAll(async () => {
+    browser = await launchBrowser(providerConfig.kleinanzeigen.url);
+  }, TEST_TIMEOUT);
 
-        expect(listing).toBeInstanceOf(Array);
-        const notificationObj = get();
-        expect(notificationObj).toBeTypeOf('object');
-        expect(notificationObj.serviceName).toBe('kleinanzeigen');
-        notificationObj.payload.forEach((notify) => {
-          /** check the actual structure **/
-          expect(notify.id).toBeTypeOf('string');
-          expect(notify.title).toBeTypeOf('string');
-          expect(notify.link).toBeTypeOf('string');
-          expect(notify.address).toBeTypeOf('string');
-          /** check the values if possible **/
-          expect(notify.title).not.toBe('');
-          expect(notify.link).toContain('https://www.kleinanzeigen.de');
-          expect(notify.address).not.toBe('');
-        });
-        resolve();
-      });
-    });
+  afterAll(async () => {
+    await closeBrowser(browser);
   });
 
+  it(
+    'should test kleinanzeigen provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'kleinanzeigen',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+      provider.init(providerConfig.kleinanzeigen, [], []);
+      return await new Promise((resolve, reject) => {
+        const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+        fredy.execute().then((listing) => {
+          if (listing == null || listing.length === 0) {
+            reject('Listings is empty!');
+            return;
+          }
+
+          liveListings = listing;
+          expect(listing).toBeInstanceOf(Array);
+          const notificationObj = get();
+          expect(notificationObj).toBeTypeOf('object');
+          expect(notificationObj.serviceName).toBe('kleinanzeigen');
+          notificationObj.payload.forEach((notify) => {
+            /** check the actual structure **/
+            expect(notify.id).toBeTypeOf('string');
+            expect(notify.title).toBeTypeOf('string');
+            expect(notify.link).toBeTypeOf('string');
+            expect(notify.address).toBeTypeOf('string');
+            /** check the values if possible **/
+            expect(notify.title).not.toBe('');
+            expect(notify.link).toContain('https://www.kleinanzeigen.de');
+            expect(notify.address).not.toBe('');
+          });
+          resolve();
+        });
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
-    });
+    it(
+      'should enrich listings with details',
+      async () => {
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
-      vi.restoreAllMocks();
-    });
+        // Call fetchDetails directly on the first live listing — no need to
+        // re-scrape the search page. The shared browser keeps the session warm.
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
-      const Fredy = await mockFredy();
-      provider.init(providerConfig.kleinanzeigen, [], []);
-      const mockedJob = { id: 'kleinanzeigen', notificationAdapter: null, specFilter: null, spatialFilter: null };
-
-      const fredy = new Fredy(
-        provider.config,
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.kleinanzeigen.de');
-        expect(listing.address).toBeTypeOf('string');
-        expect(listing.address).not.toBe('');
-        expect(listing.description).toBeTypeOf('string');
-        expect(listing.description).not.toBe('');
-      });
-    });
+        expect(enriched).toBeTruthy();
+        expect(enriched.link).toContain('https://www.kleinanzeigen.de');
+        expect(enriched.address).toBeTypeOf('string');
+        expect(enriched.address).not.toBe('');
+        expect(enriched.description).toBeTypeOf('string');
+        expect(enriched.description).not.toBe('');
+      },
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/sparkasse.test.js

@@ -9,81 +9,97 @@ import { mockFredy, providerConfig } from '../utils.js';
 import { expect, vi } from 'vitest';
 import * as provider from '../../lib/provider/sparkasse.js';
 import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session. This prevents the second request from being
+// flagged as a cold-start bot hit.
+const TEST_TIMEOUT = 120_000;
 
 describe('#sparkasse testsuite()', () => {
-  it('should test sparkasse provider', async () => {
-    const Fredy = await mockFredy();
-    const mockedJob = {
-      id: 'sparkasse',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
-    provider.init(providerConfig.sparkasse, []);
+  let browser;
+  let liveListings;
 
-    const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  beforeAll(async () => {
+    browser = await launchBrowser(providerConfig.sparkasse.url);
+  }, TEST_TIMEOUT);
 
-    const listing = await fredy.execute();
-
-    if (listing == null || listing.length === 0) {
-      throw new Error('Listings is empty!');
-    }
-
-    expect(listing).toBeInstanceOf(Array);
-    const notificationObj = get();
-    expect(notificationObj).toBeTypeOf('object');
-    expect(notificationObj.serviceName).toBe('sparkasse');
-    notificationObj.payload.forEach((notify) => {
-      /** check the actual structure **/
-      expect(notify.id).toBeTypeOf('string');
-      expect(notify.price).toBeTypeOf('string');
-      expect(notify.price).toContain('€');
-      expect(notify.size).toBeTypeOf('string');
-      expect(notify.size).toContain('m²');
-      expect(notify.title).toBeTypeOf('string');
-      expect(notify.link).toBeTypeOf('string');
-      expect(notify.address).toBeTypeOf('string');
-      /** check the values if possible **/
-      expect(notify.size).toBeTypeOf('string');
-      expect(notify.title).not.toBe('');
-      expect(notify.address).not.toBe('');
-    });
+  afterAll(async () => {
+    await closeBrowser(browser);
   });
 
+  it(
+    'should test sparkasse provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'sparkasse',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+      provider.init(providerConfig.sparkasse, []);
+
+      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+      liveListings = await fredy.execute();
+
+      if (liveListings == null || liveListings.length === 0) {
+        throw new Error('Listings is empty!');
+      }
+
+      expect(liveListings).toBeInstanceOf(Array);
+      const notificationObj = get();
+      expect(notificationObj).toBeTypeOf('object');
+      expect(notificationObj.serviceName).toBe('sparkasse');
+      notificationObj.payload.forEach((notify) => {
+        /** check the actual structure **/
+        expect(notify.id).toBeTypeOf('string');
+        expect(notify.price).toBeTypeOf('string');
+        expect(notify.price).toContain('€');
+        expect(notify.size).toBeTypeOf('string');
+        expect(notify.size).toContain('m²');
+        expect(notify.title).toBeTypeOf('string');
+        expect(notify.link).toBeTypeOf('string');
+        expect(notify.address).toBeTypeOf('string');
+        /** check the values if possible **/
+        expect(notify.size).toBeTypeOf('string');
+        expect(notify.title).not.toBe('');
+        expect(notify.address).not.toBe('');
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
     beforeEach(() => {
       vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
     });
 
     afterEach(() => {
       vi.restoreAllMocks();
     });
 
-    it('should enrich listings with details', async () => {
-      const Fredy = await mockFredy();
-      provider.init(providerConfig.sparkasse, []);
-      const mockedJob = { id: 'sparkasse', notificationAdapter: null, specFilter: null, spatialFilter: null };
+    it(
+      'should enrich listings with details',
+      async () => {
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-      const fredy = new Fredy(
-        provider.config,
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://immobilien.sparkasse.de');
-        expect(listing.address).toBeTypeOf('string');
-        expect(listing.address).not.toBe('');
-        // description is enriched from the detail page; falls back gracefully if bot-detected
-        if (listing.description != null) {
-          expect(listing.description).toBeTypeOf('string');
-          expect(listing.description).not.toBe('');
+        // Call fetchDetails directly on the first live listing — no need to
+        // re-scrape the search page. The shared browser keeps the session warm.
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
+
+        expect(enriched).toBeTruthy();
+        expect(enriched.link).toContain('https://immobilien.sparkasse.de');
+        expect(enriched.address).toBeTypeOf('string');
+        expect(enriched.address).not.toBe('');
+        // description is enriched from the detail page; falls back gracefully if blocked
+        if (enriched.description != null) {
+          expect(enriched.description).toBeTypeOf('string');
+          expect(enriched.description).not.toBe('');
         }
-      });
-    });
+      },
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/wgGesucht.test.js

@@ -6,77 +6,85 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { mockFredy, providerConfig } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/wgGesucht.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session, avoiding double cold-start bot detection.
+const TEST_TIMEOUT = 120_000;
 
 describe('#wgGesucht testsuite()', () => {
   provider.init(providerConfig.wgGesucht, [], []);
-  it('should test wgGesucht provider', { timeout: 120000 }, async () => {
-    const Fredy = await mockFredy();
-    const mockedJob = {
-      id: 'wgGesucht',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
 
-    return await new Promise((resolve, reject) => {
-      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  let browser;
+  let liveListings;
 
-      fredy.execute().then((listing) => {
-        if (listing == null || listing.length === 0) {
-          reject('Listings is empty!');
-          return;
-        }
+  beforeAll(async () => {
+    browser = await launchBrowser(providerConfig.wgGesucht.url);
+  }, TEST_TIMEOUT);
 
-        expect(listing).toBeInstanceOf(Array);
-        const notificationObj = get();
-        expect(notificationObj.serviceName).toBe('wgGesucht');
-        notificationObj.payload.forEach((notify) => {
-          expect(notify).toBeTypeOf('object');
-          /** check the actual structure **/
-          expect(notify.id).toBeTypeOf('string');
-          expect(notify.title).toBeTypeOf('string');
-          // expect(notify.details).toBeTypeOf('string');
-          expect(notify.price).toBeTypeOf('string');
-          expect(notify.price).toContain('€');
-          expect(notify.link).toBeTypeOf('string');
-        });
-        resolve();
-      });
-    });
+  afterAll(async () => {
+    await closeBrowser(browser);
   });
 
+  it(
+    'should test wgGesucht provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'wgGesucht',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+
+      return await new Promise((resolve, reject) => {
+        const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+        fredy.execute().then((listing) => {
+          if (listing == null || listing.length === 0) {
+            reject('Listings is empty!');
+            return;
+          }
+
+          liveListings = listing;
+          expect(listing).toBeInstanceOf(Array);
+          const notificationObj = get();
+          expect(notificationObj.serviceName).toBe('wgGesucht');
+          notificationObj.payload.forEach((notify) => {
+            expect(notify).toBeTypeOf('object');
+            /** check the actual structure **/
+            expect(notify.id).toBeTypeOf('string');
+            expect(notify.title).toBeTypeOf('string');
+            // expect(notify.details).toBeTypeOf('string');
+            expect(notify.price).toBeTypeOf('string');
+            expect(notify.price).toContain('€');
+            expect(notify.link).toBeTypeOf('string');
+          });
+          resolve();
+        });
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
-    });
+    it(
+      'should enrich listings with details',
+      async () => {
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
-      vi.restoreAllMocks();
-    });
+        // Call fetchDetails directly on the first live listing — no need to
+        // re-scrape the search page. The shared browser keeps the session warm.
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
-      const Fredy = await mockFredy();
-      provider.init(providerConfig.wgGesucht, [], []);
-      const mockedJob = { id: 'wgGesucht', notificationAdapter: null, specFilter: null, spatialFilter: null };
-
-      const fredy = new Fredy(
-        provider.config,
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.wg-gesucht.de');
-        expect(listing.description).toBeTypeOf('string');
-        expect(listing.description).not.toBe('');
-      });
-    });
+        expect(enriched).toBeTruthy();
+        expect(enriched.link).toContain('https://www.wg-gesucht.de');
+        expect(enriched.description).toBeTypeOf('string');
+        expect(enriched.description).not.toBe('');
+      },
+      TEST_TIMEOUT,
+    );
   });
 });

test/services/extractor/puppeteerExtractor.test.js

@@ -0,0 +1,37 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { vi, describe, it, expect, beforeEach } from 'vitest';
+
+// Mock the CloakBrowser launcher so no real Chromium binary is needed and we can
+// assert which options get forwarded to it.
+const { launchMock } = vi.hoisted(() => ({ launchMock: vi.fn() }));
+
+vi.mock('cloakbrowser/puppeteer', () => ({
+  launch: launchMock,
+}));
+
+const { launchBrowser } = await import('../../../lib/services/extractor/puppeteerExtractor.js');
+
+describe('launchBrowser proxy forwarding', () => {
+  beforeEach(() => {
+    launchMock.mockReset();
+    launchMock.mockResolvedValue({ close: async () => {} });
+  });
+
+  it('forwards proxyUrl to CloakBrowser as the proxy option', async () => {
+    await launchBrowser('https://www.immowelt.de/', { proxyUrl: 'http://user:pass@host:8080' });
+
+    expect(launchMock).toHaveBeenCalledTimes(1);
+    expect(launchMock.mock.calls[0][0]).toMatchObject({ proxy: 'http://user:pass@host:8080' });
+  });
+
+  it('does not set a proxy when no proxyUrl is given', async () => {
+    await launchBrowser('https://www.immowelt.de/', {});
+
+    expect(launchMock).toHaveBeenCalledTimes(1);
+    expect(launchMock.mock.calls[0][0].proxy).toBeUndefined();
+  });
+});

test/services/immoscout/immoscout-web-translator.test.js

@@ -10,6 +10,17 @@ import { readFile } from 'fs/promises';
 export const testData = JSON.parse(await readFile(new URL('./testdata.json', import.meta.url)));
 
 describe('#immoscout-mobile URL conversion', () => {
+  // Test shape URL conversion
+  it('should convert a full web URL with shape to mobile URL', () => {
+    const webUrl =
+      'https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list';
+    const expectedMobileUrl =
+      'https://api.mobile.immobilienscout24.de/search/list?ground=240.0-&price=-600000.0&realestatetype=housebuy&searchType=shape&shape=ior~H_kxmAr%60Pig%60%40fzHm%7Bp%40qs%40%7D%60l%40crCidPuol%40wy~%40onVb~Ek%60KhdPcoEhkS%7CHf%7B%7CAtpDtqLjiGngQ%7CxLrlOxyVbzR';
+
+    const actualMobileUrl = convertWebToMobile(webUrl);
+    expect(actualMobileUrl).toBe(expectedMobileUrl);
+  });
+
   // Test URL conversion
   it('should convert a full web URL to mobile URL', () => {
     const webUrl =

test/services/immoscout/testdata.json

@@ -18,5 +18,9 @@
   "rentHouse": {
     "url": "https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/haus-mieten?enteredFrom=one_step_search",
     "type": "houserent"
+  },
+  "buyHouseWithShape": {
+    "url": "https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list",
+    "type": "housebuy"
   }
 }

test/services/jobs/jobExecutionService.test.js

@@ -18,6 +18,7 @@ describe('services/jobs/jobExecutionService', () => {
     const busPath = root + '/lib/services/events/event-bus.js';
     const jobStoragePath = root + '/lib/services/storage/jobStorage.js';
     const userStoragePath = root + '/lib/services/storage/userStorage.js';
+    const settingsStoragePath = root + '/lib/services/storage/settingsStorage.js';
     const brokerPath = root + '/lib/services/sse/sse-broker.js';
     const utilsPath = root + '/lib/utils.js';
     const loggerPath = root + '/lib/services/logger.js';
@@ -33,11 +34,15 @@ describe('services/jobs/jobExecutionService', () => {
       getUsers: () => state.users.slice(),
       getUser: (id) => state.users.find((u) => u.id === id) || null,
     }));
+    vi.doMock(settingsStoragePath, () => ({
+      getSettings: async () => ({}),
+    }));
     vi.doMock(brokerPath, () => ({
       sendToUsers: (...args) => calls.sent.push(args),
     }));
     vi.doMock(utilsPath, () => ({
       duringWorkingHoursOrNotSet: () => false,
+      getPackageVersion: async () => '0.0.0-test',
     }));
     vi.doMock(loggerPath, () => {
       const m = { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} };

test/utils.js

@@ -29,7 +29,7 @@ vi.mock('../lib/services/extractor/puppeteerExtractor.js', async (importOriginal
   const { readFixture } = await import('./offlineFixtures.js');
   return {
     default: (url) => readFixture(url),
-    launchBrowser: async () => ({ close: async () => {}, __fredy_removeUserDataDir: false }),
+    launchBrowser: async () => ({ close: async () => {}, isConnected: () => true }),
     closeBrowser: async () => {},
   };
 });

tools/testFixtures/downloadFixtures.js

@@ -95,7 +95,10 @@ async function downloadHtmlProvider(name, providerConfig, launchBrowser, closeBr
 
   const browser = await launchBrowser(providerConfig.url, {});
   try {
-    const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, { browser });
+    const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, {
+      browser,
+      name: 'dowload_fixtures',
+    });
 
     if (!html) {
       console.warn(`  Failed to download ${name}`);

ui/src/components/grid/jobs/JobGrid.jsx

@@ -174,7 +174,7 @@ const JobGrid = () => {
       Toast.success('Job status successfully changed');
       loadData();
     } catch (error) {
-      Toast.error(error);
+      Toast.error(error.error);
     }
   };
 

ui/src/components/listings/ListingsOverview.jsx

@@ -197,7 +197,7 @@ const ListingsOverview = () => {
 
         <Button
           icon={sortDir === 'asc' ? <IconArrowUp /> : <IconArrowDown />}
-          onClick={() => setSortDir((d) => (d === 'asc' ? 'desc' : 'asc'))}
+          onClick={() => setSortDir(sortDir === 'asc' ? 'desc' : 'asc')}
           title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
         />
 

ui/src/components/map/Map.jsx

@@ -8,6 +8,7 @@ import maplibregl from 'maplibre-gl';
 import 'maplibre-gl/dist/maplibre-gl.css';
 import '@mapbox/mapbox-gl-draw/dist/mapbox-gl-draw.css';
 import { fixMapboxDrawCompatibility, addDrawingControl, setupAreaFilterEventListeners } from './MapDrawingExtension.js';
+import { getBoundsFromCoords } from '../../views/listings/mapUtils.js';
 import './Map.less';
 
 export const GERMANY_BOUNDS = [
@@ -66,6 +67,7 @@ export default function Map({
   const mapContainerRef = useRef(null);
   const mapRef = useRef(null);
   const drawRef = useRef(null);
+  const hasFittedToInitialAreaRef = useRef(false);
 
   // Initialize map - ONLY when container changes, never reinitialize
   useEffect(() => {
@@ -128,6 +130,17 @@ export default function Map({
       } catch (error) {
         console.error('Error loading spatial filter:', error);
       }
+
+      if (!hasFittedToInitialAreaRef.current) {
+        const coords = initialSpatialFilter.features.flatMap((feature) =>
+          feature.geometry?.type === 'Polygon' ? feature.geometry.coordinates.flat() : [],
+        );
+        const bounds = getBoundsFromCoords(coords);
+        if (bounds) {
+          mapRef.current.fitBounds(bounds, { padding: 50, maxZoom: 15, duration: 0 });
+          hasFittedToInitialAreaRef.current = true;
+        }
+      }
     }
 
     // Setup drawing event listeners

ui/src/views/generalSettings/GeneralSettings.jsx

@@ -54,8 +54,10 @@ const GeneralSettings = function GeneralSettings() {
   const [loading, setLoading] = React.useState(true);
 
   const settings = useSelector((state) => state.generalSettings.settings);
+  const currentUser = useSelector((state) => state.user.currentUser);
 
   const [interval, setInterval] = React.useState('');
+  const [proxyUrl, setProxyUrl] = React.useState('');
   const [port, setPort] = React.useState('');
   const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
   const [workingHourTo, setWorkingHourTo] = React.useState(null);
@@ -90,6 +92,7 @@ const GeneralSettings = function GeneralSettings() {
   React.useEffect(() => {
     async function init() {
       setInterval(settings?.interval);
+      setProxyUrl(settings?.proxyUrl ?? '');
       setPort(settings?.port);
       setWorkingHourFrom(settings?.workingHours?.from);
       setWorkingHourTo(settings?.workingHours?.to);
@@ -132,6 +135,7 @@ const GeneralSettings = function GeneralSettings() {
     try {
       await xhrPost('/api/admin/generalSettings', {
         interval,
+        proxyUrl: proxyUrl?.trim() ?? '',
         port,
         workingHours: {
           from: workingHourFrom,
@@ -375,6 +379,18 @@ const GeneralSettings = function GeneralSettings() {
                   </div>
                 </SegmentPart>
 
+                <SegmentPart
+                  name="Proxy URL"
+                  helpText="Optional. Routes the scraping browser through a proxy. Server/datacenter IPs are frequently blocked by providers (e.g. immowelt) regardless of browser fingerprint, a German residential proxy makes requests look like a normal household and is the most effective fix. Format: http://user:pass@host:port or socks5://user:pass@host:port. Leave empty to disable."
+                >
+                  <Input
+                    type="text"
+                    placeholder="http://user:pass@host:port"
+                    value={proxyUrl}
+                    onChange={(value) => setProxyUrl(value)}
+                  />
+                </SegmentPart>
+
                 <div className="generalSettings__save-row">
                   <Button type="primary" theme="solid" onClick={handleStore} icon={<IconSave />}>
                     Save
@@ -467,12 +483,26 @@ const GeneralSettings = function GeneralSettings() {
               itemKey="backup"
             >
               <div className="generalSettings__tab-content">
+                {demoMode && !currentUser?.isAdmin && (
+                  <Banner
+                    fullMode={false}
+                    type="warning"
+                    closeIcon={null}
+                    style={{ marginBottom: '12px' }}
+                    description="Backup and restore are not available in demo mode."
+                  />
+                )}
                 <SegmentPart
                   name="Backup & Restore"
                   helpText="Download a zipped backup of your database or restore from a backup zip."
                 >
                   <div style={{ display: 'flex', gap: '0.5rem', alignItems: 'center', flexWrap: 'wrap' }}>
-                    <Button theme="solid" icon={<IconSave />} onClick={handleDownloadBackup}>
+                    <Button
+                      theme="solid"
+                      icon={<IconSave />}
+                      onClick={handleDownloadBackup}
+                      disabled={demoMode && !currentUser?.isAdmin}
+                    >
                       Download Backup
                     </Button>
                     <input
@@ -482,7 +512,12 @@ const GeneralSettings = function GeneralSettings() {
                       style={{ display: 'none' }}
                       onChange={handleSelectRestoreFile}
                     />
-                    <Button onClick={handleOpenFilePicker} theme="light" icon={<IconFolder />}>
+                    <Button
+                      onClick={handleOpenFilePicker}
+                      theme="light"
+                      icon={<IconFolder />}
+                      disabled={demoMode && !currentUser?.isAdmin}
+                    >
                       Restore from Zip
                     </Button>
                   </div>

ui/src/views/jobs/mutation/components/provider/ProviderMutator.jsx

@@ -141,21 +141,6 @@ export default function ProviderMutator({
           </p>
         </>
       )}
-      <Banner
-        fullMode={false}
-        type="warning"
-        closeIcon={null}
-        title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Warning</div>}
-        style={{ marginBottom: '1rem' }}
-        description={
-          <div>
-            <p>
-              Currently, our Immoscout implementation does not support drawing shapes on a map. Use a radius instead.
-            </p>
-          </div>
-        }
-      />
-
       <Select
         filter
         placeholder="Select a provider"

ui/src/views/user/Users.jsx

@@ -37,7 +37,7 @@ const Users = function Users() {
       await actions.jobsData.getJobs();
       await actions.user.getUsers();
     } catch (error) {
-      Toast.error(error);
+      Toast.error(error.error);
       setUserIdToBeRemoved(null);
     }
   };

vitest.config.js

@@ -10,6 +10,7 @@ export default defineConfig({
     globals: true,
     environment: 'node',
     include: ['test/**/*.test.js'],
+    globalSetup: ['./test/globalSetup.js'],
     testTimeout: 60000,
     reporters: ['verbose'],
   },