fixing #319 & #318

* feat: remember listing delete preference

Persist soft/hard choice and skip-confirm in user settings.

Dockerfile

@@ -1,16 +1,15 @@
 FROM node:22-slim
 
-ARG TARGETARCH
+# System deps for CloakBrowser + build tools for native modules (better-sqlite3)
-
+# fonts-noto-color-emoji and fonts-freefont-ttf are required so canvas fingerprint
-# System deps for Chrome for Testing + build tools for native modules (better-sqlite3)
+# hashes match real browsers; missing emoji fonts cause bot detection on Kasada/Akamai.
-# On ARM64 we also install system Chromium (Chrome for Testing has no ARM64 binary)
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl ca-certificates fonts-liberation libasound2 \
     libatk-bridge2.0-0 libatk1.0-0 libcups2 libdbus-1-3 \
     libdrm2 libgbm1 libgtk-3-0 libnspr4 libnss3 \
     libx11-xcb1 libxcomposite1 libxdamage1 libxrandr2 xdg-utils \
+    fonts-noto-color-emoji fonts-freefont-ttf \
     python3 make g++ \
-  && if [ "$TARGETARCH" = "arm64" ]; then apt-get install -y --no-install-recommends chromium; fi \
   && rm -rf /var/lib/apt/lists/* \
   && mkdir -p /db /conf /fredy
 
@@ -26,8 +25,8 @@ RUN yarn config set network-timeout 600000 \
   && yarn --frozen-lockfile \
   && yarn cache clean
 
-# on arm64 use the system Chromium installed above
+# Pre-download the CloakBrowser stealth Chromium binary (supports x86_64 and arm64)
-RUN if [ "$TARGETARCH" != "arm64" ]; then npx puppeteer browsers install chrome; fi
+RUN node -e "import('cloakbrowser').then(({ensureBinary}) => ensureBinary())"
 
 # Purge build tools now that native modules are compiled
 RUN apt-get purge -y python3 make g++ \

README.md

@@ -23,7 +23,7 @@
 
 
 
-# Fredy 🏡 – Your Self-Hosted Real Estate Finder for Germany
+# Fredy 🏡 - Your Self-Hosted Real Estate Finder for Germany
 
 Finding an apartment or house in Germany can be stressful and
 time-consuming.\
@@ -167,6 +167,40 @@ For more information on how to set it up and use it, please refer to the [MCP Re
 
 Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
 
+## 🛡️ Bot Detection & Proxies
+
+Most browser-based providers (immowelt, immonet, kleinanzeigen, ...) are scraped through a hardened headless browser ([CloakBrowser](https://www.npmjs.com/package/cloakbrowser)). It makes the **browser fingerprint** indistinguishable from a real Chrome, which is enough when you run Fredy on a normal home connection.
+
+On a **server / VPS the requests usually originate from a datacenter IP**, and providers behind anti-bot systems (e.g. AWS CloudFront/WAF) block those based on **IP reputation alone**, no matter how perfect the fingerprint is. The typical symptom: it works locally but you get `We have been detected as a bot :-/` on the server.
+
+### The fix: a residential proxy
+
+A **residential proxy** routes Fredy's browser through the internet connection of a real household, so the provider sees a "normal user" IP instead of a datacenter. For German portals, use a **German (DE) residential** (or mobile/4G) proxy. Plain VPNs and **datacenter proxies do not help** here, they share the same bad reputation as your server.
+
+**Configure it** under **Settings → Execution → Proxy URL**. Supported formats:
+
+```
+http://user:pass@host:port
+socks5://user:pass@host:port
+```
+
+Leave the field empty to disable. The proxy applies to all headless-browser providers and takes effect on the next job run (no restart needed). Immoscout uses a separate mobile API and is not affected.
+
+### Where to get a residential proxy
+
+Residential proxies are a paid service (usually billed per GB, Fredy's traffic is small). Well-known providers offering German residential IPs include:
+
+| Provider | Notes |
+|---|---|
+| [IPRoyal](https://iproyal.com) | Pay-as-you-go, no monthly minimum, good for low volume |
+| [Webshare](https://www.webshare.io) | Cheap entry tier, has a small free plan to test with |
+| [Decodo (formerly Smartproxy)](https://decodo.com) | Easy setup, country/city targeting |
+| [SOAX](https://soax.com) | Residential + mobile, fine-grained geo-targeting |
+| [Bright Data](https://brightdata.com) | Largest pool, most features, higher complexity/price |
+| [Oxylabs](https://oxylabs.io) | Enterprise-grade, larger plans |
+
+This is not an endorsement, pick whatever fits your budget. For low-volume use like Fredy, a pay-as-you-go plan (e.g. IPRoyal) or a cheap entry tier (e.g. Webshare) is usually plenty. Make sure to select **Germany** as the proxy location and keep the search interval reasonable (the higher the interval, the less you look like a bot).
+
 ## Analytics
 
 Fredy is completely free (and will always remain free). However, it would be a huge help if you’d allow me to collect some analytical data.

index.js

@@ -15,6 +15,15 @@ import { initGeocodingCron } from './lib/services/crons/geocoding-cron.js';
 import { getSettings } from './lib/services/storage/settingsStorage.js';
 import SqliteConnection, { computeDbPath } from './lib/services/storage/SqliteConnection.js';
 import { initJobExecutionService } from './lib/services/jobs/jobExecutionService.js';
+import { ensureValidBinary } from './lib/services/ensureValidBinary.js';
+
+// Ensure the CloakBrowser stealth Chromium binary is present and complete before
+// jobs run.  ensureValidBinary() also detects and auto-heals partial extractions
+// (e.g. a newer version that was downloaded but only the chrome executable was
+// written) so Chrome never crashes with "Invalid file descriptor to ICU data".
+logger.info('Checking CloakBrowser binary...');
+await ensureValidBinary();
+logger.info('CloakBrowser binary ready.');
 
 //in the config, we store the path of the sqlite file, thus we must check if it is available
 const isConfigAccessible = await checkIfConfigIsAccessible();

lib/FredyPipelineExecutioner.js

@@ -5,9 +5,10 @@
 
 import { NoNewListingsWarning } from './errors.js';
 import {
-  storeListings,
-  getKnownListingHashesForJobAndProvider,
   deleteListingsById,
+  getKnownListingHashesForJobAndProvider,
+  storeListings,
+  updateListingDistance,
 } from './services/storage/listingsStorage.js';
 import { getJob } from './services/storage/jobStorage.js';
 import * as notify from './notification/notify.js';
@@ -16,8 +17,7 @@ import urlModifier from './services/queryStringMutator.js';
 import logger from './services/logger.js';
 import { geocodeAddress } from './services/geocoding/geoCodingService.js';
 import { distanceMeters } from './services/listings/distanceCalculator.js';
-import { getUserSettings, getSettings } from './services/storage/settingsStorage.js';
+import { getSettings, getUserSettings } from './services/storage/settingsStorage.js';
-import { updateListingDistance } from './services/storage/listingsStorage.js';
 import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
 import { formatListing } from './utils/formatListing.js';
 
@@ -97,9 +97,9 @@ class FredyPipelineExecutioner {
   }
 
   /**
-   * Optionally enrich new listings with data from their detail pages.
+   * Optionally, enrich new listings with data from their detail pages.
    * Only called when the provider config defines a `fetchDetails` function.
-   * Runs all fetches in parallel. Each individual fetch must handle its own errors
+   * Runs all fetches in parallel. Each fetch must handle its own errors
    * and always resolve (never reject) to avoid aborting other listings.
    *
    * @param {Listing[]} newListings New listings to enrich.
@@ -132,7 +132,7 @@ class FredyPipelineExecutioner {
     for (const listing of newListings) {
       if (listing.address) {
         const coords = await geocodeAddress(listing.address);
-        if (coords) {
+        if (coords && coords.lat !== -1 && coords.lng !== -1) {
           listing.latitude = coords.lat;
           listing.longitude = coords.lng;
         }
@@ -227,7 +227,7 @@ class FredyPipelineExecutioner {
     const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
     return new Promise((resolve, reject) => {
       extractor
-        .execute(url, this._providerConfig.waitForSelector)
+        .execute(url, this._providerConfig.waitForSelector, this._providerId)
         .then(() => {
           const listings = extractor.parseResponseText(
             this._providerConfig.crawlContainer,
@@ -264,15 +264,15 @@ class FredyPipelineExecutioner {
     const requiredKeys = this._providerConfig.requiredFieldNames;
     const requireValues = ['id', 'link', 'title'];
 
-    const filteredListings = listings
+    return (
-      // this should never filter some listings out, because the normalize function should always extract all fields.
+      listings
-      .filter((item) => requiredKeys.every((key) => key in item))
+        // this should never filter some listings out, because the normalize function should always extract all fields.
-      // TODO: move blacklist filter to this file, so it will handle for all providers in same way.
+        .filter((item) => requiredKeys.every((key) => key in item))
-      .filter(this._providerConfig.filter)
+        // TODO: move blacklist filter to this file, so it will handle for all providers in same way.
-      // filter out listings that are missing required fields
+        .filter(this._providerConfig.filter)
-      .filter((item) => requireValues.every((key) => item[key] != null));
+        // filter out listings that are missing required fields
-
+        .filter((item) => requireValues.every((key) => item[key] != null))
-    return filteredListings;
+    );
   }
 
   /**

lib/TRACKING_POIS.js

@@ -7,4 +7,9 @@ export const TRACKING_POIS = {
   DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
   WELCOME_FINISHED: 'WELCOME_FINISHED',
   WELCOME_SKIPPED: 'WELCOME_SKIPPED',
+  JOBS_TABLE_VIEW: 'JOBS_TABLE_VIEW',
+  LISTING_TABLE_VIEW: 'LISTING_TABLE_VIEW',
+  BASE_URL_SETTING: 'BASE_URL_SETTING',
+  SET_PROXY_SETTING: 'SET_PROXY_SETTING',
+  DETECTED_AS_BOT: 'DETECTED_AS_BOT',
 };

lib/api/api.js

@@ -76,13 +76,13 @@ fastify.register(async (app) => {
   app.register(dashboardPlugin, { prefix: '/api/dashboard' });
   app.register(userSettingsPlugin, { prefix: '/api/user/settings' });
   app.register(trackingPlugin, { prefix: '/api/tracking' });
+  app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
 });
 
 // Admin-only routes
 fastify.register(async (app) => {
   app.addHook('preHandler', authHook);
   app.addHook('preHandler', adminHook);
-  app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
   app.register(backupPlugin, { prefix: '/api/admin/backup' });
   app.register(userPlugin, { prefix: '/api/admin/users' });
 });

lib/api/routes/backupRouter.js

@@ -9,6 +9,10 @@ import {
   precheckRestore,
   restoreFromZip,
 } from '../../services/storage/backupRestoreService.js';
+import { getSettings } from '../../services/storage/settingsStorage.js';
+import { isAdmin } from '../security.js';
+
+const DEMO_MODE_ERROR = 'Backup and restore are not available in demo mode.';
 
 /**
  * @param {import('fastify').FastifyInstance} fastify
@@ -21,7 +25,11 @@ export default async function backupPlugin(fastify) {
     (req, body, done) => done(null, body),
   );
 
-  fastify.get('/', async (_request, reply) => {
+  fastify.get('/', async (request, reply) => {
+    const settings = await getSettings();
+    if (settings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: DEMO_MODE_ERROR });
+    }
     const zipBuffer = await createBackupZip();
     const fileName = await buildBackupFileName();
     reply.header('Content-Type', 'application/zip');
@@ -30,6 +38,10 @@ export default async function backupPlugin(fastify) {
   });
 
   fastify.post('/restore', async (request, reply) => {
+    const settings = await getSettings();
+    if (settings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: DEMO_MODE_ERROR });
+    }
     const { dryRun = 'false', force = 'false' } = request.query || {};
     const doDryRun = String(dryRun) === 'true';
     const doForce = String(force) === 'true';

lib/api/routes/generalSettingsRoute.js

@@ -9,6 +9,8 @@ import { ensureDemoUserExists } from '../../services/storage/userStorage.js';
 import logger from '../../services/logger.js';
 import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
 import { isAdmin } from '../security.js';
+import { trackPoi } from '../../services/tracking/Tracker.js';
+import { TRACKING_POIS } from '../../TRACKING_POIS.js';
 
 /**
  * @param {import('fastify').FastifyInstance} fastify
@@ -25,16 +27,26 @@ export default async function generalSettingsPlugin(fastify) {
     }
     const localSettings = await getSettings();
 
-    if (localSettings.demoMode && !isAdmin(request)) {
+    if (!isAdmin(request)) {
-      return reply.code(403).send({ error: 'In demo mode, it is not allowed to change these settings.' });
+      const reason = localSettings.demoMode
+        ? 'In demo mode, it is not allowed to change these settings.'
+        : 'Only admins can change these settings.';
+      return reply.code(403).send({ error: reason });
     }
 
     try {
       if (typeof sqlitepath !== 'undefined') {
         fs.writeFileSync(`${getDirName()}/../conf/config.json`, JSON.stringify({ sqlitepath }));
       }
+
       upsertSettings(appSettings);
       ensureDemoUserExists();
+      if (appSettings.baseUrl != null) {
+        await trackPoi(TRACKING_POIS.BASE_URL_SETTING);
+      }
+      if (appSettings.proxyUrl != null) {
+        await trackPoi(TRACKING_POIS.SET_PROXY_SETTING);
+      }
     } catch (err) {
       logger.error(err);
       return reply.code(500).send({ error: 'Error while trying to write settings.' });

lib/api/routes/notificationAdapterRouter.js

@@ -18,7 +18,7 @@ const notificationAdapter = await Promise.all(
  */
 export default async function notificationAdapterPlugin(fastify) {
   fastify.get('/', async () => {
-    return notificationAdapter.map((adapter) => adapter.config);
+    return notificationAdapter.map((adapter) => adapter.config).filter(Boolean);
   });
 
   fastify.post('/try', async (request, reply) => {

lib/api/routes/userSettingsRoute.js

@@ -118,6 +118,10 @@ export default async function userSettingsPlugin(fastify) {
       return reply.code(400).send({ error: 'listings_view_mode must be "grid" or "table".' });
     }
 
+    if (listings_view_mode === 'table') {
+      await trackPoi(TRACKING_POIS.LISTING_TABLE_VIEW);
+    }
+
     try {
       upsertSettings({ listings_view_mode }, userId);
       return { success: true };
@@ -126,4 +130,49 @@ export default async function userSettingsPlugin(fastify) {
       return reply.code(500).send({ error: error.message });
     }
   });
+
+  fastify.post('/jobs-view-mode', async (request, reply) => {
+    const userId = request.session.currentUser;
+    const { jobs_view_mode } = request.body;
+
+    if (jobs_view_mode !== 'grid' && jobs_view_mode !== 'table') {
+      return reply.code(400).send({ error: 'jobs_view_mode must be "grid" or "table".' });
+    }
+
+    if (jobs_view_mode === 'table') {
+      await trackPoi(TRACKING_POIS.JOBS_TABLE_VIEW);
+    }
+
+    try {
+      upsertSettings({ jobs_view_mode }, userId);
+      return { success: true };
+    } catch (error) {
+      logger.error('Error updating jobs view mode setting', error);
+      return reply.code(500).send({ error: error.message });
+    }
+  });
+
+  fastify.post('/listing-deletion-preference', async (request, reply) => {
+    const userId = request.session.currentUser;
+    const { listing_deletion_preference } = request.body;
+
+    const globalSettings = await getSettings();
+    if (globalSettings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
+    }
+
+    if (listing_deletion_preference == null) {
+      return reply.code(400).send({ error: 'listing_deletion_preference is required.' });
+    }
+
+    const { skipPrompt, hardDelete } = listing_deletion_preference;
+
+    try {
+      upsertSettings({ listing_deletion_preference: { skipPrompt, hardDelete } }, userId);
+      return { success: true };
+    } catch (error) {
+      logger.error('Error updating listing deletion preference', error);
+      return reply.code(500).send({ error: error.message });
+    }
+  });
 }

lib/notification/adapter/telegram.js

@@ -9,6 +9,7 @@ import fetch from 'node-fetch';
 import pThrottle from 'p-throttle';
 import { normalizeImageUrl } from '../../utils.js';
 import logger from '../../services/logger.js';
+import { shouldUseMultipart, buildPhotoFormData } from './telegramPhotoUploader.js';
 
 const RATE_LIMIT_INTERVAL = 1000;
 const chatThrottleMap = new Map();
@@ -177,11 +178,13 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
   const jobName = job == null ? jobKey : job.name;
 
   const throttledCall = getThrottled(chatId, async function (endpoint, body) {
-    const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, {
+    // FormData (multipart) vs JSON. node-fetch sets its own multipart boundary
-      method: 'post',
+    // header, so we must NOT supply Content-Type ourselves in that case.
-      body: JSON.stringify(body),
+    const isFormData = body instanceof FormData;
-      headers: { 'Content-Type': 'application/json' },
+    const opts = isFormData
-    });
+      ? { method: 'post', body }
+      : { method: 'post', body: JSON.stringify(body), headers: { 'Content-Type': 'application/json' } };
+    const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, opts);
 
     if (!res.ok) {
       const errorBody = await res.text();
@@ -208,16 +211,28 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
       });
     }
 
-    return await throttledCall('sendPhoto', {
+    const caption = plainText
-      chat_id: chatId,
+      ? buildCaptionPlain(jobName, serviceName, o, baseUrl)
-      photo: img,
+      : buildCaption(jobName, serviceName, o, baseUrl);
-      caption: plainText
+    const parseMode = plainText ? undefined : 'HTML';
-        ? buildCaptionPlain(jobName, serviceName, o, baseUrl)
+
-        : buildCaption(jobName, serviceName, o, baseUrl),
+    // .webp URLs (Immowelt/Cloudimage) fail Telegram's URL-based sendPhoto with
-      ...(plainText ? {} : { parse_mode: 'HTML' }),
+    // "failed to get HTTP URL content". Upload the bytes via multipart instead;
-      ...(message_thread_id ? { message_thread_id } : {}),
+    // the rendered chat message is identical.
-    }).catch(async (e) => {
+    const photoCall = shouldUseMultipart(img)
-      logger.error(`Error sending photo to Telegram and use a fallback: ${e.message}`);
+      ? buildPhotoFormData({ chatId, imageUrl: img, caption, parseMode, messageThreadId: message_thread_id }).then(
+          (fd) => throttledCall('sendPhoto', fd),
+        )
+      : throttledCall('sendPhoto', {
+          chat_id: chatId,
+          photo: img,
+          caption,
+          ...(parseMode ? { parse_mode: parseMode } : {}),
+          ...(message_thread_id ? { message_thread_id } : {}),
+        });
+
+    return await photoCall.catch(async (e) => {
+      logger.warn(`Error sending photo to Telegram and use a fallback: ${e.message}`);
       return await throttledCall('sendMessage', textPayload).catch((e) => {
         logger.error(`Error sending message to Telegram: ${e.message}`);
         throw e;

lib/notification/adapter/telegramPhotoUploader.js

@@ -0,0 +1,106 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+/**
+ * Helpers for sending photos to Telegram via `multipart/form-data` instead of
+ * the HTTP-URL path. Used when the URL is one that Telegram's URL-fetcher will
+ * reject - notably `.webp` images from Cloudimage (mms.immowelt.de), which
+ * Telegram refuses with "Bad Request: failed to get HTTP URL content".
+ *
+ * The HTTP-URL path is faster and is still the default in telegram.js; this
+ * module is the fallback for URLs whose extension makes Telegram fail.
+ */
+
+/** Telegram's sendPhoto limit when uploading bytes via multipart/form-data. */
+const TELEGRAM_MULTIPART_MAX_BYTES = 10 * 1024 * 1024;
+
+/** Accept header used when re-fetching the image ourselves.
+ *  Deliberately excludes `image/webp` so CDNs that content-negotiate
+ *  (like Cloudimage on mms.immowelt.de) transcode WEBP to JPEG. */
+const NON_WEBP_ACCEPT = 'image/jpeg,image/png,image/*;q=0.8';
+
+/**
+ * Returns true if the URL's path ends in a `.webp` extension. Such URLs need
+ * multipart upload because Telegram identifies media types from the URL path
+ * and rejects `.webp` in sendPhoto via HTTP URL.
+ *
+ * Conservative: returns false for null/empty/non-string input, malformed URLs,
+ * and non-https schemes.
+ *
+ * @param {string|null|undefined} url
+ * @returns {boolean}
+ */
+export function shouldUseMultipart(url) {
+  if (typeof url !== 'string' || url.length === 0) return false;
+  let parsed;
+  try {
+    parsed = new URL(url);
+  } catch {
+    return false;
+  }
+  if (parsed.protocol !== 'https:') return false;
+  return /\.webp$/i.test(parsed.pathname);
+}
+
+/**
+ * Fetch an image from `imageUrl` and build a `FormData` body suitable for
+ * POSTing to `https://api.telegram.org/bot<token>/sendPhoto`.
+ *
+ * - Sends an `Accept` header that excludes `image/webp` so origin/CDN servers
+ *   that content-negotiate return JPEG bytes.
+ * - Rejects images larger than Telegram's 10 MB multipart limit, both
+ *   advertised via `Content-Length` and (defensively) after download.
+ * - The `photo` field is named with a `.jpg` extension because Telegram
+ *   identifies file type from the filename.
+ *
+ * Throws if the image fetch fails, the size limit is exceeded, or the URL is
+ * unreachable. The caller is responsible for catching and falling back.
+ *
+ * @param {Object} args
+ * @param {string|number} args.chatId
+ * @param {string} args.imageUrl
+ * @param {string} args.caption
+ * @param {string} [args.parseMode]      - Telegram parse_mode, e.g. 'HTML'.
+ * @param {number} [args.messageThreadId] - Telegram supergroup topic id.
+ * @returns {Promise<FormData>}
+ */
+export async function buildPhotoFormData({ chatId, imageUrl, caption, parseMode, messageThreadId }) {
+  const res = await fetch(imageUrl, {
+    method: 'GET',
+    headers: { Accept: NON_WEBP_ACCEPT },
+  });
+
+  if (!res.ok) {
+    throw new Error(`Failed to fetch image for multipart upload (${res.status}): ${imageUrl}`);
+  }
+
+  const advertised = Number(res.headers.get('content-length'));
+  if (Number.isFinite(advertised) && advertised > TELEGRAM_MULTIPART_MAX_BYTES) {
+    throw new Error(
+      `Image exceeds Telegram multipart size limit (advertised ${advertised} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
+    );
+  }
+
+  const buf = await res.arrayBuffer();
+  if (buf.byteLength > TELEGRAM_MULTIPART_MAX_BYTES) {
+    throw new Error(
+      `Image exceeds Telegram multipart size limit (downloaded ${buf.byteLength} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
+    );
+  }
+
+  // Telegram identifies the media type from the filename extension. We always
+  // upload as .jpg because the Accept header forces JPEG bytes from CDNs that
+  // honor it; for the rare CDN that ignores Accept and still returns WEBP, the
+  // .jpg filename is a small lie but Telegram's image pipeline accepts it.
+  const blob = new Blob([buf], { type: 'image/jpeg' });
+
+  const fd = new FormData();
+  fd.append('chat_id', String(chatId));
+  fd.append('caption', caption);
+  if (parseMode) fd.append('parse_mode', parseMode);
+  if (messageThreadId != null) fd.append('message_thread_id', String(messageThreadId));
+  fd.append('photo', blob, 'photo.jpg');
+  return fd;
+}

lib/provider/immobilienDe.js

@@ -26,7 +26,7 @@ function parseId(shortenedLink) {
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immobilienDe_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);

lib/provider/immowelt.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immowelt_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);
@@ -87,7 +87,19 @@ const config = {
   crawlContainer:
     'div[data-testid="serp-core-scrollablelistview-testid"]:not(div[data-testid="serp-enlargementlist-testid"] div[data-testid="serp-card-testid"]) div[data-testid="serp-core-classified-card-testid"]',
   sortByDateParam: 'order=DateDesc',
-  waitForSelector: 'div[data-testid="serp-gridcontainer-testid"]',
+  // waitForSelector is null: extract the full page via page.content() so the
+  // Cheerio crawler can search anywhere in the rendered document.
+  // preNavigateUrl visits the homepage first to establish a trusted session
+  // before hitting the search URL; this prevents CDN-level bot challenges that
+  // fire on cold sessions. waitForNetworkIdle (phase 2) then catches React's
+  // listing API round-trip that fires well after domcontentloaded.
+  waitForSelector: null,
+  puppeteerOptions: {
+    puppeteerTimeout: 60_000,
+    preNavigateUrl: 'https://www.immowelt.de/',
+    waitForNetworkIdle: true,
+    waitForNetworkIdleTimeout: 60_000,
+  },
   crawlFields: {
     id: 'a@href',
     price: 'div[data-testid="cardmfe-price-testid"] | removeNewline | trim',

lib/provider/kleinanzeigen.js

@@ -128,7 +128,7 @@ async function enrichListingFromDetails(listing, browser) {
   if (!absoluteLink) return listing;
 
   try {
-    const html = await puppeteerExtractor(absoluteLink, null, { browser });
+    const html = await puppeteerExtractor(absoluteLink, null, { browser, name: 'kleinanzeigen_details' });
     if (!html) return { ...listing, link: absoluteLink };
 
     const { detailAddress, detailDescription } = extractDetailFromHtml(html);
@@ -196,8 +196,8 @@ const config = {
     id: '.aditem@data-adid',
     price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
     tags: '.aditem-main--middle--tags | removeNewline | trim',
-    title: '.aditem-main .text-module-begin a | removeNewline | trim',
+    title: '.aditem-main .text-module-begin | removeNewline | trim',
-    link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
+    link: '.aditem@data-href',
     description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
     address: '.aditem-main--top--left | trim | removeNewline',
     image: 'img@src',

lib/provider/sparkasse.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, 'body', { browser });
+    const html = await puppeteerExtractor(listing.link, 'body', { browser, name: 'sparkasse_details' });
 
     const $ = cheerio.load(html);
     const nextDataRaw = $('#__NEXT_DATA__').text;

lib/provider/wgGesucht.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'wgGesucht_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);

lib/services/ensureValidBinary.js

@@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { ensureBinary } from 'cloakbrowser';
+import fs from 'fs';
+import path from 'path';
+import os from 'os';
+
+/**
+ * Resource files required on Linux/Windows — they must live next to the chrome binary.
+ * macOS packages these inside the .app bundle's Frameworks directory so a different
+ * check is used there (see isBinaryComplete).
+ */
+const LINUX_WIN_REQUIRED_FILES = ['icudtl.dat', 'resources.pak'];
+
+/**
+ * Return the top-level versioned installation directory for any platform.
+ *
+ * - Linux/Windows: binaryPath is   ~/.cloakbrowser/chromium-X.Y.Z/chrome
+ *                  → dirname        ~/.cloakbrowser/chromium-X.Y.Z/
+ * - macOS:         binaryPath is   ~/.cloakbrowser/chromium-X.Y.Z/Chromium.app/Contents/MacOS/Chromium
+ *                  → 4 levels up    ~/.cloakbrowser/chromium-X.Y.Z/
+ *
+ * @param {string} binaryPath
+ * @returns {string}
+ */
+function getVersionedDir(binaryPath) {
+  if (process.platform === 'darwin') {
+    return path.resolve(path.dirname(binaryPath), '../../..');
+  }
+  return path.dirname(binaryPath);
+}
+
+/**
+ * Return true when the binary at binaryPath belongs to a complete installation.
+ *
+ * On macOS the binary lives inside an .app bundle:
+ *   Chromium.app/Contents/MacOS/Chromium
+ * Resource files (icudtl.dat etc.) are deep inside
+ *   Chromium.app/Contents/Frameworks/…
+ * so checking for them next to the binary is wrong. Instead we verify the two
+ * structural markers that are only present after a full extraction: Info.plist
+ * and the Frameworks directory inside Contents/.
+ *
+ * On Linux/Windows the binary and all resource files are siblings in the same
+ * directory.
+ *
+ * @param {string} binaryPath
+ * @returns {boolean}
+ */
+function isBinaryComplete(binaryPath) {
+  if (process.platform === 'darwin') {
+    const contentsDir = path.resolve(path.dirname(binaryPath), '..');
+    return fs.existsSync(path.join(contentsDir, 'Info.plist')) && fs.existsSync(path.join(contentsDir, 'Frameworks'));
+  }
+  const dir = path.dirname(binaryPath);
+  return LINUX_WIN_REQUIRED_FILES.every((f) => fs.existsSync(path.join(dir, f)));
+}
+
+/**
+ * Return a human-readable description of which required files/dirs are missing.
+ *
+ * @param {string} binaryPath
+ * @returns {string}
+ */
+function missingDescription(binaryPath) {
+  if (process.platform === 'darwin') {
+    const contentsDir = path.resolve(path.dirname(binaryPath), '..');
+    return ['Info.plist', 'Frameworks'].filter((f) => !fs.existsSync(path.join(contentsDir, f))).join(', ');
+  }
+  const dir = path.dirname(binaryPath);
+  return LINUX_WIN_REQUIRED_FILES.filter((f) => !fs.existsSync(path.join(dir, f))).join(', ');
+}
+
+/**
+ * Remove a corrupt binary installation and all `latest_version*` markers from
+ * the CloakBrowser cache so the next `ensureBinary()` call falls back to the
+ * package-bundled version.
+ *
+ * Removes the full versioned directory (e.g. chromium-X.Y.Z/) on all platforms,
+ * not just the subdirectory that contains the binary.
+ *
+ * @param {string} binaryPath - Path to the (corrupt) chrome/Chromium binary.
+ */
+function removeCorruptInstallation(binaryPath) {
+  const versionedDir = getVersionedDir(binaryPath);
+  const cacheDir = process.env.CLOAKBROWSER_CACHE_DIR || path.join(os.homedir(), '.cloakbrowser');
+
+  fs.rmSync(versionedDir, { recursive: true, force: true });
+
+  try {
+    for (const entry of fs.readdirSync(cacheDir)) {
+      if (entry.startsWith('latest_version')) {
+        fs.rmSync(path.join(cacheDir, entry), { force: true });
+      }
+    }
+  } catch {
+    // Cache dir may not exist if versionedDir was the only entry — ignore.
+  }
+}
+
+/**
+ * Ensure the CloakBrowser stealth Chromium binary is present **and** complete.
+ *
+ * `cloakbrowser`'s own `ensureBinary()` only checks that the chrome/Chromium
+ * file exists. An incomplete extraction (e.g. interrupted download, disk full)
+ * can leave a directory that contains the executable but is missing essential
+ * resource files. Chrome then crashes immediately on launch.
+ *
+ * This wrapper validates the path returned by `ensureBinary()`. If the
+ * installation is incomplete it removes the corrupt directory, clears the
+ * version marker files, and calls `ensureBinary()` again so it falls back to
+ * (or re-downloads) a complete build.
+ *
+ * The validated path is also pinned via `CLOAKBROWSER_BINARY_PATH` so that
+ * CloakBrowser's own internal `ensureBinary()` call inside `launch()` always
+ * picks up the same, verified binary.
+ *
+ * @returns {Promise<string>} Absolute path to the validated binary.
+ * @throws {Error} When even the fallback binary is incomplete.
+ */
+export async function ensureValidBinary() {
+  const binaryPath = await ensureBinary();
+
+  if (isBinaryComplete(binaryPath)) {
+    process.env.CLOAKBROWSER_BINARY_PATH = binaryPath;
+    return binaryPath;
+  }
+
+  console.warn(
+    `[fredy] CloakBrowser installation at ${getVersionedDir(binaryPath)} is missing: ${missingDescription(binaryPath)}. Removing and retrying.`,
+  );
+
+  removeCorruptInstallation(binaryPath);
+
+  const fallbackPath = await ensureBinary();
+  if (!isBinaryComplete(fallbackPath)) {
+    throw new Error(
+      `CloakBrowser binary at ${getVersionedDir(fallbackPath)} is still missing required files after re-download: ${missingDescription(fallbackPath)}`,
+    );
+  }
+
+  process.env.CLOAKBROWSER_BINARY_PATH = fallbackPath;
+  return fallbackPath;
+}

lib/services/extractor/extractor.js

@@ -29,11 +29,12 @@ export default class Extractor {
    * your response will never contain what you are really looking for
    * @param url
    * @param waitForSelector
+   * @param jobKey
    */
-  execute = async (url, waitForSelector = null) => {
+  execute = async (url, waitForSelector = null, jobKey = null) => {
     this.responseText = null;
     try {
-      this.responseText = await puppeteerExtractor(url, waitForSelector, this.options);
+      this.responseText = await puppeteerExtractor(url, waitForSelector, { ...this.options, name: jobKey });
       if (this.responseText != null) {
         loadParser(this.responseText);
       }

lib/services/extractor/puppeteerExtractor.js

@@ -3,121 +3,133 @@
  * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
  */
 
-import puppeteer from 'puppeteer-extra';
+import { launch } from 'cloakbrowser/puppeteer';
-import StealthPlugin from 'puppeteer-extra-plugin-stealth';
+import { botDetected, debug } from './utils.js';
-import { debug, botDetected } from './utils.js';
+import { getPreLaunchConfig } from './botPrevention.js';
-import {
-  getPreLaunchConfig,
-  applyBotPreventionToPage,
-  applyLanguagePersistence,
-  applyPostNavigationHumanSignals,
-} from './botPrevention.js';
 import logger from '../logger.js';
-import fs from 'fs';
+import { trackPoi } from '../tracking/Tracker.js';
-import os from 'os';
+import { TRACKING_POIS } from '../../TRACKING_POIS.js';
-import path from 'path';
-
-puppeteer.use(StealthPlugin());
 
+/**
+ * Launch a CloakBrowser/Puppeteer browser instance with stealth and humanizer enabled.
+ *
+ * CloakBrowser applies 49 C++ source-level patches (canvas, WebGL, audio, WebRTC,
+ * navigator.*, automation signals) that are indistinguishable from a real browser.
+ * All fingerprinting and human-behaviour simulation is handled natively; no CDP
+ * overrides (setUserAgent, setExtraHTTPHeaders, evaluateOnNewDocument) are applied
+ * here because they would create detectable inconsistencies on top of the C++ patches.
+ *
+ * @param {string} url - Initial URL (used to derive locale/timezone hints).
+ * @param {object} [options]
+ * @param {boolean} [options.puppeteerHeadless]
+ * @param {number}  [options.puppeteerTimeout]
+ * @param {string}  [options.proxyUrl]
+ * @param {string}  [options.timezone]
+ * @param {string}  [options.acceptLanguage]
+ * @param {object}  [options.viewport]
+ * @returns {Promise<import('puppeteer-core').Browser>}
+ */
 export async function launchBrowser(url, options) {
   const preCfg = getPreLaunchConfig(url, options || {});
-  const launchArgs = [
+
+  // Docker requires --no-sandbox; CloakBrowser handles all stealth args internally.
+  // --ignore-certificate-errors is needed because CloakBrowser ships its own Chromium
+  // binary with an independent CA bundle that may not trust proxies or interceptors
+  // present in the host environment.
+  const args = [
     '--no-sandbox',
-    '--disable-gpu',
     '--disable-setuid-sandbox',
     '--disable-dev-shm-usage',
-    '--disable-crash-reporter',
     '--no-first-run',
     '--no-default-browser-check',
-    preCfg.langArg,
+    '--ignore-certificate-errors',
+    // Disables the zygote process model. Required in some container environments
+    // (e.g. limited kernel namespaces) where the zygote cannot acquire the
+    // locks it needs and exits with "Invalid file descriptor to ICU data received".
+    '--no-zygote',
     preCfg.windowSizeArg,
-    ...preCfg.extraArgs,
   ];
-  if (options?.proxyUrl) {
-    launchArgs.push(`--proxy-server=${options.proxyUrl}`);
-  }
 
-  let userDataDir;
+  return await launch({
-  let removeUserDataDir = false;
-  if (options && options.userDataDir) {
-    userDataDir = options.userDataDir;
-  } else {
-    const prefix = path.join(os.tmpdir(), 'puppeteer-fredy-');
-    userDataDir = fs.mkdtempSync(prefix);
-    removeUserDataDir = true;
-  }
-
-  // On ARM64 Docker, Chrome for Testing has no native binary - use system Chromium instead.
-  const executablePath =
-    options?.executablePath ||
-    (process.arch === 'arm64' && process.env.IS_DOCKER === 'true' ? '/usr/bin/chromium' : undefined);
-
-  const browser = await puppeteer.launch({
     headless: options?.puppeteerHeadless ?? true,
-    args: launchArgs,
+    humanize: true,
-    timeout: options?.puppeteerTimeout || 45_000,
+    args,
-    userDataDir,
+    // locale sets Accept-Language headers and JS navigator.language consistently
-    executablePath,
+    locale: preCfg.langForFlag,
+    ...(options?.proxyUrl ? { proxy: options.proxyUrl } : {}),
+    ...(preCfg.timezone ? { timezone: preCfg.timezone } : {}),
   });
-
-  browser.__fredy_userDataDir = userDataDir;
-  browser.__fredy_removeUserDataDir = removeUserDataDir;
-
-  return browser;
 }
 
+/**
+ * Close a browser instance returned by {@link launchBrowser}.
+ *
+ * @param {import('puppeteer-core').Browser | null} browser
+ */
 export async function closeBrowser(browser) {
   if (!browser) return;
-  const userDataDir = browser.__fredy_userDataDir;
-  const removeUserDataDir = browser.__fredy_removeUserDataDir;
   try {
     await browser.close();
   } catch {
     // ignore
   }
-  if (removeUserDataDir && userDataDir) {
-    try {
-      await fs.promises.rm(userDataDir, { recursive: true, force: true });
-    } catch {
-      // ignore
-    }
-  }
 }
 
+/**
+ * Open a page in a (possibly reused) browser, navigate to `url`, and return the HTML source.
+ * Returns `null` when a bot-detection page is encountered or on timeout.
+ *
+ * @param {string} url
+ * @param {string | null} waitForSelector
+ * @param {object} [options]
+ * @returns {Promise<string | null>}
+ */
 export default async function execute(url, waitForSelector, options) {
   let browser = options?.browser;
   let isExternalBrowser = !!browser;
   let page;
   let result;
   try {
-    debug(`Sending request to ${url} using Puppeteer.`);
+    debug(`Sending request to ${url} using CloakBrowser.`);
 
     if (!isExternalBrowser) {
       browser = await launchBrowser(url, options);
     }
 
     page = await browser.newPage();
-    const preCfg = getPreLaunchConfig(url, options || {});
-    await applyBotPreventionToPage(page, preCfg);
-    // Provide languages value before navigation
-    await applyLanguagePersistence(page, preCfg);
 
-    // Optional cookies
     if (Array.isArray(options?.cookies) && options.cookies.length > 0) {
       await page.setCookie(...options.cookies);
     }
 
-    // Navigation
+    // Warm-up navigation: visit a trusted page first so the site sees an
+    // established session before the actual target URL. Silently ignored on
+    // failure so it never blocks the main request.
+    if (options?.preNavigateUrl) {
+      try {
+        await page.goto(options.preNavigateUrl, { waitUntil: 'domcontentloaded', timeout: 30_000 });
+        await new Promise((r) => setTimeout(r, 1500 + Math.random() * 2000));
+      } catch {
+        // ignore
+      }
+    }
+
     const response = await page.goto(url, {
       waitUntil: options?.waitUntil || 'domcontentloaded',
       timeout: options?.puppeteerTimeout || 60000,
     });
 
-    // Optionally wait and add subtle human-like interactions
+    // Optional second idle wait: useful for React SPAs that trigger API calls
-    await applyPostNavigationHumanSignals(page, preCfg);
+    // after domcontentloaded. Times out silently so we use whatever is rendered.
+    if (options?.waitForNetworkIdle) {
+      try {
+        await page.waitForNetworkIdle({ timeout: options?.waitForNetworkIdleTimeout ?? 60_000 });
+      } catch {
+        // ignore — we proceed with whatever the DOM contains at this point
+      }
+    }
 
     let pageSource;
-    // if we're extracting data from a SPA, we must wait for the selector
     if (waitForSelector != null) {
       const selectorTimeout = options?.puppeteerSelectorTimeout ?? options?.puppeteerTimeout ?? 30_000;
       await page.waitForSelector(waitForSelector, { timeout: selectorTimeout });
@@ -133,15 +145,22 @@ export default async function execute(url, waitForSelector, options) {
 
     if (botDetected(pageSource, statusCode)) {
       logger.warn('We have been detected as a bot :-/ Tried url: => ', url);
+
+      if (options != null && options.name != null) {
+        await trackPoi(TRACKING_POIS.DETECTED_AS_BOT + '_' + options.name);
+      } else {
+        await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
+      }
+
       result = null;
     } else {
       result = pageSource || (await page.content());
     }
   } catch (error) {
     if (error?.name?.includes('Timeout')) {
-      logger.debug('Error executing with puppeteer executor', error);
+      logger.debug('Error executing with CloakBrowser executor', error);
     } else {
-      logger.warn('Error executing with puppeteer executor', error);
+      logger.warn('Error executing with CloakBrowser executor', error);
     }
     result = null;
   } finally {

lib/services/immoscout/immoscout-web-translator.js

@@ -141,6 +141,43 @@ const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
   'barrierefreie-wohnung-mieten': { equipment: ['handicappedaccessible'] },
 };
 
+// SEO-optimized rental paths used by the ImmoScout web UI when the user
+// configures a maximum warmrent. Example: "wohnung-bis-800-euro-warm" means
+// "apartment for rent up to 800 EUR warmrent". The web UI generates these
+// paths instead of explicit `price` / `pricetype` query parameters.
+// Note: only the warmrent variant uses an SEO slug; max coldrent searches
+// use the regular "wohnung-mieten" path with explicit `price` and
+// `pricetype=rentpermonth` query params, which the existing translator
+// already handles.
+const SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE = {
+  wohnung: 'apartmentrent',
+  haus: 'houserent',
+};
+const SEO_MAX_WARMRENT_PATH_PATTERN = /^(?<type>wohnung|haus)-bis-(?<price>\d+)-euro-warm$/;
+
+/**
+ * Parses SEO-optimized ImmoScout web paths that encode a maximum warmrent, such
+ * as "wohnung-bis-800-euro-warm". Returns the corresponding mobile API real
+ * estate type and the implicit price/pricetype parameters, or null if the path
+ * does not match the known SEO max-warmrent pattern.
+ *
+ * @param {string} realTypeKey The last segment of the URL path.
+ * @returns {{ realType: string, additionalParams: Record<string, string> } | null}
+ */
+function parseSeoMaxWarmrentPath(realTypeKey) {
+  const match = realTypeKey.match(SEO_MAX_WARMRENT_PATH_PATTERN);
+  if (!match) return null;
+
+  const { type, price } = match.groups;
+  return {
+    realType: SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE[type],
+    additionalParams: {
+      price: `-${price}`,
+      pricetype: 'calculatedtotalrent',
+    },
+  };
+}
+
 export function convertWebToMobile(webUrl) {
   let url;
   try {
@@ -164,14 +201,17 @@ export function convertWebToMobile(webUrl) {
       additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey];
       realType = REAL_ESTATE_TYPE['wohnung-mieten'];
     } else {
-      throw new Error(`Real estate type not found: ${realTypeKey}`);
+      // Test for SEO max-warmrent path, e.g. "wohnung-bis-800-euro-warm"
+      const seoMaxWarmrent = parseSeoMaxWarmrentPath(realTypeKey);
+      if (seoMaxWarmrent) {
+        realType = seoMaxWarmrent.realType;
+        additionalParamsFromWebPath = seoMaxWarmrent.additionalParams;
+      } else {
+        throw new Error(`Real estate type not found: ${realTypeKey}`);
+      }
     }
   }
 
-  if (segments.includes('shape')) {
-    throw new Error('Shape is currently not supported using Immoscout');
-  }
-
   const { query: rawParams } = queryString.parseUrl(webUrl, { arrayFormat: 'comma' });
   const webParams = Object.fromEntries(
     Object.entries(rawParams).filter(([key]) => key !== 'enteredFrom' && PARAM_NAME_MAP[key]),
@@ -179,18 +219,31 @@ export function convertWebToMobile(webUrl) {
 
   const geocodes = `/${segments.slice(2, segments.length - 1).join('/')}`;
   const isRadius = segments.includes('radius');
+  const isShape = segments.includes('shape');
   const mobileParams = {
-    searchType: isRadius ? 'radius' : 'region',
+    searchType: isRadius ? 'radius' : isShape ? 'shape' : 'region',
     realestatetype: realType,
-    ...(isRadius ? {} : { geocodes }),
+    ...(isRadius || isShape ? {} : { geocodes }),
     ...additionalParamsFromWebPath,
   };
 
+  if (isShape && !webParams.shape) {
+    throw new Error('Shape search URL is missing the required "shape" query parameter');
+  }
+
+  if (isShape && webParams.shape) {
+    const browserShape = webParams.shape;
+    const normalized = browserShape.replace(/\.\./g, '==').replace(/\./g, '=');
+    const polyline = Buffer.from(normalized, 'base64').toString('utf-8');
+    mobileParams.shape = polyline;
+  }
+
   if (webParams.geocoordinates) {
     mobileParams.geocoordinates = webParams.geocoordinates;
   }
 
   for (const [key, val] of Object.entries(webParams)) {
+    if (key === 'shape') continue;
     if (key === 'equipment') {
       const items = [].concat(val).flatMap((v) => `${v}`.split(','));
       const currentEquipmentParams = mobileParams[PARAM_NAME_MAP[key]];

lib/services/jobs/jobExecutionService.js

@@ -14,6 +14,7 @@ import * as similarityCache from '../similarity-check/similarityCache.js';
 import { isRunning, markFinished, markRunning } from './run-state.js';
 import { sendToUsers } from '../sse/sse-broker.js';
 import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
+import { getSettings } from '../storage/settingsStorage.js';
 
 /**
  * Initializes the job execution service.
@@ -160,6 +161,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
     }
     let browser;
     try {
+      // Read the proxy live (not from the startup snapshot) so changing it in the
+      // UI takes effect on the next run without a backend restart. An empty value
+      // disables the proxy. Routing the headless browser through a (German
+      // residential) proxy avoids datacenter-IP based bot detection on the
+      // Puppeteer-based providers (immowelt, immonet, kleinanzeigen, ...).
+      const liveSettings = await getSettings();
+      const proxyUrl = typeof liveSettings?.proxyUrl === 'string' ? liveSettings.proxyUrl.trim() : '';
+
       const jobProviders = job.provider.filter(
         (p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
       );
@@ -168,14 +177,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
           const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
           matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
 
-          if (browser && !browser.isConnected()) {
+          if (browser && !browser.connected) {
             logger.debug('Browser is disconnected, nullifying to launch a new one.');
             await puppeteerExtractor.closeBrowser(browser);
             browser = null;
           }
 
           if (!browser && matchedProvider.config.getListings == null) {
-            browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
+            browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, proxyUrl ? { proxyUrl } : {});
           }
 
           await new FredyPipelineExecutioner(matchedProvider.config, job, prov.id, similarityCache, browser).execute();

lib/services/storage/listingsStorage.js

@@ -214,6 +214,8 @@ export const storeListings = (jobId, providerId, listings) => {
         longitude: item.longitude || null,
       };
       stmt.run(params);
+      // Propagate the DB primary key back so downstream pipeline steps use the correct id
+      item.id = params.id;
     }
   });
 
@@ -417,9 +419,10 @@ export const deleteListingsByJobId = (jobId, hardDelete = false) => {
 };
 
 /**
- * Delete listings by a list of listing IDs.
+ * Delete listings by a list of listing IDs (the nanoid primary key stored in the `id` column).
+ * Used by API routes that receive row IDs from the client.
  *
- * @param {string[]} ids - Array of listing IDs to delete.
+ * @param {string[]} ids - Array of DB row IDs to delete.
  * @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
  * @returns {any} The result from SqliteConnection.execute.
  */

package.json

@@ -1,6 +1,6 @@
 {
   "name": "fredy",
-  "version": "21.2.0",
+  "version": "22.2.2",
   "description": "[F]ind [R]eal [E]states [d]amn eas[y].",
   "scripts": {
     "prepare": "husky",
@@ -62,9 +62,9 @@
     "Firefox ESR"
   ],
   "dependencies": {
-    "@douyinfe/semi-icons": "^2.96.1",
+    "@douyinfe/semi-icons": "^2.99.3",
-    "@douyinfe/semi-ui": "2.96.1",
+    "@douyinfe/semi-ui": "2.99.3",
-    "@douyinfe/semi-ui-19": "^2.96.1",
+    "@douyinfe/semi-ui-19": "^2.99.3",
     "@fastify/cookie": "^11.0.2",
     "@fastify/helmet": "^13.0.2",
     "@fastify/session": "^11.1.1",
@@ -73,11 +73,12 @@
     "@modelcontextprotocol/sdk": "^1.29.0",
     "@sendgrid/mail": "8.1.6",
     "@turf/boolean-point-in-polygon": "^7.3.5",
-    "@vitejs/plugin-react": "6.0.1",
+    "@vitejs/plugin-react": "6.0.2",
     "adm-zip": "^0.5.17",
-    "better-sqlite3": "^12.9.0",
+    "better-sqlite3": "^12.10.0",
     "chart.js": "^4.5.1",
     "cheerio": "^1.2.0",
+    "cloakbrowser": "^0.3.31",
     "fastify": "^5.8.5",
     "handlebars": "4.7.9",
     "maplibre-gl": "^5.24.0",
@@ -85,43 +86,41 @@
     "node-cron": "^4.2.1",
     "node-fetch": "3.3.2",
     "node-mailjet": "6.0.11",
-    "nodemailer": "^8.0.7",
+    "nodemailer": "^8.0.10",
     "p-throttle": "^8.1.0",
     "package-up": "^5.0.0",
-    "puppeteer": "^24.43.0",
+    "puppeteer-core": "^25.1.0",
-    "puppeteer-extra": "^3.3.6",
+    "query-string": "9.4.0",
-    "puppeteer-extra-plugin-stealth": "^2.11.2",
+    "react": "19.2.7",
-    "query-string": "9.3.1",
-    "react": "19.2.6",
     "react-chartjs-2": "^5.3.1",
-    "react-dom": "19.2.6",
+    "react-dom": "19.2.7",
     "react-range-slider-input": "^3.3.5",
-    "react-router": "7.15.0",
+    "react-router": "7.16.0",
-    "react-router-dom": "7.15.0",
+    "react-router-dom": "7.16.0",
-    "resend": "^6.12.3",
+    "resend": "^6.12.4",
-    "semver": "^7.7.4",
+    "semver": "^7.8.1",
     "slack": "11.0.2",
-    "vite": "8.0.11",
+    "vite": "8.0.16",
     "x-var": "^3.0.1",
-    "zustand": "^5.0.13"
+    "zustand": "^5.0.14"
   },
   "devDependencies": {
-    "@babel/core": "7.29.0",
+    "@babel/core": "7.29.7",
-    "@babel/eslint-parser": "7.28.6",
+    "@babel/eslint-parser": "7.29.7",
-    "@babel/preset-env": "7.29.5",
+    "@babel/preset-env": "7.29.7",
-    "@babel/preset-react": "7.28.5",
+    "@babel/preset-react": "7.29.7",
     "@eslint/js": "^10.0.1",
     "chalk": "^5.6.2",
-    "eslint": "10.3.0",
+    "eslint": "10.4.1",
     "eslint-config-prettier": "10.1.8",
     "eslint-plugin-react": "7.37.5",
     "globals": "^17.6.0",
     "history": "5.3.0",
     "husky": "9.1.7",
     "less": "4.6.4",
-    "lint-staged": "16.4.0",
+    "lint-staged": "17.0.7",
     "nodemon": "^3.1.14",
     "prettier": "3.8.3",
-    "vitest": "^4.1.5"
+    "vitest": "^4.1.8"
   }
 }

test/globalSetup.js

@@ -0,0 +1,18 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { ensureValidBinary } from '../lib/services/ensureValidBinary.js';
+
+/**
+ * Vitest global setup — runs once in the main process before any workers start.
+ * Downloads and validates the CloakBrowser stealth Chromium binary.
+ * ensureValidBinary() also removes and re-downloads partial/corrupt installations
+ * so tests never fail with "Invalid file descriptor to ICU data received".
+ * Skipped in offline mode because the browser is fully mocked there.
+ */
+export async function setup() {
+  if (process.env.TEST_MODE === 'offline') return;
+  await ensureValidBinary();
+}

test/mocks/mockStore.js

@@ -32,4 +32,7 @@ export const deletedIds = [];
 export const deleteListingsById = (ids) => {
   deletedIds.push(...ids);
 };
+export const deleteListingsByHash = (hashes) => {
+  deletedIds.push(...hashes);
+};
 /* eslint-enable no-unused-vars */

test/notification/telegram.test.js

@@ -0,0 +1,360 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+
+// Mock external deps BEFORE importing the module under test.
+vi.mock('node-fetch', () => ({ default: vi.fn() }));
+vi.mock('../../lib/services/storage/jobStorage.js', () => ({
+  getJob: (jobKey) => ({ id: jobKey, name: jobKey }),
+}));
+vi.mock('../../lib/services/markdown.js', () => ({
+  markdown2Html: () => '',
+}));
+
+// Helpers to build mock fetch responses.
+function jsonOk(body = { ok: true }) {
+  return {
+    ok: true,
+    status: 200,
+    text: async () => JSON.stringify(body),
+  };
+}
+
+function jsonErr(status, body) {
+  return {
+    ok: false,
+    status,
+    text: async () => JSON.stringify(body),
+  };
+}
+
+function imageOk(bytes = new Uint8Array([0xff, 0xd8, 0xff])) {
+  return {
+    ok: true,
+    status: 200,
+    headers: {
+      get: (h) => {
+        const k = h.toLowerCase();
+        if (k === 'content-type') return 'image/jpeg';
+        if (k === 'content-length') return String(bytes.byteLength);
+        return null;
+      },
+    },
+    arrayBuffer: async () => bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength),
+  };
+}
+
+// Globals are mocked too so buildPhotoFormData (which uses global fetch) can be
+// intercepted by the same single mock.
+let mockNodeFetch;
+let mockGlobalFetch;
+let send;
+
+beforeEach(async () => {
+  // Reset modules to get a fresh import with our mocks applied.
+  vi.resetModules();
+  const nodeFetchMod = await import('node-fetch');
+  mockNodeFetch = nodeFetchMod.default;
+  mockNodeFetch.mockReset();
+
+  mockGlobalFetch = vi.fn();
+  vi.stubGlobal('fetch', mockGlobalFetch);
+
+  ({ send } = await import('../../lib/notification/adapter/telegram.js'));
+});
+
+afterEach(() => {
+  vi.unstubAllGlobals();
+  vi.useRealTimers();
+});
+
+const baseConfig = {
+  id: 'telegram',
+  fields: { token: 'TKN', chatId: '999' },
+};
+
+describe('telegram send() - HTTP URL path (default for .jpg / .png)', () => {
+  it('POSTs JSON to sendPhoto for a .jpg image URL', async () => {
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 'Listing',
+          link: 'https://example.com/a',
+          address: 'Addr',
+          price: '500€',
+          size: '50m²',
+          image: 'https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    const [url, opts] = mockNodeFetch.mock.calls[0];
+    expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
+    expect(opts.method).toBe('post');
+    expect(opts.headers?.['Content-Type']).toBe('application/json');
+    const body = JSON.parse(opts.body);
+    expect(body.chat_id).toBe('999');
+    expect(body.photo).toBe('https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394');
+    expect(body.parse_mode).toBe('HTML');
+  });
+
+  it('does NOT pre-fetch the image when using HTTP URL path', async () => {
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/x.jpg',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    // global fetch (used by buildPhotoFormData) must not be called
+    expect(mockGlobalFetch).not.toHaveBeenCalled();
+  });
+
+  it('falls back to sendMessage when sendPhoto fails', async () => {
+    mockNodeFetch
+      .mockResolvedValueOnce(jsonErr(400, { ok: false, description: 'boom' }))
+      .mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/x.jpg',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(2);
+    expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendPhoto');
+    expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+  });
+});
+
+describe('telegram send() - multipart path (.webp URLs)', () => {
+  it('pre-fetches the image then POSTs FormData to sendPhoto for a .webp URL', async () => {
+    // 1st: GET image via global fetch
+    mockGlobalFetch.mockResolvedValueOnce(imageOk());
+    // 2nd: POST sendPhoto via node-fetch
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 'Listing',
+          link: 'https://example.com/a',
+          address: 'Addr',
+          price: '500€',
+          size: '50m²',
+          image: 'https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    // image was fetched
+    expect(mockGlobalFetch).toHaveBeenCalledTimes(1);
+    expect(mockGlobalFetch.mock.calls[0][0]).toBe('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394');
+
+    // sendPhoto called via node-fetch with FormData
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    const [url, opts] = mockNodeFetch.mock.calls[0];
+    expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
+    expect(opts.method).toBe('post');
+    expect(opts.body).toBeInstanceOf(FormData);
+    // No explicit Content-Type header - fetch sets multipart boundary itself
+    expect(opts.headers).toBeUndefined();
+    expect(opts.body.get('chat_id')).toBe('999');
+    expect(opts.body.get('parse_mode')).toBe('HTML');
+    const photo = opts.body.get('photo');
+    expect(photo).toBeTruthy();
+    expect(photo.size).toBeGreaterThan(0);
+  });
+
+  it('falls back to sendMessage when the image pre-fetch fails for a .webp URL', async () => {
+    // image fetch fails (404 from CDN)
+    mockGlobalFetch.mockResolvedValueOnce({
+      ok: false,
+      status: 404,
+      headers: { get: () => null },
+      arrayBuffer: async () => new ArrayBuffer(0),
+    });
+    // then sendMessage succeeds via node-fetch
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/gone.webp',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+  });
+
+  it('falls back to sendMessage when multipart sendPhoto returns a Telegram error', async () => {
+    mockGlobalFetch.mockResolvedValueOnce(imageOk());
+    mockNodeFetch
+      .mockResolvedValueOnce(jsonErr(400, { description: 'broke' })) // multipart sendPhoto
+      .mockResolvedValueOnce(jsonOk()); // sendMessage fallback
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/x.webp',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(2);
+    expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+  });
+});
+
+describe('telegram send() - mixed batch (regression-safety)', () => {
+  it('handles a batch with both .jpg and .webp - jpg uses URL, webp uses multipart', async () => {
+    // .webp image fetch
+    mockGlobalFetch.mockResolvedValueOnce(imageOk());
+    // both sendPhoto calls succeed
+    mockNodeFetch
+      .mockResolvedValueOnce(jsonOk()) // could be either listing first
+      .mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'jpg-listing',
+          title: 'a',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/a.jpg',
+        },
+        {
+          id: 'webp-listing',
+          title: 'b',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/b.webp',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockGlobalFetch).toHaveBeenCalledTimes(1); // only webp pre-fetches
+    expect(mockNodeFetch).toHaveBeenCalledTimes(2);
+
+    // Verify one call had FormData and one had JSON body
+    const bodies = mockNodeFetch.mock.calls.map((c) => c[1].body);
+    const hasFormData = bodies.some((b) => b instanceof FormData);
+    const hasJson = bodies.some((b) => typeof b === 'string' && b.startsWith('{'));
+    expect(hasFormData).toBe(true);
+    expect(hasJson).toBe(true);
+  });
+
+  it('uses sendMessage (not sendPhoto) when image is null', async () => {
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: null,
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+    expect(mockGlobalFetch).not.toHaveBeenCalled();
+  });
+});
+
+describe('telegram send() - config validation', () => {
+  it('throws when telegram adapter config is missing', () => {
+    expect(() =>
+      send({
+        serviceName: 's',
+        newListings: [],
+        notificationConfig: [],
+        jobKey: 'k',
+      }),
+    ).toThrow(/configuration missing/);
+  });
+
+  it('throws when token or chatId is missing', () => {
+    expect(() =>
+      send({
+        serviceName: 's',
+        newListings: [],
+        notificationConfig: [{ id: 'telegram', fields: { token: '' } }],
+        jobKey: 'k',
+      }),
+    ).toThrow(/token.*chatId/);
+  });
+});

test/notification/telegramPhotoUploader.test.js

@@ -0,0 +1,287 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+import { shouldUseMultipart, buildPhotoFormData } from '../../lib/notification/adapter/telegramPhotoUploader.js';
+
+describe('shouldUseMultipart', () => {
+  it('returns true for .webp URL with query string', () => {
+    expect(shouldUseMultipart('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394')).toBe(true);
+  });
+
+  it('returns true for .webp URL without query string', () => {
+    expect(shouldUseMultipart('https://example.com/photo.webp')).toBe(true);
+  });
+
+  it('returns true for uppercase .WEBP extension', () => {
+    expect(shouldUseMultipart('https://example.com/IMG.WEBP?x=1')).toBe(true);
+  });
+
+  it('returns false for .jpg URL with query string', () => {
+    expect(shouldUseMultipart('https://mms.immowelt.de/a/b/c/d/xyz.jpg?ci_seal=hash&w=525&h=394')).toBe(false);
+  });
+
+  it('returns false for .jpeg URL', () => {
+    expect(shouldUseMultipart('https://example.com/photo.jpeg')).toBe(false);
+  });
+
+  it('returns false for .png URL with query string', () => {
+    expect(shouldUseMultipart('https://example.com/photo.png?w=100')).toBe(false);
+  });
+
+  it('returns false for .gif URL', () => {
+    expect(shouldUseMultipart('https://example.com/photo.gif')).toBe(false);
+  });
+
+  it('returns false for null', () => {
+    expect(shouldUseMultipart(null)).toBe(false);
+  });
+
+  it('returns false for undefined', () => {
+    expect(shouldUseMultipart(undefined)).toBe(false);
+  });
+
+  it('returns false for empty string', () => {
+    expect(shouldUseMultipart('')).toBe(false);
+  });
+
+  it('returns false for malformed URL', () => {
+    expect(shouldUseMultipart('not a url')).toBe(false);
+  });
+
+  it('returns false for URL where webp is in the query but not the path', () => {
+    expect(shouldUseMultipart('https://example.com/photo.jpg?format=webp')).toBe(false);
+  });
+
+  it('returns false for URL with no extension at all', () => {
+    expect(shouldUseMultipart('https://example.com/photo')).toBe(false);
+  });
+
+  it('returns false for non-https schemes', () => {
+    // file/data/ftp URLs should not be relevant; safer to skip multipart
+    expect(shouldUseMultipart('http://example.com/photo.webp')).toBe(false);
+  });
+});
+
+describe('buildPhotoFormData', () => {
+  let mockFetch;
+
+  beforeEach(() => {
+    mockFetch = vi.fn();
+    vi.stubGlobal('fetch', mockFetch);
+  });
+
+  afterEach(() => {
+    vi.unstubAllGlobals();
+  });
+
+  function makeImageResponse({ contentType = 'image/jpeg', bytes = new Uint8Array([0xff, 0xd8, 0xff]) } = {}) {
+    const buf = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
+    return {
+      ok: true,
+      status: 200,
+      headers: {
+        get: (h) =>
+          h.toLowerCase() === 'content-type'
+            ? contentType
+            : h.toLowerCase() === 'content-length'
+              ? String(bytes.byteLength)
+              : null,
+      },
+      arrayBuffer: async () => buf,
+    };
+  }
+
+  it('fetches image with Accept header that excludes webp so the CDN transcodes to JPEG', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    await buildPhotoFormData({
+      chatId: '123',
+      imageUrl: 'https://example.com/photo.webp',
+      caption: 'hi',
+      parseMode: 'HTML',
+    });
+
+    expect(mockFetch).toHaveBeenCalledTimes(1);
+    const [url, opts] = mockFetch.mock.calls[0];
+    expect(url).toBe('https://example.com/photo.webp');
+    expect(opts?.headers?.Accept || opts?.headers?.accept).toMatch(/image\/jpeg/);
+    expect(opts?.headers?.Accept || opts?.headers?.accept).not.toMatch(/image\/webp/);
+  });
+
+  it('returns FormData containing chat_id, caption, parse_mode, and photo fields', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '12345',
+      imageUrl: 'https://example.com/abc.webp',
+      caption: 'My caption',
+      parseMode: 'HTML',
+    });
+
+    expect(fd).toBeInstanceOf(FormData);
+    expect(fd.get('chat_id')).toBe('12345');
+    expect(fd.get('caption')).toBe('My caption');
+    expect(fd.get('parse_mode')).toBe('HTML');
+    const photo = fd.get('photo');
+    expect(photo).toBeTruthy();
+    // File-like (Blob); has a name and a size
+    expect(typeof photo.name).toBe('string');
+    expect(photo.size).toBeGreaterThan(0);
+  });
+
+  it('uses a .jpg filename (Telegram uses URL/filename extension for type detection)', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    const photo = fd.get('photo');
+    expect(photo.name).toMatch(/\.jpg$/i);
+  });
+
+  it('includes message_thread_id when provided', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+      messageThreadId: 42,
+    });
+
+    expect(fd.get('message_thread_id')).toBe('42');
+  });
+
+  it('omits message_thread_id when not provided', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    expect(fd.get('message_thread_id')).toBeNull();
+  });
+
+  it('omits parse_mode when not provided (plain text mode)', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+    });
+
+    expect(fd.get('parse_mode')).toBeNull();
+  });
+
+  it('throws when the image fetch returns non-200', async () => {
+    mockFetch.mockResolvedValueOnce({
+      ok: false,
+      status: 404,
+      headers: { get: () => null },
+      arrayBuffer: async () => new ArrayBuffer(0),
+    });
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/gone.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/404/);
+  });
+
+  it('throws when the image fetch throws (network error)', async () => {
+    mockFetch.mockRejectedValueOnce(new Error('ECONNREFUSED'));
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/x.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/ECONNREFUSED/);
+  });
+
+  it('throws when the image exceeds 10 MB (Telegram multipart limit)', async () => {
+    // 11 MB
+    const big = new Uint8Array(11 * 1024 * 1024);
+    mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes: big }));
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/huge.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/size|large|10/i);
+  });
+
+  it('rejects early when content-length header advertises > 10 MB (avoids download)', async () => {
+    mockFetch.mockResolvedValueOnce({
+      ok: true,
+      status: 200,
+      headers: {
+        get: (h) => {
+          const k = h.toLowerCase();
+          if (k === 'content-type') return 'image/jpeg';
+          if (k === 'content-length') return String(50 * 1024 * 1024);
+          return null;
+        },
+      },
+      arrayBuffer: async () => {
+        throw new Error('should not be called - size check should reject first');
+      },
+    });
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/huge.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/size|large|10/i);
+  });
+
+  it('accepts exactly 10 MB images (boundary)', async () => {
+    const bytes = new Uint8Array(10 * 1024 * 1024);
+    mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes }));
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/exact.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    expect(fd.get('photo').size).toBe(10 * 1024 * 1024);
+  });
+
+  it('coerces non-string chatId (number) to string in form data', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: 999,
+      imageUrl: 'https://example.com/x.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    expect(fd.get('chat_id')).toBe('999');
+  });
+});

test/offlineFixtures.js

@@ -38,6 +38,20 @@ async function tryReadFile(filepath) {
   }
 }
 
+function withRealEstateType(data, realEstateType) {
+  if (!realEstateType?.length || !Array.isArray(data?.resultListItems)) {
+    return data;
+  }
+
+  const cloned = typeof structuredClone === 'function' ? structuredClone(data) : JSON.parse(JSON.stringify(data));
+  for (const item of cloned.resultListItems) {
+    if (item?.type === 'EXPOSE_RESULT' && item?.item) {
+      item.item.realEstateType = realEstateType;
+    }
+  }
+  return cloned;
+}
+
 /**
  * Returns fixture HTML for the given URL by mapping hostname → provider name,
  * then distinguishing list vs detail pages by comparing the URL path against
@@ -83,7 +97,10 @@ export function buildFetchMock() {
         const raw = await tryReadFile(path.join(FIXTURES_DIR, 'immoscout_list.json'));
         listData = raw ? JSON.parse(raw) : { resultListItems: [] };
       }
-      return { ok: true, status: 200, json: () => Promise.resolve(listData) };
+
+      const requestedType = new URL(urlStr).searchParams.get('realestatetype');
+      const responseData = withRealEstateType(listData, requestedType);
+      return { ok: true, status: 200, json: () => Promise.resolve(responseData) };
     }
 
     if (urlStr.includes('api.mobile.immobilienscout24.de/expose/')) {

test/provider/immobilienDe.test.js

@@ -6,83 +6,89 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { providerConfig, mockFredy } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/immobilienDe.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session, avoiding double cold-start bot detection.
+const TEST_TIMEOUT = 120_000;
 
 describe('#immobilien.de testsuite()', () => {
   provider.init(providerConfig.immobilienDe, [], []);
-  it('should test immobilien.de provider', async () => {
-    const mockedJob = {
-      id: 'test1',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
 
-    const Fredy = await mockFredy();
+  let browser;
-    const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  let liveListings;
-    const listing = await fredy.execute();
 
-    if (listing == null || listing.length === 0) {
+  beforeAll(async () => {
-      throw new Error('Listings is empty!');
+    browser = await launchBrowser(providerConfig.immobilienDe.url);
-    }
+  }, TEST_TIMEOUT);
 
-    expect(listing).toBeInstanceOf(Array);
+  afterAll(async () => {
-    const notificationObj = get();
+    await closeBrowser(browser);
-    expect(notificationObj).toBeTypeOf('object');
-    expect(notificationObj.serviceName).toBe('immobilienDe');
-    notificationObj.payload.forEach((notify) => {
-      /** check the actual structure **/
-      expect(notify.id).toBeTypeOf('string');
-      expect(notify.price).toBeTypeOf('string');
-      expect(notify.size).toBeTypeOf('string');
-      expect(notify.title).toBeTypeOf('string');
-      expect(notify.link).toBeTypeOf('string');
-      expect(notify.address).toBeTypeOf('string');
-      /** check the values if possible **/
-      expect(notify.price).toContain('€');
-      expect(notify.size).toContain('m²');
-      expect(notify.title).not.toBe('');
-      expect(notify.link).toContain('https://www.immobilien.de');
-      expect(notify.address).not.toBe('');
-    });
   });
 
+  it(
+    'should test immobilien.de provider',
+    async () => {
+      const mockedJob = {
+        id: 'test1',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+
+      const Fredy = await mockFredy();
+      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+      liveListings = await fredy.execute();
+
+      if (liveListings == null || liveListings.length === 0) {
+        throw new Error('Listings is empty!');
+      }
+
+      expect(liveListings).toBeInstanceOf(Array);
+      const notificationObj = get();
+      expect(notificationObj).toBeTypeOf('object');
+      expect(notificationObj.serviceName).toBe('immobilienDe');
+      notificationObj.payload.forEach((notify) => {
+        /** check the actual structure **/
+        expect(notify.id).toBeTypeOf('string');
+        expect(notify.price).toBeTypeOf('string');
+        expect(notify.size).toBeTypeOf('string');
+        expect(notify.title).toBeTypeOf('string');
+        expect(notify.link).toBeTypeOf('string');
+        expect(notify.address).toBeTypeOf('string');
+        /** check the values if possible **/
+        expect(notify.price).toContain('€');
+        expect(notify.size).toContain('m²');
+        expect(notify.title).not.toBe('');
+        expect(notify.link).toContain('https://www.immobilien.de');
+        expect(notify.address).not.toBe('');
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
+    it(
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
+      'should enrich listings with details',
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
+      async () => {
-    });
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
+        // Call fetchDetails directly on the first live listing — no need to
-      vi.restoreAllMocks();
+        // re-scrape the search page. The shared browser keeps the session warm.
-    });
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
+        if (enriched == null) return;
-      const Fredy = await mockFredy();
+        expect(enriched.link).toContain('https://www.immobilien.de');
-      provider.init(providerConfig.immobilienDe, [], []);
+        expect(enriched.address).toBeTypeOf('string');
-      const mockedJob = { id: 'test1', notificationAdapter: null, specFilter: null, spatialFilter: null };
+        expect(enriched.address).not.toBe('');
-
+        // description may be null if selectors don't match yet — falls back gracefully
-      const fredy = new Fredy(
+        if (enriched.description != null) {
-        provider.config,
+          expect(enriched.description).toBeTypeOf('string');
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      if (listings == null) return;
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.immobilien.de');
-        expect(listing.address).toBeTypeOf('string');
-        expect(listing.address).not.toBe('');
-        // description may be null if selectors don't match yet - falls back gracefully
-        if (listing.description != null) {
-          expect(listing.description).toBeTypeOf('string');
         }
-      });
+      },
-    });
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/immoscout.test.js

@@ -3,85 +3,85 @@
  * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
  */
 
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { mockFredy, providerConfig } from '../utils.js';
 import { get } from '../mocks/mockNotification.js';
 import * as provider from '../../lib/provider/immoscout.js';
-import * as mockStore from '../mocks/mockStore.js';
+
+// immoscout uses the mobile REST API (fetch-based, no browser). Both tests share
+// the same module-level listings so the API is only queried once, avoiding
+// duplicate requests that could trigger rate-limiting.
+const TEST_TIMEOUT = 120_000;
 
 describe('#immoscout provider testsuite()', () => {
   provider.init(providerConfig.immoscout, [], []);
-  it('should test immoscout provider', async () => {
-    const Fredy = await mockFredy();
-    const mockedJob = {
-      id: '',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
 
-    return await new Promise((resolve, reject) => {
+  let liveListings;
-      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
-      fredy.execute().then((listings) => {
-        if (listings == null || listings.length === 0) {
-          reject('Listings is empty!');
-          return;
-        }
 
-        expect(listings).toBeInstanceOf(Array);
+  it(
-        const notificationObj = get();
+    'should test immoscout provider',
-        expect(notificationObj).toBeTypeOf('object');
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: '',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
 
-        // check if there is at least one valid notification
+      return await new Promise((resolve, reject) => {
-        const hasValidNotification = notificationObj.payload.some((notify) => {
+        const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
-          return (
+        fredy.execute().then((listings) => {
-            typeof notify.id === 'string' &&
+          if (listings == null || listings.length === 0) {
-            typeof notify.price === 'string' &&
+            reject('Listings is empty!');
-            notify.price.includes('€') &&
+            return;
-            typeof notify.size === 'string' &&
+          }
-            notify.size.includes('m²') &&
+
-            typeof notify.title === 'string' &&
+          liveListings = listings;
-            notify.title !== '' &&
+          expect(listings).toBeInstanceOf(Array);
-            typeof notify.link === 'string' &&
+          const notificationObj = get();
-            notify.link.includes('https://www.immobilienscout24.de/') &&
+          expect(notificationObj).toBeTypeOf('object');
-            typeof notify.address === 'string'
+
-          );
+          // check if there is at least one valid notification
+          const hasValidNotification = notificationObj.payload.some((notify) => {
+            return (
+              typeof notify.id === 'string' &&
+              typeof notify.price === 'string' &&
+              notify.price.includes('€') &&
+              typeof notify.size === 'string' &&
+              notify.size.includes('m²') &&
+              typeof notify.title === 'string' &&
+              notify.title !== '' &&
+              typeof notify.link === 'string' &&
+              notify.link.includes('https://www.immobilienscout24.de/') &&
+              typeof notify.address === 'string'
+            );
+          });
+
+          expect(hasValidNotification).toBe(true);
+          resolve();
         });
-
-        expect(hasValidNotification).toBe(true);
-        resolve();
       });
-    });
+    },
-  });
+    TEST_TIMEOUT,
+  );
 
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
+    it(
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
+      'should enrich listings with details',
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
+      async () => {
-    });
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
+        // Call fetchDetails directly on the first live listing — no need to
-      vi.restoreAllMocks();
+        // re-query the search API. immoscout uses fetch (no browser).
-    });
+        const enriched = await provider.config.fetchDetails(liveListings[0]);
 
-    it('should enrich listings with details', async () => {
+        expect(enriched).toBeTruthy();
-      const Fredy = await mockFredy();
+        expect(enriched.description).toBeTypeOf('string');
-      provider.init(providerConfig.immoscout, [], []);
+        expect(enriched.description).not.toBe('');
-      const mockedJob = { id: '', notificationAdapter: null, specFilter: null, spatialFilter: null };
+      },
-      const fredy = new Fredy(
+      TEST_TIMEOUT,
-        provider.config,
+    );
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.description).toBeTypeOf('string');
-        expect(listing.description).not.toBe('');
-      });
-    });
   });
 });

test/provider/immowelt.test.js

@@ -6,87 +6,95 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { mockFredy, providerConfig } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/immowelt.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session. Immowelt's CDN challenges cold sessions
+// aggressively; a shared warm browser prevents the second request from being
+// blocked as a bot hit.
+const TEST_TIMEOUT = 180_000;
 
 describe('#immowelt testsuite()', () => {
-  it('should test immowelt provider', async () => {
+  let browser;
-    const Fredy = await mockFredy();
+  let liveListings;
-    const mockedJob = {
-      id: 'immowelt',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
-    provider.init(providerConfig.immowelt, [], []);
 
-    const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  beforeAll(async () => {
+    browser = await launchBrowser(providerConfig.immowelt.url);
+  }, TEST_TIMEOUT);
 
-    const listing = await fredy.execute();
+  afterAll(async () => {
-
+    await closeBrowser(browser);
-    if (listing == null || listing.length === 0) {
-      throw new Error('Listings is empty!');
-    }
-
-    expect(listing).toBeInstanceOf(Array);
-    const notificationObj = get();
-    expect(notificationObj).toBeTypeOf('object');
-    expect(notificationObj.serviceName).toBe('immowelt');
-    notificationObj.payload.forEach((notify) => {
-      /** check the actual structure **/
-      expect(notify.id).toBeTypeOf('string');
-      if (notify.price != null) {
-        expect(notify.price).toBeTypeOf('string');
-        expect(notify.price).toContain('€');
-      }
-      expect(notify.title).toBeTypeOf('string');
-      expect(notify.link).toBeTypeOf('string');
-      expect(notify.address).toBeTypeOf('string');
-      /** check the values if possible **/
-      if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
-        expect(notify.size).toBeTypeOf('string');
-        expect(notify.size).toContain('m²');
-      }
-      expect(notify.title).not.toBe('');
-      expect(notify.link).toContain('https://www.immowelt.de');
-      expect(notify.address).not.toBe('');
-    });
   });
 
+  it(
+    'should test immowelt provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'immowelt',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+      provider.init(providerConfig.immowelt, [], []);
+
+      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+      liveListings = await fredy.execute();
+
+      if (liveListings == null || liveListings.length === 0) {
+        throw new Error('Listings is empty!');
+      }
+
+      expect(liveListings).toBeInstanceOf(Array);
+      const notificationObj = get();
+      expect(notificationObj).toBeTypeOf('object');
+      expect(notificationObj.serviceName).toBe('immowelt');
+      notificationObj.payload.forEach((notify) => {
+        /** check the actual structure **/
+        expect(notify.id).toBeTypeOf('string');
+        if (notify.price != null) {
+          expect(notify.price).toBeTypeOf('string');
+          expect(notify.price).toContain('€');
+        }
+        expect(notify.title).toBeTypeOf('string');
+        expect(notify.link).toBeTypeOf('string');
+        expect(notify.address).toBeTypeOf('string');
+        /** check the values if possible **/
+        if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
+          expect(notify.size).toBeTypeOf('string');
+          expect(notify.size).toContain('m²');
+        }
+        expect(notify.title).not.toBe('');
+        expect(notify.link).toContain('https://www.immowelt.de');
+        expect(notify.address).not.toBe('');
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
+    it(
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
+      'should enrich listings with details',
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
+      async () => {
-    });
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
+        // Call fetchDetails directly on the first live listing — no need to
-      vi.restoreAllMocks();
+        // re-scrape the search page. The shared browser keeps the session warm.
-    });
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
+        expect(enriched).toBeTruthy();
-      const Fredy = await mockFredy();
+        expect(enriched.link).toContain('https://www.immowelt.de');
-      provider.init(providerConfig.immowelt, [], []);
+        expect(enriched.address).toBeTypeOf('string');
-      const mockedJob = { id: 'immowelt', notificationAdapter: null, specFilter: null, spatialFilter: null };
+        expect(enriched.address).not.toBe('');
-
-      const fredy = new Fredy(
-        provider.config,
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.immowelt.de');
-        expect(listing.address).toBeTypeOf('string');
-        expect(listing.address).not.toBe('');
         // description is enriched from the detail page; falls back gracefully if blocked
-        if (listing.description != null) {
+        if (enriched.description != null) {
-          expect(listing.description).toBeTypeOf('string');
+          expect(enriched.description).toBeTypeOf('string');
         }
-      });
+      },
-    });
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/kleinanzeigen.test.js

@@ -6,80 +6,88 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { mockFredy, providerConfig } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/kleinanzeigen.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session. Kleinanzeigen rate-limits cold browser
+// sessions; a shared warm browser prevents the second request from being blocked.
+const TEST_TIMEOUT = 180_000;
 
 describe('#kleinanzeigen testsuite()', () => {
-  it('should test kleinanzeigen provider', async () => {
+  let browser;
-    const Fredy = await mockFredy();
+  let liveListings;
-    const mockedJob = {
-      id: 'kleinanzeigen',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
-    provider.init(providerConfig.kleinanzeigen, [], []);
-    return await new Promise((resolve, reject) => {
-      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
 
-      fredy.execute().then((listing) => {
+  beforeAll(async () => {
-        if (listing == null || listing.length === 0) {
+    browser = await launchBrowser(providerConfig.kleinanzeigen.url);
-          reject('Listings is empty!');
+  }, TEST_TIMEOUT);
-          return;
-        }
 
-        expect(listing).toBeInstanceOf(Array);
+  afterAll(async () => {
-        const notificationObj = get();
+    await closeBrowser(browser);
-        expect(notificationObj).toBeTypeOf('object');
-        expect(notificationObj.serviceName).toBe('kleinanzeigen');
-        notificationObj.payload.forEach((notify) => {
-          /** check the actual structure **/
-          expect(notify.id).toBeTypeOf('string');
-          expect(notify.title).toBeTypeOf('string');
-          expect(notify.link).toBeTypeOf('string');
-          expect(notify.address).toBeTypeOf('string');
-          /** check the values if possible **/
-          expect(notify.title).not.toBe('');
-          expect(notify.link).toContain('https://www.kleinanzeigen.de');
-          expect(notify.address).not.toBe('');
-        });
-        resolve();
-      });
-    });
   });
 
+  it(
+    'should test kleinanzeigen provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'kleinanzeigen',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+      provider.init(providerConfig.kleinanzeigen, [], []);
+      return await new Promise((resolve, reject) => {
+        const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+        fredy.execute().then((listing) => {
+          if (listing == null || listing.length === 0) {
+            reject('Listings is empty!');
+            return;
+          }
+
+          liveListings = listing;
+          expect(listing).toBeInstanceOf(Array);
+          const notificationObj = get();
+          expect(notificationObj).toBeTypeOf('object');
+          expect(notificationObj.serviceName).toBe('kleinanzeigen');
+          notificationObj.payload.forEach((notify) => {
+            /** check the actual structure **/
+            expect(notify.id).toBeTypeOf('string');
+            expect(notify.title).toBeTypeOf('string');
+            expect(notify.link).toBeTypeOf('string');
+            expect(notify.address).toBeTypeOf('string');
+            /** check the values if possible **/
+            expect(notify.title).not.toBe('');
+            expect(notify.link).toContain('https://www.kleinanzeigen.de');
+            expect(notify.address).not.toBe('');
+          });
+          resolve();
+        });
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
+    it(
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
+      'should enrich listings with details',
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
+      async () => {
-    });
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
+        // Call fetchDetails directly on the first live listing — no need to
-      vi.restoreAllMocks();
+        // re-scrape the search page. The shared browser keeps the session warm.
-    });
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
+        expect(enriched).toBeTruthy();
-      const Fredy = await mockFredy();
+        expect(enriched.link).toContain('https://www.kleinanzeigen.de');
-      provider.init(providerConfig.kleinanzeigen, [], []);
+        expect(enriched.address).toBeTypeOf('string');
-      const mockedJob = { id: 'kleinanzeigen', notificationAdapter: null, specFilter: null, spatialFilter: null };
+        expect(enriched.address).not.toBe('');
-
+        expect(enriched.description).toBeTypeOf('string');
-      const fredy = new Fredy(
+        expect(enriched.description).not.toBe('');
-        provider.config,
+      },
-        mockedJob,
+      TEST_TIMEOUT,
-        provider.metaInformation.id,
+    );
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.kleinanzeigen.de');
-        expect(listing.address).toBeTypeOf('string');
-        expect(listing.address).not.toBe('');
-        expect(listing.description).toBeTypeOf('string');
-        expect(listing.description).not.toBe('');
-      });
-    });
   });
 });

test/provider/sparkasse.test.js

@@ -9,81 +9,97 @@ import { mockFredy, providerConfig } from '../utils.js';
 import { expect, vi } from 'vitest';
 import * as provider from '../../lib/provider/sparkasse.js';
 import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session. This prevents the second request from being
+// flagged as a cold-start bot hit.
+const TEST_TIMEOUT = 120_000;
 
 describe('#sparkasse testsuite()', () => {
-  it('should test sparkasse provider', async () => {
+  let browser;
-    const Fredy = await mockFredy();
+  let liveListings;
-    const mockedJob = {
-      id: 'sparkasse',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
-    provider.init(providerConfig.sparkasse, []);
 
-    const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  beforeAll(async () => {
+    browser = await launchBrowser(providerConfig.sparkasse.url);
+  }, TEST_TIMEOUT);
 
-    const listing = await fredy.execute();
+  afterAll(async () => {
-
+    await closeBrowser(browser);
-    if (listing == null || listing.length === 0) {
-      throw new Error('Listings is empty!');
-    }
-
-    expect(listing).toBeInstanceOf(Array);
-    const notificationObj = get();
-    expect(notificationObj).toBeTypeOf('object');
-    expect(notificationObj.serviceName).toBe('sparkasse');
-    notificationObj.payload.forEach((notify) => {
-      /** check the actual structure **/
-      expect(notify.id).toBeTypeOf('string');
-      expect(notify.price).toBeTypeOf('string');
-      expect(notify.price).toContain('€');
-      expect(notify.size).toBeTypeOf('string');
-      expect(notify.size).toContain('m²');
-      expect(notify.title).toBeTypeOf('string');
-      expect(notify.link).toBeTypeOf('string');
-      expect(notify.address).toBeTypeOf('string');
-      /** check the values if possible **/
-      expect(notify.size).toBeTypeOf('string');
-      expect(notify.title).not.toBe('');
-      expect(notify.address).not.toBe('');
-    });
   });
 
+  it(
+    'should test sparkasse provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'sparkasse',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+      provider.init(providerConfig.sparkasse, []);
+
+      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+      liveListings = await fredy.execute();
+
+      if (liveListings == null || liveListings.length === 0) {
+        throw new Error('Listings is empty!');
+      }
+
+      expect(liveListings).toBeInstanceOf(Array);
+      const notificationObj = get();
+      expect(notificationObj).toBeTypeOf('object');
+      expect(notificationObj.serviceName).toBe('sparkasse');
+      notificationObj.payload.forEach((notify) => {
+        /** check the actual structure **/
+        expect(notify.id).toBeTypeOf('string');
+        expect(notify.price).toBeTypeOf('string');
+        expect(notify.price).toContain('€');
+        expect(notify.size).toBeTypeOf('string');
+        expect(notify.size).toContain('m²');
+        expect(notify.title).toBeTypeOf('string');
+        expect(notify.link).toBeTypeOf('string');
+        expect(notify.address).toBeTypeOf('string');
+        /** check the values if possible **/
+        expect(notify.size).toBeTypeOf('string');
+        expect(notify.title).not.toBe('');
+        expect(notify.address).not.toBe('');
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
     beforeEach(() => {
       vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
     });
 
     afterEach(() => {
       vi.restoreAllMocks();
     });
 
-    it('should enrich listings with details', async () => {
+    it(
-      const Fredy = await mockFredy();
+      'should enrich listings with details',
-      provider.init(providerConfig.sparkasse, []);
+      async () => {
-      const mockedJob = { id: 'sparkasse', notificationAdapter: null, specFilter: null, spatialFilter: null };
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-      const fredy = new Fredy(
+        // Call fetchDetails directly on the first live listing — no need to
-        provider.config,
+        // re-scrape the search page. The shared browser keeps the session warm.
-        mockedJob,
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
-        provider.metaInformation.id,
+
-        { checkAndAddEntry: () => false },
+        expect(enriched).toBeTruthy();
-        undefined,
+        expect(enriched.link).toContain('https://immobilien.sparkasse.de');
-      );
+        expect(enriched.address).toBeTypeOf('string');
-      const listings = await fredy.execute();
+        expect(enriched.address).not.toBe('');
-      expect(listings).toBeInstanceOf(Array);
+        // description is enriched from the detail page; falls back gracefully if blocked
-      listings.forEach((listing) => {
+        if (enriched.description != null) {
-        expect(listing.link).toContain('https://immobilien.sparkasse.de');
+          expect(enriched.description).toBeTypeOf('string');
-        expect(listing.address).toBeTypeOf('string');
+          expect(enriched.description).not.toBe('');
-        expect(listing.address).not.toBe('');
-        // description is enriched from the detail page; falls back gracefully if bot-detected
-        if (listing.description != null) {
-          expect(listing.description).toBeTypeOf('string');
-          expect(listing.description).not.toBe('');
         }
-      });
+      },
-    });
+      TEST_TIMEOUT,
+    );
   });
 });

test/provider/wgGesucht.test.js

@@ -6,77 +6,85 @@
 import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
 import { get } from '../mocks/mockNotification.js';
 import { mockFredy, providerConfig } from '../utils.js';
-import { expect, vi } from 'vitest';
+import { expect } from 'vitest';
 import * as provider from '../../lib/provider/wgGesucht.js';
-import * as mockStore from '../mocks/mockStore.js';
+import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
+
+// One browser shared across the whole suite so both requests (search + detail)
+// come from the same warm session, avoiding double cold-start bot detection.
+const TEST_TIMEOUT = 120_000;
 
 describe('#wgGesucht testsuite()', () => {
   provider.init(providerConfig.wgGesucht, [], []);
-  it('should test wgGesucht provider', { timeout: 120000 }, async () => {
-    const Fredy = await mockFredy();
-    const mockedJob = {
-      id: 'wgGesucht',
-      notificationAdapter: null,
-      spatialFilter: null,
-      specFilter: null,
-    };
 
-    return await new Promise((resolve, reject) => {
+  let browser;
-      const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
+  let liveListings;
 
-      fredy.execute().then((listing) => {
+  beforeAll(async () => {
-        if (listing == null || listing.length === 0) {
+    browser = await launchBrowser(providerConfig.wgGesucht.url);
-          reject('Listings is empty!');
+  }, TEST_TIMEOUT);
-          return;
-        }
 
-        expect(listing).toBeInstanceOf(Array);
+  afterAll(async () => {
-        const notificationObj = get();
+    await closeBrowser(browser);
-        expect(notificationObj.serviceName).toBe('wgGesucht');
-        notificationObj.payload.forEach((notify) => {
-          expect(notify).toBeTypeOf('object');
-          /** check the actual structure **/
-          expect(notify.id).toBeTypeOf('string');
-          expect(notify.title).toBeTypeOf('string');
-          // expect(notify.details).toBeTypeOf('string');
-          expect(notify.price).toBeTypeOf('string');
-          expect(notify.price).toContain('€');
-          expect(notify.link).toBeTypeOf('string');
-        });
-        resolve();
-      });
-    });
   });
 
+  it(
+    'should test wgGesucht provider',
+    async () => {
+      const Fredy = await mockFredy();
+      const mockedJob = {
+        id: 'wgGesucht',
+        notificationAdapter: null,
+        spatialFilter: null,
+        specFilter: null,
+      };
+
+      return await new Promise((resolve, reject) => {
+        const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
+
+        fredy.execute().then((listing) => {
+          if (listing == null || listing.length === 0) {
+            reject('Listings is empty!');
+            return;
+          }
+
+          liveListings = listing;
+          expect(listing).toBeInstanceOf(Array);
+          const notificationObj = get();
+          expect(notificationObj.serviceName).toBe('wgGesucht');
+          notificationObj.payload.forEach((notify) => {
+            expect(notify).toBeTypeOf('object');
+            /** check the actual structure **/
+            expect(notify.id).toBeTypeOf('string');
+            expect(notify.title).toBeTypeOf('string');
+            // expect(notify.details).toBeTypeOf('string');
+            expect(notify.price).toBeTypeOf('string');
+            expect(notify.price).toContain('€');
+            expect(notify.link).toBeTypeOf('string');
+          });
+          resolve();
+        });
+      });
+    },
+    TEST_TIMEOUT,
+  );
+
   describe('with provider_details enabled', () => {
-    beforeEach(() => {
+    it(
-      vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
+      'should enrich listings with details',
-      vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
+      async () => {
-    });
+        if (!liveListings?.length) throw new Error('No listings from first test to enrich');
 
-    afterEach(() => {
+        // Call fetchDetails directly on the first live listing — no need to
-      vi.restoreAllMocks();
+        // re-scrape the search page. The shared browser keeps the session warm.
-    });
+        const enriched = await provider.config.fetchDetails(liveListings[0], browser);
 
-    it('should enrich listings with details', async () => {
+        expect(enriched).toBeTruthy();
-      const Fredy = await mockFredy();
+        expect(enriched.link).toContain('https://www.wg-gesucht.de');
-      provider.init(providerConfig.wgGesucht, [], []);
+        expect(enriched.description).toBeTypeOf('string');
-      const mockedJob = { id: 'wgGesucht', notificationAdapter: null, specFilter: null, spatialFilter: null };
+        expect(enriched.description).not.toBe('');
-
+      },
-      const fredy = new Fredy(
+      TEST_TIMEOUT,
-        provider.config,
+    );
-        mockedJob,
-        provider.metaInformation.id,
-        { checkAndAddEntry: () => false },
-        undefined,
-      );
-      const listings = await fredy.execute();
-      expect(listings).toBeInstanceOf(Array);
-      listings.forEach((listing) => {
-        expect(listing.link).toContain('https://www.wg-gesucht.de');
-        expect(listing.description).toBeTypeOf('string');
-        expect(listing.description).not.toBe('');
-      });
-    });
   });
 });

test/services/extractor/puppeteerExtractor.test.js

@@ -0,0 +1,37 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { vi, describe, it, expect, beforeEach } from 'vitest';
+
+// Mock the CloakBrowser launcher so no real Chromium binary is needed and we can
+// assert which options get forwarded to it.
+const { launchMock } = vi.hoisted(() => ({ launchMock: vi.fn() }));
+
+vi.mock('cloakbrowser/puppeteer', () => ({
+  launch: launchMock,
+}));
+
+const { launchBrowser } = await import('../../../lib/services/extractor/puppeteerExtractor.js');
+
+describe('launchBrowser proxy forwarding', () => {
+  beforeEach(() => {
+    launchMock.mockReset();
+    launchMock.mockResolvedValue({ close: async () => {} });
+  });
+
+  it('forwards proxyUrl to CloakBrowser as the proxy option', async () => {
+    await launchBrowser('https://www.immowelt.de/', { proxyUrl: 'http://user:pass@host:8080' });
+
+    expect(launchMock).toHaveBeenCalledTimes(1);
+    expect(launchMock.mock.calls[0][0]).toMatchObject({ proxy: 'http://user:pass@host:8080' });
+  });
+
+  it('does not set a proxy when no proxyUrl is given', async () => {
+    await launchBrowser('https://www.immowelt.de/', {});
+
+    expect(launchMock).toHaveBeenCalledTimes(1);
+    expect(launchMock.mock.calls[0][0].proxy).toBeUndefined();
+  });
+});

test/services/immoscout/immoscout-web-translator.test.js

@@ -4,12 +4,28 @@
  */
 
 import { convertWebToMobile } from '../../../lib/services/immoscout/immoscout-web-translator.js';
-import { expect } from 'vitest';
+import { expect, vi } from 'vitest';
 import { readFile } from 'fs/promises';
+import { buildFetchMock } from '../../offlineFixtures.js';
 
 export const testData = JSON.parse(await readFile(new URL('./testdata.json', import.meta.url)));
 
+if (process.env.TEST_MODE === 'offline') {
+  vi.stubGlobal('fetch', buildFetchMock());
+}
+
 describe('#immoscout-mobile URL conversion', () => {
+  // Test shape URL conversion
+  it('should convert a full web URL with shape to mobile URL', () => {
+    const webUrl =
+      'https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list';
+    const expectedMobileUrl =
+      'https://api.mobile.immobilienscout24.de/search/list?ground=240.0-&price=-600000.0&realestatetype=housebuy&searchType=shape&shape=ior~H_kxmAr%60Pig%60%40fzHm%7Bp%40qs%40%7D%60l%40crCidPuol%40wy~%40onVb~Ek%60KhdPcoEhkS%7CHf%7B%7CAtpDtqLjiGngQ%7CxLrlOxyVbzR';
+
+    const actualMobileUrl = convertWebToMobile(webUrl);
+    expect(actualMobileUrl).toBe(expectedMobileUrl);
+  });
+
   // Test URL conversion
   it('should convert a full web URL to mobile URL', () => {
     const webUrl =
@@ -30,6 +46,60 @@ describe('#immoscout-mobile URL conversion', () => {
     expect(queryParams.get('equipment').split(',')).toEqual(expect.arrayContaining(['garden', 'balcony']));
   });
 
+  // Test URL conversion of SEO web path for max warmrent. The ImmoScout web UI
+  // generates this special SEO slug instead of explicit price/pricetype params
+  // when the user configures a "Warmmiete" filter (real-world URL).
+  it('should convert a SEO apartment max warmrent path to rent + price + pricetype', () => {
+    const webUrl =
+      'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-bis-800-euro-warm?livingspace=-800.0&enteredFrom=result_list';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('apartmentrent');
+    expect(queryParams.get('price')).toBe('-800');
+    expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
+    expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
+    expect(queryParams.get('livingspace')).toBe('-800.0');
+  });
+
+  // Same SEO pattern for houses ("haus-bis-X-euro-warm" → houserent).
+  it('should convert a SEO house max warmrent path to rent + price + pricetype', () => {
+    const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/haus-bis-1500-euro-warm';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('houserent');
+    expect(queryParams.get('price')).toBe('-1500');
+    expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
+  });
+
+  // Sanity check: max coldrent ("Kaltmiete") does NOT use an SEO slug. The web
+  // UI keeps the regular "wohnung-mieten" path and passes explicit
+  // price + pricetype query params, which the existing translator already
+  // handles (real-world URL).
+  it('should convert a max coldrent search via the regular wohnung-mieten path', () => {
+    const webUrl =
+      'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-mieten?price=-800.0&livingspace=-800.0&pricetype=rentpermonth&enteredFrom=result_list';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('apartmentrent');
+    expect(queryParams.get('price')).toBe('-800.0');
+    expect(queryParams.get('pricetype')).toBe('rentpermonth');
+    expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
+  });
+
+  // Explicit query params win over the SEO slug's implicit defaults.
+  it('should let explicit query params override SEO path price defaults', () => {
+    const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-bis-800-euro-warm?price=100-500';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('apartmentrent');
+    expect(queryParams.get('price')).toBe('100-500');
+    expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
+  });
+
   // Test URL conversion with unsupported query parameters
   it('should remove unsupported query parameters', () => {
     const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?minimuminternetspeed=100000';

test/services/immoscout/testdata.json

@@ -18,5 +18,9 @@
   "rentHouse": {
     "url": "https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/haus-mieten?enteredFrom=one_step_search",
     "type": "houserent"
+  },
+  "buyHouseWithShape": {
+    "url": "https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list",
+    "type": "housebuy"
   }
 }

test/services/jobs/jobExecutionService.test.js

@@ -18,6 +18,7 @@ describe('services/jobs/jobExecutionService', () => {
     const busPath = root + '/lib/services/events/event-bus.js';
     const jobStoragePath = root + '/lib/services/storage/jobStorage.js';
     const userStoragePath = root + '/lib/services/storage/userStorage.js';
+    const settingsStoragePath = root + '/lib/services/storage/settingsStorage.js';
     const brokerPath = root + '/lib/services/sse/sse-broker.js';
     const utilsPath = root + '/lib/utils.js';
     const loggerPath = root + '/lib/services/logger.js';
@@ -33,11 +34,15 @@ describe('services/jobs/jobExecutionService', () => {
       getUsers: () => state.users.slice(),
       getUser: (id) => state.users.find((u) => u.id === id) || null,
     }));
+    vi.doMock(settingsStoragePath, () => ({
+      getSettings: async () => ({}),
+    }));
     vi.doMock(brokerPath, () => ({
       sendToUsers: (...args) => calls.sent.push(args),
     }));
     vi.doMock(utilsPath, () => ({
       duringWorkingHoursOrNotSet: () => false,
+      getPackageVersion: async () => '0.0.0-test',
     }));
     vi.doMock(loggerPath, () => {
       const m = { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} };

test/utils.js

@@ -29,7 +29,7 @@ vi.mock('../lib/services/extractor/puppeteerExtractor.js', async (importOriginal
   const { readFixture } = await import('./offlineFixtures.js');
   return {
     default: (url) => readFixture(url),
-    launchBrowser: async () => ({ close: async () => {}, __fredy_removeUserDataDir: false }),
+    launchBrowser: async () => ({ close: async () => {}, isConnected: () => true }),
     closeBrowser: async () => {},
   };
 });

tools/devMock.js

@@ -155,6 +155,7 @@ const routes = {
   'GET /api/dashboard': dashboard,
   'GET /api/demo': { demoMode: false },
   'POST /api/user/settings/news-hash': {},
+  'POST /api/user/settings/listing-deletion-preference': {},
 };
 
 const server = http.createServer((req, res) => {

tools/testFixtures/downloadFixtures.js

@@ -95,7 +95,10 @@ async function downloadHtmlProvider(name, providerConfig, launchBrowser, closeBr
 
   const browser = await launchBrowser(providerConfig.url, {});
   try {
-    const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, { browser });
+    const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, {
+      browser,
+      name: 'dowload_fixtures',
+    });
 
     if (!html) {
       console.warn(`  Failed to download ${name}`);

ui/src/assets/news/news.json

@@ -1,11 +1,16 @@
 {
-  "key": "00e6b81777a275f5a140fc9101cb943810db6a69f6eb3927319c5aee0c876515",
+  "key": "00e6b81777a275f5a140fc9101cb943810db6a69f6eb3927319c5aee0c876221",
   "content":
 [
   {
     "title": "Table overview for listings",
     "text": "Thanks to https://github.com/datenwurm, we now have a table overview for listings. If you decide to use the table view, the decision will be stored.",
     "media": "1.png"
+  },
+  {
+    "title": "Table overview for jobs",
+    "text": "Based on datenwurm's, work, I created a table overview for jobs. If you decide to use the table view, the decision will be stored.",
+    "media": "2.png"
   }
 ]
 }

ui/src/components/ListingDeletionModal.jsx

@@ -3,8 +3,8 @@
  * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
  */
 
-import { useState } from 'react';
+import { useState, useEffect } from 'react';
-import { Modal, Radio, RadioGroup, Typography } from '@douyinfe/semi-ui-19';
+import { Modal, Radio, RadioGroup, Typography, Checkbox } from '@douyinfe/semi-ui-19';
 
 const { Text } = Typography;
 
@@ -15,11 +15,24 @@ const ListingDeletionModal = ({
   title = 'Delete Listings',
   showOptions = true,
   message = 'How would you like to delete the selected listing(s)?',
+  defaultDeleteType = 'soft',
 }) => {
   const [deleteType, setDeleteType] = useState('soft');
+  const [remember, setRemember] = useState(false);
+
+  useEffect(() => {
+    if (visible) {
+      setDeleteType(defaultDeleteType);
+      setRemember(false);
+    }
+  }, [visible, defaultDeleteType]);
 
   const handleOk = () => {
-    onConfirm(!showOptions || deleteType === 'hard');
+    if (showOptions) {
+      onConfirm(deleteType === 'hard', remember);
+    } else {
+      onConfirm(true);
+    }
   };
 
   return (
@@ -36,32 +49,37 @@ const ListingDeletionModal = ({
         <Text>{message}</Text>
       </div>
       {showOptions && (
-        <RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
+        <>
-          <Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
+          <RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
-            <div style={{ marginLeft: 8 }}>
+            <Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
-              <Text strong>Mark as deleted (Soft Delete)</Text>
+              <div style={{ marginLeft: 8 }}>
-              <br />
+                <Text strong>Mark as deleted (Soft Delete)</Text>
-              <Text type="secondary">
-                Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
-                scraping session.
-              </Text>
-            </div>
-          </Radio>
-          <Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
-            <div style={{ marginLeft: 8 }}>
-              <Text strong>Remove from database (Hard Delete)</Text>
-              <br />
-              <Text type="secondary">
-                Listings are completely removed from the database.
                 <br />
-                <Text type="warning">
+                <Text type="secondary">
-                  Consequence: They might re-appear when scraping the next time because Fredy won't know they were
+                  Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
-                  previously found.
+                  scraping session.
                 </Text>
-              </Text>
+              </div>
-            </div>
+            </Radio>
-          </Radio>
+            <Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
-        </RadioGroup>
+              <div style={{ marginLeft: 8 }}>
+                <Text strong>Remove from database (Hard Delete)</Text>
+                <br />
+                <Text type="secondary">
+                  Listings are completely removed from the database.
+                  <br />
+                  <Text type="warning">
+                    Consequence: They might re-appear when scraping the next time because Fredy won't know they were
+                    previously found.
+                  </Text>
+                </Text>
+              </div>
+            </Radio>
+          </RadioGroup>
+          <Checkbox checked={remember} onChange={(e) => setRemember(e.target.checked)} style={{ marginTop: 16 }}>
+            Remember my choice and skip this dialog next time
+          </Checkbox>
+        </>
       )}
     </Modal>
   );

ui/src/components/grid/jobs/JobGrid.jsx

@@ -21,6 +21,7 @@ import {
   Empty,
   Radio,
   RadioGroup,
+  Tooltip,
 } from '@douyinfe/semi-ui-19';
 import {
   IconAlertTriangle,
@@ -35,6 +36,8 @@ import {
   IconArrowUp,
   IconArrowDown,
   IconHome,
+  IconGridView,
+  IconList,
 } from '@douyinfe/semi-icons';
 import { useNavigate } from 'react-router-dom';
 import ListingDeletionModal from '../../ListingDeletionModal.jsx';
@@ -42,6 +45,7 @@ import { useActions, useSelector } from '../../../services/state/store.js';
 import { xhrDelete, xhrPut, xhrPost } from '../../../services/xhr.js';
 import { debounce } from '../../../utils';
 import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-illustrations';
+import JobsTable from '../../table/JobsTable.jsx';
 
 import './JobGrid.less';
 
@@ -54,6 +58,11 @@ const JobGrid = () => {
   const actions = useActions();
   const navigate = useNavigate();
 
+  const userSettings = useSelector((state) => state.userSettings.settings);
+  const viewMode = userSettings?.jobs_view_mode ?? 'grid';
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
+
   const [page, setPage] = useState(1);
   const pageSize = 12;
 
@@ -135,13 +144,21 @@ const JobGrid = () => {
   };
 
   const onListingRemoval = (jobId) => {
-    setPendingDeletion({ type: 'listings', jobId });
+    const deletion = { type: 'listings', jobId };
+    if (listingDeletionPref?.skipPrompt) {
+      confirmDeletion(listingDeletionPref.hardDelete, false, deletion);
+      return;
+    }
+    setPendingDeletion(deletion);
     setDeleteModalVisible(true);
   };
 
-  const confirmDeletion = async (hardDelete) => {
+  const confirmDeletion = async (hardDelete, remember, deletion = pendingDeletion) => {
-    const { type, jobId } = pendingDeletion;
+    const { type, jobId } = deletion;
     try {
+      if (remember && type === 'listings') {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
       if (type === 'job') {
         await xhrDelete('/api/jobs', { jobId });
         Toast.success('Job and listings successfully removed');
@@ -167,7 +184,7 @@ const JobGrid = () => {
       Toast.success('Job status successfully changed');
       loadData();
     } catch (error) {
-      Toast.error(error);
+      Toast.error(error.error);
     }
   };
 
@@ -234,6 +251,27 @@ const JobGrid = () => {
           onClick={() => setSortDir((d) => (d === 'asc' ? 'desc' : 'asc'))}
           title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
         />
+
+        <div className="jobGrid__topbar__view-toggle">
+          <Tooltip content="Grid view">
+            <Button
+              icon={<IconGridView />}
+              theme={viewMode === 'grid' ? 'solid' : 'borderless'}
+              onClick={() => actions.userSettings.setJobsViewMode('grid')}
+              aria-label="Grid view"
+              aria-pressed={viewMode === 'grid'}
+            />
+          </Tooltip>
+          <Tooltip content="Table view">
+            <Button
+              icon={<IconList />}
+              theme={viewMode === 'table' ? 'solid' : 'borderless'}
+              onClick={() => actions.userSettings.setJobsViewMode('table')}
+              aria-label="Table view"
+              aria-pressed={viewMode === 'table'}
+            />
+          </Tooltip>
+        </div>
       </div>
 
       {(jobsData?.result || []).length === 0 && (
@@ -244,136 +282,144 @@ const JobGrid = () => {
         />
       )}
 
-      <Row gutter={[16, 16]}>
+      {viewMode === 'grid' ? (
-        {(jobsData?.result || []).map((job) => (
+        <Row gutter={[16, 16]}>
-          <Col key={job.id} xs={24} sm={12} md={12} lg={8} xl={8} xxl={6}>
+          {(jobsData?.result || []).map((job) => (
-            <Card className="jobGrid__card" bodyStyle={{ padding: '16px' }}>
+            <Col key={job.id} xs={24} sm={12} md={12} lg={8} xl={8} xxl={6}>
-              <div className="jobGrid__card__header">
+              <Card className="jobGrid__card" bodyStyle={{ padding: '16px' }}>
-                <div className="jobGrid__card__name">
+                <div className="jobGrid__card__header">
-                  <span className={`jobGrid__card__dot${job.enabled ? ' jobGrid__card__dot--active' : ''}`} />
+                  <div className="jobGrid__card__name">
-                  <Title heading={5} ellipsis={{ showTooltip: true }} className="jobGrid__title">
+                    <span className={`jobGrid__card__dot${job.enabled ? ' jobGrid__card__dot--active' : ''}`} />
-                    {job.name}
+                    <Title heading={5} ellipsis={{ showTooltip: true }} className="jobGrid__title">
-                  </Title>
+                      {job.name}
+                    </Title>
+                  </div>
+                  <div style={{ display: 'flex', alignItems: 'center', gap: 6, flexShrink: 0 }}>
+                    {job.isOnlyShared && (
+                      <Popover content={getPopoverContent('This job has been shared with you — read only.')}>
+                        <div>
+                          <IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
+                        </div>
+                      </Popover>
+                    )}
+                    {job.running && (
+                      <Tag color="green" variant="light" size="small">
+                        RUNNING
+                      </Tag>
+                    )}
+                  </div>
                 </div>
-                <div style={{ display: 'flex', alignItems: 'center', gap: 6, flexShrink: 0 }}>
+
-                  {job.isOnlyShared && (
+                <div className="jobGrid__card__stats">
-                    <Popover
+                  <div className="jobGrid__card__stat jobGrid__card__stat--blue">
-                      content={getPopoverContent(
+                    <span className="jobGrid__card__stat__number">{job.numberOfFoundListings || 0}</span>
-                        'This job has been shared with you by another user, therefor it is read-only.',
+                    <span className="jobGrid__card__stat__label">
-                      )}
+                      <IconHome size="small" /> Listings
-                    >
+                    </span>
+                  </div>
+                  <div className="jobGrid__card__stat jobGrid__card__stat--orange">
+                    <span className="jobGrid__card__stat__number">{job.provider?.length || 0}</span>
+                    <span className="jobGrid__card__stat__label">
+                      <IconBriefcase size="small" /> Providers
+                    </span>
+                  </div>
+                  <div className="jobGrid__card__stat jobGrid__card__stat--purple">
+                    <span className="jobGrid__card__stat__number">{job.notificationAdapter?.length || 0}</span>
+                    <span className="jobGrid__card__stat__label">
+                      <IconBell size="small" /> Adapters
+                    </span>
+                  </div>
+                </div>
+
+                <Divider margin="12px" />
+
+                <div className="jobGrid__card__footer">
+                  <div style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
+                    <Switch
+                      onChange={(checked) => onJobStatusChanged(job.id, checked)}
+                      checked={job.enabled}
+                      disabled={job.isOnlyShared}
+                      size="small"
+                    />
+                    <Text type="secondary" size="small">
+                      Active
+                    </Text>
+                  </div>
+                  <div className="jobGrid__actions">
+                    <Popover content={getPopoverContent('Run Job')}>
                       <div>
-                        <IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
+                        <Button
+                          type="primary"
+                          style={{ background: '#21aa21b5' }}
+                          size="small"
+                          theme="solid"
+                          icon={<IconPlayCircle />}
+                          disabled={job.isOnlyShared || job.running}
+                          onClick={() => onJobRun(job.id)}
+                        />
                       </div>
                     </Popover>
-                  )}
+                    <Popover content={getPopoverContent('Edit a Job')}>
-                  {job.running && (
+                      <div>
-                    <Tag color="green" variant="light" size="small">
+                        <Button
-                      RUNNING
+                          type="secondary"
-                    </Tag>
+                          size="small"
-                  )}
+                          icon={<IconEdit />}
+                          disabled={job.isOnlyShared}
+                          onClick={() => navigate(`/jobs/edit/${job.id}`)}
+                        />
+                      </div>
+                    </Popover>
+                    <Popover content={getPopoverContent('Clone Job')}>
+                      <div>
+                        <Button
+                          type="tertiary"
+                          size="small"
+                          icon={<IconCopy />}
+                          disabled={job.isOnlyShared}
+                          onClick={() => navigate('/jobs/new', { state: { cloneFrom: job.id } })}
+                        />
+                      </div>
+                    </Popover>
+                    <Popover content={getPopoverContent('Delete all found Listings of this Job')}>
+                      <div>
+                        <Button
+                          type="danger"
+                          size="small"
+                          icon={<IconDescend2 />}
+                          disabled={job.isOnlyShared}
+                          onClick={() => onListingRemoval(job.id)}
+                        />
+                      </div>
+                    </Popover>
+                    <Popover content={getPopoverContent('Delete Job')}>
+                      <div>
+                        <Button
+                          type="danger"
+                          size="small"
+                          icon={<IconDelete />}
+                          disabled={job.isOnlyShared}
+                          onClick={() => onJobRemoval(job.id)}
+                        />
+                      </div>
+                    </Popover>
+                  </div>
                 </div>
-              </div>
+              </Card>
-
+            </Col>
-              <div className="jobGrid__card__stats">
+          ))}
-                <div className="jobGrid__card__stat jobGrid__card__stat--blue">
+        </Row>
-                  <span className="jobGrid__card__stat__number">{job.numberOfFoundListings || 0}</span>
+      ) : (
-                  <span className="jobGrid__card__stat__label">
+        <JobsTable
-                    <IconHome size="small" /> Listings
+          jobs={jobsData?.result || []}
-                  </span>
+          onRun={onJobRun}
-                </div>
+          onEdit={(id) => navigate(`/jobs/edit/${id}`)}
-                <div className="jobGrid__card__stat jobGrid__card__stat--orange">
+          onClone={(id) => navigate('/jobs/new', { state: { cloneFrom: id } })}
-                  <span className="jobGrid__card__stat__number">{job.provider.length || 0}</span>
+          onDeleteListings={onListingRemoval}
-                  <span className="jobGrid__card__stat__label">
+          onDeleteJob={onJobRemoval}
-                    <IconBriefcase size="small" /> Providers
+          onStatusChange={onJobStatusChanged}
-                  </span>
+        />
-                </div>
+      )}
-                <div className="jobGrid__card__stat jobGrid__card__stat--purple">
-                  <span className="jobGrid__card__stat__number">{job.notificationAdapter.length || 0}</span>
-                  <span className="jobGrid__card__stat__label">
-                    <IconBell size="small" /> Adapters
-                  </span>
-                </div>
-              </div>
-
-              <Divider margin="12px" />
-
-              <div className="jobGrid__card__footer">
-                <div style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
-                  <Switch
-                    onChange={(checked) => onJobStatusChanged(job.id, checked)}
-                    checked={job.enabled}
-                    disabled={job.isOnlyShared}
-                    size="small"
-                  />
-                  <Text type="secondary" size="small">
-                    Active
-                  </Text>
-                </div>
-                <div className="jobGrid__actions">
-                  <Popover content={getPopoverContent('Run Job')}>
-                    <div>
-                      <Button
-                        type="primary"
-                        style={{ background: '#21aa21b5' }}
-                        size="small"
-                        theme="solid"
-                        icon={<IconPlayCircle />}
-                        disabled={job.isOnlyShared || job.running}
-                        onClick={() => onJobRun(job.id)}
-                      />
-                    </div>
-                  </Popover>
-                  <Popover content={getPopoverContent('Edit a Job')}>
-                    <div>
-                      <Button
-                        type="secondary"
-                        size="small"
-                        icon={<IconEdit />}
-                        disabled={job.isOnlyShared}
-                        onClick={() => navigate(`/jobs/edit/${job.id}`)}
-                      />
-                    </div>
-                  </Popover>
-                  <Popover content={getPopoverContent('Clone Job')}>
-                    <div>
-                      <Button
-                        type="tertiary"
-                        size="small"
-                        icon={<IconCopy />}
-                        disabled={job.isOnlyShared}
-                        onClick={() => navigate('/jobs/new', { state: { cloneFrom: job.id } })}
-                      />
-                    </div>
-                  </Popover>
-                  <Popover content={getPopoverContent('Delete all found Listings of this Job')}>
-                    <div>
-                      <Button
-                        type="danger"
-                        size="small"
-                        icon={<IconDescend2 />}
-                        disabled={job.isOnlyShared}
-                        onClick={() => onListingRemoval(job.id)}
-                      />
-                    </div>
-                  </Popover>
-                  <Popover content={getPopoverContent('Delete Job')}>
-                    <div>
-                      <Button
-                        type="danger"
-                        size="small"
-                        icon={<IconDelete />}
-                        disabled={job.isOnlyShared}
-                        onClick={() => onJobRemoval(job.id)}
-                      />
-                    </div>
-                  </Popover>
-                </div>
-              </div>
-            </Card>
-          </Col>
-        ))}
-      </Row>
       {(jobsData?.result || []).length > 0 && jobsData?.totalNumber > 12 && (
         <div className="jobGrid__pagination">
           <Pagination
@@ -389,6 +435,7 @@ const JobGrid = () => {
         visible={deleteModalVisible}
         title={pendingDeletion?.type === 'job' ? 'Delete Job' : 'Delete Listings'}
         showOptions={pendingDeletion?.type !== 'job'}
+        defaultDeleteType={defaultDeleteType}
         message={
           pendingDeletion?.type === 'job'
             ? 'Are you sure you want to delete this job? All associated listings will be removed from the database.'

ui/src/components/grid/jobs/JobGrid.less

@@ -17,6 +17,12 @@
       flex: 1;
       min-width: 160px;
     }
+
+    &__view-toggle {
+      display: flex;
+      gap: 4px;
+      flex-shrink: 0;
+    }
   }
 
   &__card {

ui/src/components/listings/ListingsOverview.jsx

@@ -33,6 +33,8 @@ const ListingsOverview = () => {
   const sp = useSearchParams();
 
   const viewMode = userSettings?.listings_view_mode ?? 'grid';
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
 
   const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
   const pageSize = 40;
@@ -91,15 +93,22 @@ const ListingsOverview = () => {
   };
 
   const handleDelete = (id) => {
+    if (listingDeletionPref?.skipPrompt) {
+      confirmDeletion(listingDeletionPref.hardDelete, false, id);
+      return;
+    }
     setListingToDelete(id);
     setDeleteModalVisible(true);
   };
 
   const handleNavigate = (id) => navigate(`/listings/listing/${id}`);
 
-  const confirmDeletion = async (hardDelete) => {
+  const confirmDeletion = async (hardDelete, remember, id = listingToDelete) => {
     try {
-      await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
+      if (remember) {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
+      await xhrDelete('/api/listings/', { ids: [id], hardDelete });
       Toast.success('Listing successfully removed');
       loadData();
     } catch (error) {
@@ -197,7 +206,7 @@ const ListingsOverview = () => {
 
         <Button
           icon={sortDir === 'asc' ? <IconArrowUp /> : <IconArrowDown />}
-          onClick={() => setSortDir((d) => (d === 'asc' ? 'desc' : 'asc'))}
+          onClick={() => setSortDir(sortDir === 'asc' ? 'desc' : 'asc')}
           title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
         />
 
@@ -251,6 +260,7 @@ const ListingsOverview = () => {
 
       <ListingDeletionModal
         visible={deleteModalVisible}
+        defaultDeleteType={defaultDeleteType}
         onConfirm={confirmDeletion}
         onCancel={() => {
           setDeleteModalVisible(false);

ui/src/components/map/Map.jsx

@@ -8,6 +8,7 @@ import maplibregl from 'maplibre-gl';
 import 'maplibre-gl/dist/maplibre-gl.css';
 import '@mapbox/mapbox-gl-draw/dist/mapbox-gl-draw.css';
 import { fixMapboxDrawCompatibility, addDrawingControl, setupAreaFilterEventListeners } from './MapDrawingExtension.js';
+import { getBoundsFromCoords } from '../../views/listings/mapUtils.js';
 import './Map.less';
 
 export const GERMANY_BOUNDS = [
@@ -66,6 +67,7 @@ export default function Map({
   const mapContainerRef = useRef(null);
   const mapRef = useRef(null);
   const drawRef = useRef(null);
+  const hasFittedToInitialAreaRef = useRef(false);
 
   // Initialize map - ONLY when container changes, never reinitialize
   useEffect(() => {
@@ -128,6 +130,17 @@ export default function Map({
       } catch (error) {
         console.error('Error loading spatial filter:', error);
       }
+
+      if (!hasFittedToInitialAreaRef.current) {
+        const coords = initialSpatialFilter.features.flatMap((feature) =>
+          feature.geometry?.type === 'Polygon' ? feature.geometry.coordinates.flat() : [],
+        );
+        const bounds = getBoundsFromCoords(coords);
+        if (bounds) {
+          mapRef.current.fitBounds(bounds, { padding: 50, maxZoom: 15, duration: 0 });
+          hasFittedToInitialAreaRef.current = true;
+        }
+      }
     }
 
     // Setup drawing event listeners

ui/src/components/table/JobsTable.jsx

@@ -0,0 +1,128 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { Button, Tag, Tooltip, Switch } from '@douyinfe/semi-ui-19';
+import {
+  IconAlertTriangle,
+  IconBell,
+  IconBriefcase,
+  IconCopy,
+  IconDelete,
+  IconDescend2,
+  IconEdit,
+  IconHome,
+  IconPlayCircle,
+} from '@douyinfe/semi-icons';
+
+import './JobsTable.less';
+
+/**
+ * @param {{ jobs: object[], onRun: Function, onEdit: Function, onClone: Function, onDeleteListings: Function, onDeleteJob: Function, onStatusChange: Function }} props
+ */
+const JobsTable = ({ jobs, onRun, onEdit, onClone, onDeleteListings, onDeleteJob, onStatusChange }) => (
+  <div className="jobsTable">
+    {jobs.map((job) => (
+      <div key={job.id} className={`jobsTable__row${!job.enabled ? ' jobsTable__row--inactive' : ''}`}>
+        <div className="jobsTable__row__dot">
+          <span
+            className={`jobsTable__row__dot__indicator${job.enabled ? ' jobsTable__row__dot__indicator--active' : ''}`}
+          />
+        </div>
+
+        <div className="jobsTable__row__name" title={job.name}>
+          {job.name}
+        </div>
+
+        <div className="jobsTable__row__stat jobsTable__row__stat--blue">
+          <IconHome size="small" />
+          {job.numberOfFoundListings || 0}
+        </div>
+
+        <div className="jobsTable__row__stat jobsTable__row__stat--orange">
+          <IconBriefcase size="small" />
+          {job.provider?.length || 0}
+        </div>
+
+        <div className="jobsTable__row__stat jobsTable__row__stat--purple">
+          <IconBell size="small" />
+          {job.notificationAdapter?.length || 0}
+        </div>
+
+        <div className="jobsTable__row__badges">
+          <Switch
+            size="small"
+            checked={job.enabled}
+            disabled={job.isOnlyShared}
+            onChange={(checked) => onStatusChange(job.id, checked)}
+          />
+          {job.running && (
+            <Tag color="green" variant="light" size="small">
+              RUNNING
+            </Tag>
+          )}
+          {job.isOnlyShared && (
+            <Tooltip content="Shared with you — read only">
+              <span style={{ display: 'flex', alignItems: 'center' }}>
+                <IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
+              </span>
+            </Tooltip>
+          )}
+        </div>
+
+        <div className="jobsTable__row__actions">
+          <Tooltip content="Run Job">
+            <Button
+              type="primary"
+              style={{ background: '#21aa21b5' }}
+              size="small"
+              theme="solid"
+              icon={<IconPlayCircle />}
+              disabled={job.isOnlyShared || job.running}
+              onClick={() => onRun(job.id)}
+            />
+          </Tooltip>
+          <Tooltip content="Edit Job">
+            <Button
+              type="secondary"
+              size="small"
+              icon={<IconEdit />}
+              disabled={job.isOnlyShared}
+              onClick={() => onEdit(job.id)}
+            />
+          </Tooltip>
+          <Tooltip content="Clone Job">
+            <Button
+              type="tertiary"
+              size="small"
+              icon={<IconCopy />}
+              disabled={job.isOnlyShared}
+              onClick={() => onClone(job.id)}
+            />
+          </Tooltip>
+          <Tooltip content="Delete all found Listings">
+            <Button
+              type="danger"
+              size="small"
+              icon={<IconDescend2 />}
+              disabled={job.isOnlyShared}
+              onClick={() => onDeleteListings(job.id)}
+            />
+          </Tooltip>
+          <Tooltip content="Delete Job">
+            <Button
+              type="danger"
+              size="small"
+              icon={<IconDelete />}
+              disabled={job.isOnlyShared}
+              onClick={() => onDeleteJob(job.id)}
+            />
+          </Tooltip>
+        </div>
+      </div>
+    ))}
+  </div>
+);
+
+export default JobsTable;

ui/src/components/table/JobsTable.less

@@ -0,0 +1,105 @@
+@import '../../tokens.less';
+
+.jobsTable {
+  display: flex;
+  flex-direction: column;
+  gap: 4px;
+
+  &__row {
+    display: grid;
+    grid-template-columns: 24px 1fr 80px 80px 80px auto auto;
+    align-items: center;
+    gap: @space-3;
+    padding: 8px 12px;
+    background: @color-elevated;
+    border: 1px solid @color-border;
+    border-radius: @radius-chip;
+    transition: background @transition-fast;
+
+    &:hover {
+      background: #252525;
+    }
+
+    &--inactive {
+      opacity: 0.6;
+    }
+
+    &__dot {
+      display: flex;
+      align-items: center;
+      justify-content: center;
+
+      &__indicator {
+        width: 8px;
+        height: 8px;
+        border-radius: 50%;
+        flex-shrink: 0;
+        background-color: rgba(251, 113, 133, 0.7);
+
+        &--active {
+          background-color: rgba(52, 211, 153, 0.8);
+        }
+      }
+    }
+
+    &__name {
+      font-weight: 600;
+      font-size: @text-sm;
+      color: @color-text;
+      overflow: hidden;
+      text-overflow: ellipsis;
+      white-space: nowrap;
+    }
+
+    &__stat {
+      font-size: @text-sm;
+      font-weight: 600;
+      display: flex;
+      align-items: center;
+      gap: 4px;
+      white-space: nowrap;
+
+      &--blue {
+        color: @color-blue-text;
+      }
+
+      &--orange {
+        color: @color-orange-text;
+      }
+
+      &--purple {
+        color: @color-purple-text;
+      }
+    }
+
+    &__badges {
+      display: flex;
+      align-items: center;
+      gap: 4px;
+      min-width: 0;
+    }
+
+    &__actions {
+      display: flex;
+      align-items: center;
+      gap: 2px;
+    }
+
+    @media (max-width: 900px) {
+      grid-template-columns: 24px 1fr 80px auto auto;
+
+      .jobsTable__row__stat--orange,
+      .jobsTable__row__stat--purple {
+        display: none;
+      }
+    }
+
+    @media (max-width: 560px) {
+      grid-template-columns: 24px 1fr auto auto;
+
+      .jobsTable__row__stat--blue {
+        display: none;
+      }
+    }
+  }
+}

ui/src/services/state/store.js

@@ -335,6 +335,34 @@ export const useFredyState = create(
               throw Exception;
             }
           },
+          async setJobsViewMode(jobs_view_mode) {
+            try {
+              await xhrPost('/api/user/settings/jobs-view-mode', { jobs_view_mode });
+              set((state) => ({
+                userSettings: {
+                  ...state.userSettings,
+                  settings: { ...state.userSettings.settings, jobs_view_mode },
+                },
+              }));
+            } catch (Exception) {
+              console.error('Error while trying to update jobs view mode setting. Error:', Exception);
+              throw Exception;
+            }
+          },
+          async setListingDeletionPreference(listing_deletion_preference) {
+            try {
+              await xhrPost('/api/user/settings/listing-deletion-preference', { listing_deletion_preference });
+              set((state) => ({
+                userSettings: {
+                  ...state.userSettings,
+                  settings: { ...state.userSettings.settings, listing_deletion_preference },
+                },
+              }));
+            } catch (Exception) {
+              console.error('Error while trying to update listing deletion preference. Error:', Exception);
+              throw Exception;
+            }
+          },
         },
       };
 

ui/src/views/generalSettings/GeneralSettings.jsx

@@ -18,6 +18,9 @@ import {
   AutoComplete,
   Select,
   Banner,
+  Radio,
+  RadioGroup,
+  Typography,
 } from '@douyinfe/semi-ui-19';
 import { InputNumber } from '@douyinfe/semi-ui-19';
 import { xhrPost, xhrGet } from '../../services/xhr';
@@ -33,6 +36,8 @@ import { debounce } from '../../utils';
 import Headline from '../../components/headline/Headline.jsx';
 import './GeneralSettings.less';
 
+const { Text } = Typography;
+
 function formatFromTimestamp(ts) {
   const date = new Date(ts);
   return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
@@ -54,8 +59,10 @@ const GeneralSettings = function GeneralSettings() {
   const [loading, setLoading] = React.useState(true);
 
   const settings = useSelector((state) => state.generalSettings.settings);
+  const currentUser = useSelector((state) => state.user.currentUser);
 
   const [interval, setInterval] = React.useState('');
+  const [proxyUrl, setProxyUrl] = React.useState('');
   const [port, setPort] = React.useState('');
   const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
   const [workingHourTo, setWorkingHourTo] = React.useState(null);
@@ -72,9 +79,12 @@ const GeneralSettings = function GeneralSettings() {
   // User settings state
   const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
   const providerDetails = useSelector((state) => state.userSettings.settings.provider_details);
+  const listingDeletionPreference = useSelector((state) => state.userSettings.settings.listing_deletion_preference);
   const allProviders = useSelector((state) => state.provider);
   const [address, setAddress] = useState(homeAddress?.address || '');
   const [coords, setCoords] = useState(homeAddress?.coords || null);
+  const [listingDeleteHard, setListingDeleteHard] = useState(false);
+  const [listingDeleteSkipPrompt, setListingDeleteSkipPrompt] = useState(false);
   const saving = useIsLoading(actions.userSettings.setHomeAddress);
   const [dataSource, setDataSource] = useState([]);
 
@@ -90,6 +100,7 @@ const GeneralSettings = function GeneralSettings() {
   React.useEffect(() => {
     async function init() {
       setInterval(settings?.interval);
+      setProxyUrl(settings?.proxyUrl ?? '');
       setPort(settings?.port);
       setWorkingHourFrom(settings?.workingHours?.from);
       setWorkingHourTo(settings?.workingHours?.to);
@@ -107,6 +118,11 @@ const GeneralSettings = function GeneralSettings() {
     setCoords(homeAddress?.coords || null);
   }, [homeAddress]);
 
+  useEffect(() => {
+    setListingDeleteHard(listingDeletionPreference?.hardDelete ?? false);
+    setListingDeleteSkipPrompt(listingDeletionPreference?.skipPrompt ?? false);
+  }, [listingDeletionPreference]);
+
   const nullOrEmpty = (val) => val == null || val.length === 0;
 
   const handleStore = async () => {
@@ -132,6 +148,7 @@ const GeneralSettings = function GeneralSettings() {
     try {
       await xhrPost('/api/admin/generalSettings', {
         interval,
+        proxyUrl: proxyUrl?.trim() ?? '',
         port,
         workingHours: {
           from: workingHourFrom,
@@ -214,6 +231,10 @@ const GeneralSettings = function GeneralSettings() {
     try {
       const responseJson = await actions.userSettings.setHomeAddress(address);
       setCoords(responseJson.coords);
+      await actions.userSettings.setListingDeletionPreference({
+        skipPrompt: listingDeleteSkipPrompt,
+        hardDelete: listingDeleteHard,
+      });
       await actions.userSettings.getUserSettings();
       Toast.success('Settings saved. Distance calculations are running in the background.');
     } catch (error) {
@@ -375,6 +396,18 @@ const GeneralSettings = function GeneralSettings() {
                   </div>
                 </SegmentPart>
 
+                <SegmentPart
+                  name="Proxy URL"
+                  helpText="Optional. Routes the scraping browser through a proxy. Server/datacenter IPs are frequently blocked by providers (e.g. immowelt) regardless of browser fingerprint, a German residential proxy makes requests look like a normal household and is the most effective fix. Format: http://user:pass@host:port or socks5://user:pass@host:port. Leave empty to disable."
+                >
+                  <Input
+                    type="text"
+                    placeholder="http://user:pass@host:port"
+                    value={proxyUrl}
+                    onChange={(value) => setProxyUrl(value)}
+                  />
+                </SegmentPart>
+
                 <div className="generalSettings__save-row">
                   <Button type="primary" theme="solid" onClick={handleStore} icon={<IconSave />}>
                     Save
@@ -443,6 +476,48 @@ const GeneralSettings = function GeneralSettings() {
                   />
                 </SegmentPart>
 
+                <SegmentPart
+                  name="Listing deletion"
+                  helpText="Choose the default deletion mode. Soft delete hides them without re-scraping; hard delete removes them from the database."
+                >
+                  <RadioGroup
+                    value={listingDeleteHard ? 'hard' : 'soft'}
+                    onChange={(e) => setListingDeleteHard(e.target.value === 'hard')}
+                  >
+                    <Radio value="soft">
+                      <div>
+                        <Text strong>Mark as deleted (Soft Delete)</Text>
+                        <br />
+                        <Text type="secondary">
+                          Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during
+                          the next scraping session.
+                        </Text>
+                      </div>
+                    </Radio>
+                    <Radio value="hard">
+                      <div>
+                        <Text strong>Remove from database (Hard Delete)</Text>
+                        <br />
+                        <Text type="secondary">
+                          Listings are completely removed from the database.
+                          <br />
+                          <Text type="warning">
+                            Consequence: They might re-appear when scraping the next time because Fredy won't know they
+                            were previously found.
+                          </Text>
+                        </Text>
+                      </div>
+                    </Radio>
+                  </RadioGroup>
+                  <Checkbox
+                    checked={listingDeleteSkipPrompt}
+                    onChange={(e) => setListingDeleteSkipPrompt(e.target.checked)}
+                    style={{ marginTop: 12 }}
+                  >
+                    Skip confirmation dialog
+                  </Checkbox>
+                </SegmentPart>
+
                 <div className="generalSettings__save-row">
                   <Button
                     icon={<IconSave />}
@@ -467,12 +542,26 @@ const GeneralSettings = function GeneralSettings() {
               itemKey="backup"
             >
               <div className="generalSettings__tab-content">
+                {demoMode && !currentUser?.isAdmin && (
+                  <Banner
+                    fullMode={false}
+                    type="warning"
+                    closeIcon={null}
+                    style={{ marginBottom: '12px' }}
+                    description="Backup and restore are not available in demo mode."
+                  />
+                )}
                 <SegmentPart
                   name="Backup & Restore"
                   helpText="Download a zipped backup of your database or restore from a backup zip."
                 >
                   <div style={{ display: 'flex', gap: '0.5rem', alignItems: 'center', flexWrap: 'wrap' }}>
-                    <Button theme="solid" icon={<IconSave />} onClick={handleDownloadBackup}>
+                    <Button
+                      theme="solid"
+                      icon={<IconSave />}
+                      onClick={handleDownloadBackup}
+                      disabled={demoMode && !currentUser?.isAdmin}
+                    >
                       Download Backup
                     </Button>
                     <input
@@ -482,7 +571,12 @@ const GeneralSettings = function GeneralSettings() {
                       style={{ display: 'none' }}
                       onChange={handleSelectRestoreFile}
                     />
-                    <Button onClick={handleOpenFilePicker} theme="light" icon={<IconFolder />}>
+                    <Button
+                      onClick={handleOpenFilePicker}
+                      theme="light"
+                      icon={<IconFolder />}
+                      disabled={demoMode && !currentUser?.isAdmin}
+                    >
                       Restore from Zip
                     </Button>
                   </div>

ui/src/views/jobs/mutation/components/notificationAdapter/NotificationAdapterMutator.jsx

@@ -73,7 +73,9 @@ export default function NotificationAdapterMutator({
   const adapter = useSelector((state) => state.notificationAdapter);
 
   const preFilledSelectedAdapter =
-    editNotificationAdapter == null ? null : adapter.find((a) => a.id === editNotificationAdapter.id);
+    editNotificationAdapter == null
+      ? null
+      : adapter.filter((a) => a != null).find((a) => a.id === editNotificationAdapter.id);
 
   spreadPrefilledAdapterWithValues(preFilledSelectedAdapter, editNotificationAdapter?.fields);
 
@@ -227,9 +229,9 @@ export default function NotificationAdapterMutator({
         className="providerMutator__fields"
         value={selectedAdapter == null ? '' : selectedAdapter.id}
         optionList={adapter
+          .filter((a) => a != null)
           .map((a) => {
             return {
-              otherKey: a.id,
               value: a.id,
               label: a.name,
             };
@@ -238,7 +240,7 @@ export default function NotificationAdapterMutator({
           .filter((option) =>
             editNotificationAdapter != null
               ? true
-              : selected.find((selectedOption) => selectedOption.id === option.key) == null,
+              : selected.find((selectedOption) => selectedOption.id === option.value) == null,
           )
           .sort(sortAdapter)}
         onChange={(value) => {

ui/src/views/jobs/mutation/components/provider/ProviderMutator.jsx

@@ -141,21 +141,6 @@ export default function ProviderMutator({
           </p>
         </>
       )}
-      <Banner
-        fullMode={false}
-        type="warning"
-        closeIcon={null}
-        title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Warning</div>}
-        style={{ marginBottom: '1rem' }}
-        description={
-          <div>
-            <p>
-              Currently, our Immoscout implementation does not support drawing shapes on a map. Use a radius instead.
-            </p>
-          </div>
-        }
-      />
-
       <Select
         filter
         placeholder="Select a provider"

ui/src/views/listings/ListingDetail.jsx

@@ -57,7 +57,10 @@ export default function ListingDetail() {
   const navigate = useNavigate();
   const actions = useActions();
   const listing = useSelector((state) => state.listingsData.currentListing);
-  const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
+  const userSettings = useSelector((state) => state.userSettings.settings);
+  const homeAddress = userSettings?.home_address;
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
   const mapContainer = useRef(null);
   const map = useRef(null);
   const [loading, setLoading] = useState(true);
@@ -242,8 +245,11 @@ export default function ListingDetail() {
     };
   }, [listing, loading, homeAddress]);
 
-  const confirmDeletion = async (hardDelete) => {
+  const confirmDeletion = async (hardDelete, remember) => {
     try {
+      if (remember) {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
       await xhrDelete('/api/listings/', { ids: [listing.id], hardDelete });
       Toast.success('Listing successfully removed');
       navigate('/listings');
@@ -347,7 +353,13 @@ export default function ListingDetail() {
             </a>
             <Button
               icon={<IconDelete />}
-              onClick={() => setDeleteModalVisible(true)}
+              onClick={() => {
+                if (listingDeletionPref?.skipPrompt) {
+                  confirmDeletion(listingDeletionPref.hardDelete);
+                  return;
+                }
+                setDeleteModalVisible(true);
+              }}
               theme="light"
               type="danger"
             >
@@ -423,6 +435,7 @@ export default function ListingDetail() {
 
       <ListingDeletionModal
         visible={deleteModalVisible}
+        defaultDeleteType={defaultDeleteType}
         onConfirm={confirmDeletion}
         onCancel={() => setDeleteModalVisible(false)}
       />

ui/src/views/listings/Map.jsx

@@ -37,7 +37,10 @@ export default function MapView() {
   const sp = useSearchParams();
   const [searchParams, setSearchParams] = sp;
   const listings = useSelector((state) => state.listingsData.mapListings);
-  const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
+  const userSettings = useSelector((state) => state.userSettings.settings);
+  const homeAddress = userSettings?.home_address;
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
 
   const jobs = useSelector((state) => state.jobsData.jobs);
   const [jobId, setJobId] = useSearchParamState(sp, 'job', null, parseString);
@@ -52,10 +55,14 @@ export default function MapView() {
 
   const [deleteModalVisible, setDeleteModalVisible] = useState(false);
   const [listingToDelete, setListingToDelete] = useState(null);
+  const deleteListingRef = useRef(null);
 
-  const confirmListingDeletion = async (hardDelete) => {
+  const confirmListingDeletion = async (hardDelete, remember, id = listingToDelete) => {
     try {
-      await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
+      if (remember) {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
+      await xhrDelete('/api/listings/', { ids: [id], hardDelete });
       Toast.success('Listing successfully removed');
       fetchListings();
     } catch (error) {
@@ -66,6 +73,15 @@ export default function MapView() {
     }
   };
 
+  deleteListingRef.current = (id) => {
+    if (listingDeletionPref?.skipPrompt) {
+      confirmListingDeletion(listingDeletionPref.hardDelete, false, id);
+      return;
+    }
+    setListingToDelete(id);
+    setDeleteModalVisible(true);
+  };
+
   useEffect(() => {
     // Only reset to full range when no URL override is set
     if (urlPriceMax === null) {
@@ -88,10 +104,7 @@ export default function MapView() {
   };
 
   useEffect(() => {
-    window.deleteListing = (id) => {
+    window.deleteListing = (id) => deleteListingRef.current(id);
-      setListingToDelete(id);
-      setDeleteModalVisible(true);
-    };
 
     window.viewDetails = (id) => {
       navigate(`/listings/listing/${id}`);
@@ -472,6 +485,7 @@ export default function MapView() {
 
         <ListingDeletionModal
           visible={deleteModalVisible}
+          defaultDeleteType={defaultDeleteType}
           onConfirm={confirmListingDeletion}
           onCancel={() => {
             setDeleteModalVisible(false);

ui/src/views/user/Users.jsx

@@ -37,7 +37,7 @@ const Users = function Users() {
       await actions.jobsData.getJobs();
       await actions.user.getUsers();
     } catch (error) {
-      Toast.error(error);
+      Toast.error(error.error);
       setUserIdToBeRemoved(null);
     }
   };

vitest.config.js

@@ -10,6 +10,7 @@ export default defineConfig({
     globals: true,
     environment: 'node',
     include: ['test/**/*.test.js'],
+    globalSetup: ['./test/globalSetup.js'],
     testTimeout: 60000,
     reporters: ['verbose'],
   },