fixing #319 & #318

* feat: remember listing delete preference

Persist soft/hard choice and skip-confirm in user settings.

README.md

@@ -23,7 +23,7 @@
 
 
 
-# Fredy 🏡 – Your Self-Hosted Real Estate Finder for Germany
+# Fredy 🏡 - Your Self-Hosted Real Estate Finder for Germany
 
 Finding an apartment or house in Germany can be stressful and
 time-consuming.\
@@ -167,6 +167,40 @@ For more information on how to set it up and use it, please refer to the [MCP Re
 
 Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
 
+## 🛡️ Bot Detection & Proxies
+
+Most browser-based providers (immowelt, immonet, kleinanzeigen, ...) are scraped through a hardened headless browser ([CloakBrowser](https://www.npmjs.com/package/cloakbrowser)). It makes the **browser fingerprint** indistinguishable from a real Chrome, which is enough when you run Fredy on a normal home connection.
+
+On a **server / VPS the requests usually originate from a datacenter IP**, and providers behind anti-bot systems (e.g. AWS CloudFront/WAF) block those based on **IP reputation alone**, no matter how perfect the fingerprint is. The typical symptom: it works locally but you get `We have been detected as a bot :-/` on the server.
+
+### The fix: a residential proxy
+
+A **residential proxy** routes Fredy's browser through the internet connection of a real household, so the provider sees a "normal user" IP instead of a datacenter. For German portals, use a **German (DE) residential** (or mobile/4G) proxy. Plain VPNs and **datacenter proxies do not help** here, they share the same bad reputation as your server.
+
+**Configure it** under **Settings → Execution → Proxy URL**. Supported formats:
+
+```
+http://user:pass@host:port
+socks5://user:pass@host:port
+```
+
+Leave the field empty to disable. The proxy applies to all headless-browser providers and takes effect on the next job run (no restart needed). Immoscout uses a separate mobile API and is not affected.
+
+### Where to get a residential proxy
+
+Residential proxies are a paid service (usually billed per GB, Fredy's traffic is small). Well-known providers offering German residential IPs include:
+
+| Provider | Notes |
+|---|---|
+| [IPRoyal](https://iproyal.com) | Pay-as-you-go, no monthly minimum, good for low volume |
+| [Webshare](https://www.webshare.io) | Cheap entry tier, has a small free plan to test with |
+| [Decodo (formerly Smartproxy)](https://decodo.com) | Easy setup, country/city targeting |
+| [SOAX](https://soax.com) | Residential + mobile, fine-grained geo-targeting |
+| [Bright Data](https://brightdata.com) | Largest pool, most features, higher complexity/price |
+| [Oxylabs](https://oxylabs.io) | Enterprise-grade, larger plans |
+
+This is not an endorsement, pick whatever fits your budget. For low-volume use like Fredy, a pay-as-you-go plan (e.g. IPRoyal) or a cheap entry tier (e.g. Webshare) is usually plenty. Make sure to select **Germany** as the proxy location and keep the search interval reasonable (the higher the interval, the less you look like a bot).
+
 ## Analytics
 
 Fredy is completely free (and will always remain free). However, it would be a huge help if you’d allow me to collect some analytical data.

lib/FredyPipelineExecutioner.js

@@ -5,9 +5,10 @@
 
 import { NoNewListingsWarning } from './errors.js';
 import {
-  storeListings,
-  getKnownListingHashesForJobAndProvider,
   deleteListingsById,
+  getKnownListingHashesForJobAndProvider,
+  storeListings,
+  updateListingDistance,
 } from './services/storage/listingsStorage.js';
 import { getJob } from './services/storage/jobStorage.js';
 import * as notify from './notification/notify.js';
@@ -16,8 +17,7 @@ import urlModifier from './services/queryStringMutator.js';
 import logger from './services/logger.js';
 import { geocodeAddress } from './services/geocoding/geoCodingService.js';
 import { distanceMeters } from './services/listings/distanceCalculator.js';
-import { getUserSettings, getSettings } from './services/storage/settingsStorage.js';
-import { updateListingDistance } from './services/storage/listingsStorage.js';
+import { getSettings, getUserSettings } from './services/storage/settingsStorage.js';
 import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
 import { formatListing } from './utils/formatListing.js';
 
@@ -97,9 +97,9 @@ class FredyPipelineExecutioner {
   }
 
   /**
-   * Optionally enrich new listings with data from their detail pages.
+   * Optionally, enrich new listings with data from their detail pages.
    * Only called when the provider config defines a `fetchDetails` function.
-   * Runs all fetches in parallel. Each individual fetch must handle its own errors
+   * Runs all fetches in parallel. Each fetch must handle its own errors
    * and always resolve (never reject) to avoid aborting other listings.
    *
    * @param {Listing[]} newListings New listings to enrich.
@@ -132,7 +132,7 @@ class FredyPipelineExecutioner {
     for (const listing of newListings) {
       if (listing.address) {
         const coords = await geocodeAddress(listing.address);
-        if (coords) {
+        if (coords && coords.lat !== -1 && coords.lng !== -1) {
           listing.latitude = coords.lat;
           listing.longitude = coords.lng;
         }
@@ -227,7 +227,7 @@ class FredyPipelineExecutioner {
     const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
     return new Promise((resolve, reject) => {
       extractor
-        .execute(url, this._providerConfig.waitForSelector)
+        .execute(url, this._providerConfig.waitForSelector, this._providerId)
         .then(() => {
           const listings = extractor.parseResponseText(
             this._providerConfig.crawlContainer,
@@ -264,15 +264,15 @@ class FredyPipelineExecutioner {
     const requiredKeys = this._providerConfig.requiredFieldNames;
     const requireValues = ['id', 'link', 'title'];
 
-    const filteredListings = listings
-      // this should never filter some listings out, because the normalize function should always extract all fields.
-      .filter((item) => requiredKeys.every((key) => key in item))
-      // TODO: move blacklist filter to this file, so it will handle for all providers in same way.
-      .filter(this._providerConfig.filter)
-      // filter out listings that are missing required fields
-      .filter((item) => requireValues.every((key) => item[key] != null));
-
-    return filteredListings;
+    return (
+      listings
+        // this should never filter some listings out, because the normalize function should always extract all fields.
+        .filter((item) => requiredKeys.every((key) => key in item))
+        // TODO: move blacklist filter to this file, so it will handle for all providers in same way.
+        .filter(this._providerConfig.filter)
+        // filter out listings that are missing required fields
+        .filter((item) => requireValues.every((key) => item[key] != null))
+    );
   }
 
   /**

lib/TRACKING_POIS.js

@@ -7,4 +7,9 @@ export const TRACKING_POIS = {
   DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
   WELCOME_FINISHED: 'WELCOME_FINISHED',
   WELCOME_SKIPPED: 'WELCOME_SKIPPED',
+  JOBS_TABLE_VIEW: 'JOBS_TABLE_VIEW',
+  LISTING_TABLE_VIEW: 'LISTING_TABLE_VIEW',
+  BASE_URL_SETTING: 'BASE_URL_SETTING',
+  SET_PROXY_SETTING: 'SET_PROXY_SETTING',
+  DETECTED_AS_BOT: 'DETECTED_AS_BOT',
 };

lib/api/api.js

@@ -76,13 +76,13 @@ fastify.register(async (app) => {
   app.register(dashboardPlugin, { prefix: '/api/dashboard' });
   app.register(userSettingsPlugin, { prefix: '/api/user/settings' });
   app.register(trackingPlugin, { prefix: '/api/tracking' });
+  app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
 });
 
 // Admin-only routes
 fastify.register(async (app) => {
   app.addHook('preHandler', authHook);
   app.addHook('preHandler', adminHook);
-  app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
   app.register(backupPlugin, { prefix: '/api/admin/backup' });
   app.register(userPlugin, { prefix: '/api/admin/users' });
 });

lib/api/routes/backupRouter.js

@@ -9,6 +9,10 @@ import {
   precheckRestore,
   restoreFromZip,
 } from '../../services/storage/backupRestoreService.js';
+import { getSettings } from '../../services/storage/settingsStorage.js';
+import { isAdmin } from '../security.js';
+
+const DEMO_MODE_ERROR = 'Backup and restore are not available in demo mode.';
 
 /**
  * @param {import('fastify').FastifyInstance} fastify
@@ -21,7 +25,11 @@ export default async function backupPlugin(fastify) {
     (req, body, done) => done(null, body),
   );
 
-  fastify.get('/', async (_request, reply) => {
+  fastify.get('/', async (request, reply) => {
+    const settings = await getSettings();
+    if (settings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: DEMO_MODE_ERROR });
+    }
     const zipBuffer = await createBackupZip();
     const fileName = await buildBackupFileName();
     reply.header('Content-Type', 'application/zip');
@@ -30,6 +38,10 @@ export default async function backupPlugin(fastify) {
   });
 
   fastify.post('/restore', async (request, reply) => {
+    const settings = await getSettings();
+    if (settings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: DEMO_MODE_ERROR });
+    }
     const { dryRun = 'false', force = 'false' } = request.query || {};
     const doDryRun = String(dryRun) === 'true';
     const doForce = String(force) === 'true';

lib/api/routes/generalSettingsRoute.js

@@ -9,6 +9,8 @@ import { ensureDemoUserExists } from '../../services/storage/userStorage.js';
 import logger from '../../services/logger.js';
 import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
 import { isAdmin } from '../security.js';
+import { trackPoi } from '../../services/tracking/Tracker.js';
+import { TRACKING_POIS } from '../../TRACKING_POIS.js';
 
 /**
  * @param {import('fastify').FastifyInstance} fastify
@@ -25,16 +27,26 @@ export default async function generalSettingsPlugin(fastify) {
     }
     const localSettings = await getSettings();
 
-    if (localSettings.demoMode && !isAdmin(request)) {
-      return reply.code(403).send({ error: 'In demo mode, it is not allowed to change these settings.' });
+    if (!isAdmin(request)) {
+      const reason = localSettings.demoMode
+        ? 'In demo mode, it is not allowed to change these settings.'
+        : 'Only admins can change these settings.';
+      return reply.code(403).send({ error: reason });
     }
 
     try {
       if (typeof sqlitepath !== 'undefined') {
         fs.writeFileSync(`${getDirName()}/../conf/config.json`, JSON.stringify({ sqlitepath }));
       }
+
       upsertSettings(appSettings);
       ensureDemoUserExists();
+      if (appSettings.baseUrl != null) {
+        await trackPoi(TRACKING_POIS.BASE_URL_SETTING);
+      }
+      if (appSettings.proxyUrl != null) {
+        await trackPoi(TRACKING_POIS.SET_PROXY_SETTING);
+      }
     } catch (err) {
       logger.error(err);
       return reply.code(500).send({ error: 'Error while trying to write settings.' });

lib/api/routes/notificationAdapterRouter.js

@@ -18,7 +18,7 @@ const notificationAdapter = await Promise.all(
  */
 export default async function notificationAdapterPlugin(fastify) {
   fastify.get('/', async () => {
-    return notificationAdapter.map((adapter) => adapter.config);
+    return notificationAdapter.map((adapter) => adapter.config).filter(Boolean);
   });
 
   fastify.post('/try', async (request, reply) => {

lib/api/routes/userSettingsRoute.js

@@ -118,6 +118,10 @@ export default async function userSettingsPlugin(fastify) {
       return reply.code(400).send({ error: 'listings_view_mode must be "grid" or "table".' });
     }
 
+    if (listings_view_mode === 'table') {
+      await trackPoi(TRACKING_POIS.LISTING_TABLE_VIEW);
+    }
+
     try {
       upsertSettings({ listings_view_mode }, userId);
       return { success: true };
@@ -135,6 +139,10 @@ export default async function userSettingsPlugin(fastify) {
       return reply.code(400).send({ error: 'jobs_view_mode must be "grid" or "table".' });
     }
 
+    if (jobs_view_mode === 'table') {
+      await trackPoi(TRACKING_POIS.JOBS_TABLE_VIEW);
+    }
+
     try {
       upsertSettings({ jobs_view_mode }, userId);
       return { success: true };
@@ -143,4 +151,28 @@ export default async function userSettingsPlugin(fastify) {
       return reply.code(500).send({ error: error.message });
     }
   });
+
+  fastify.post('/listing-deletion-preference', async (request, reply) => {
+    const userId = request.session.currentUser;
+    const { listing_deletion_preference } = request.body;
+
+    const globalSettings = await getSettings();
+    if (globalSettings.demoMode && !isAdmin(request)) {
+      return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
+    }
+
+    if (listing_deletion_preference == null) {
+      return reply.code(400).send({ error: 'listing_deletion_preference is required.' });
+    }
+
+    const { skipPrompt, hardDelete } = listing_deletion_preference;
+
+    try {
+      upsertSettings({ listing_deletion_preference: { skipPrompt, hardDelete } }, userId);
+      return { success: true };
+    } catch (error) {
+      logger.error('Error updating listing deletion preference', error);
+      return reply.code(500).send({ error: error.message });
+    }
+  });
 }

lib/notification/adapter/telegram.js

@@ -9,6 +9,7 @@ import fetch from 'node-fetch';
 import pThrottle from 'p-throttle';
 import { normalizeImageUrl } from '../../utils.js';
 import logger from '../../services/logger.js';
+import { shouldUseMultipart, buildPhotoFormData } from './telegramPhotoUploader.js';
 
 const RATE_LIMIT_INTERVAL = 1000;
 const chatThrottleMap = new Map();
@@ -177,11 +178,13 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
   const jobName = job == null ? jobKey : job.name;
 
   const throttledCall = getThrottled(chatId, async function (endpoint, body) {
-    const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, {
-      method: 'post',
-      body: JSON.stringify(body),
-      headers: { 'Content-Type': 'application/json' },
-    });
+    // FormData (multipart) vs JSON. node-fetch sets its own multipart boundary
+    // header, so we must NOT supply Content-Type ourselves in that case.
+    const isFormData = body instanceof FormData;
+    const opts = isFormData
+      ? { method: 'post', body }
+      : { method: 'post', body: JSON.stringify(body), headers: { 'Content-Type': 'application/json' } };
+    const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, opts);
 
     if (!res.ok) {
       const errorBody = await res.text();
@@ -208,16 +211,28 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
       });
     }
 
-    return await throttledCall('sendPhoto', {
-      chat_id: chatId,
-      photo: img,
-      caption: plainText
-        ? buildCaptionPlain(jobName, serviceName, o, baseUrl)
-        : buildCaption(jobName, serviceName, o, baseUrl),
-      ...(plainText ? {} : { parse_mode: 'HTML' }),
-      ...(message_thread_id ? { message_thread_id } : {}),
-    }).catch(async (e) => {
-      logger.error(`Error sending photo to Telegram and use a fallback: ${e.message}`);
+    const caption = plainText
+      ? buildCaptionPlain(jobName, serviceName, o, baseUrl)
+      : buildCaption(jobName, serviceName, o, baseUrl);
+    const parseMode = plainText ? undefined : 'HTML';
+
+    // .webp URLs (Immowelt/Cloudimage) fail Telegram's URL-based sendPhoto with
+    // "failed to get HTTP URL content". Upload the bytes via multipart instead;
+    // the rendered chat message is identical.
+    const photoCall = shouldUseMultipart(img)
+      ? buildPhotoFormData({ chatId, imageUrl: img, caption, parseMode, messageThreadId: message_thread_id }).then(
+          (fd) => throttledCall('sendPhoto', fd),
+        )
+      : throttledCall('sendPhoto', {
+          chat_id: chatId,
+          photo: img,
+          caption,
+          ...(parseMode ? { parse_mode: parseMode } : {}),
+          ...(message_thread_id ? { message_thread_id } : {}),
+        });
+
+    return await photoCall.catch(async (e) => {
+      logger.warn(`Error sending photo to Telegram and use a fallback: ${e.message}`);
       return await throttledCall('sendMessage', textPayload).catch((e) => {
         logger.error(`Error sending message to Telegram: ${e.message}`);
         throw e;

lib/notification/adapter/telegramPhotoUploader.js

@@ -0,0 +1,106 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+/**
+ * Helpers for sending photos to Telegram via `multipart/form-data` instead of
+ * the HTTP-URL path. Used when the URL is one that Telegram's URL-fetcher will
+ * reject - notably `.webp` images from Cloudimage (mms.immowelt.de), which
+ * Telegram refuses with "Bad Request: failed to get HTTP URL content".
+ *
+ * The HTTP-URL path is faster and is still the default in telegram.js; this
+ * module is the fallback for URLs whose extension makes Telegram fail.
+ */
+
+/** Telegram's sendPhoto limit when uploading bytes via multipart/form-data. */
+const TELEGRAM_MULTIPART_MAX_BYTES = 10 * 1024 * 1024;
+
+/** Accept header used when re-fetching the image ourselves.
+ *  Deliberately excludes `image/webp` so CDNs that content-negotiate
+ *  (like Cloudimage on mms.immowelt.de) transcode WEBP to JPEG. */
+const NON_WEBP_ACCEPT = 'image/jpeg,image/png,image/*;q=0.8';
+
+/**
+ * Returns true if the URL's path ends in a `.webp` extension. Such URLs need
+ * multipart upload because Telegram identifies media types from the URL path
+ * and rejects `.webp` in sendPhoto via HTTP URL.
+ *
+ * Conservative: returns false for null/empty/non-string input, malformed URLs,
+ * and non-https schemes.
+ *
+ * @param {string|null|undefined} url
+ * @returns {boolean}
+ */
+export function shouldUseMultipart(url) {
+  if (typeof url !== 'string' || url.length === 0) return false;
+  let parsed;
+  try {
+    parsed = new URL(url);
+  } catch {
+    return false;
+  }
+  if (parsed.protocol !== 'https:') return false;
+  return /\.webp$/i.test(parsed.pathname);
+}
+
+/**
+ * Fetch an image from `imageUrl` and build a `FormData` body suitable for
+ * POSTing to `https://api.telegram.org/bot<token>/sendPhoto`.
+ *
+ * - Sends an `Accept` header that excludes `image/webp` so origin/CDN servers
+ *   that content-negotiate return JPEG bytes.
+ * - Rejects images larger than Telegram's 10 MB multipart limit, both
+ *   advertised via `Content-Length` and (defensively) after download.
+ * - The `photo` field is named with a `.jpg` extension because Telegram
+ *   identifies file type from the filename.
+ *
+ * Throws if the image fetch fails, the size limit is exceeded, or the URL is
+ * unreachable. The caller is responsible for catching and falling back.
+ *
+ * @param {Object} args
+ * @param {string|number} args.chatId
+ * @param {string} args.imageUrl
+ * @param {string} args.caption
+ * @param {string} [args.parseMode]      - Telegram parse_mode, e.g. 'HTML'.
+ * @param {number} [args.messageThreadId] - Telegram supergroup topic id.
+ * @returns {Promise<FormData>}
+ */
+export async function buildPhotoFormData({ chatId, imageUrl, caption, parseMode, messageThreadId }) {
+  const res = await fetch(imageUrl, {
+    method: 'GET',
+    headers: { Accept: NON_WEBP_ACCEPT },
+  });
+
+  if (!res.ok) {
+    throw new Error(`Failed to fetch image for multipart upload (${res.status}): ${imageUrl}`);
+  }
+
+  const advertised = Number(res.headers.get('content-length'));
+  if (Number.isFinite(advertised) && advertised > TELEGRAM_MULTIPART_MAX_BYTES) {
+    throw new Error(
+      `Image exceeds Telegram multipart size limit (advertised ${advertised} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
+    );
+  }
+
+  const buf = await res.arrayBuffer();
+  if (buf.byteLength > TELEGRAM_MULTIPART_MAX_BYTES) {
+    throw new Error(
+      `Image exceeds Telegram multipart size limit (downloaded ${buf.byteLength} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
+    );
+  }
+
+  // Telegram identifies the media type from the filename extension. We always
+  // upload as .jpg because the Accept header forces JPEG bytes from CDNs that
+  // honor it; for the rare CDN that ignores Accept and still returns WEBP, the
+  // .jpg filename is a small lie but Telegram's image pipeline accepts it.
+  const blob = new Blob([buf], { type: 'image/jpeg' });
+
+  const fd = new FormData();
+  fd.append('chat_id', String(chatId));
+  fd.append('caption', caption);
+  if (parseMode) fd.append('parse_mode', parseMode);
+  if (messageThreadId != null) fd.append('message_thread_id', String(messageThreadId));
+  fd.append('photo', blob, 'photo.jpg');
+  return fd;
+}

lib/provider/immobilienDe.js

@@ -26,7 +26,7 @@ function parseId(shortenedLink) {
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immobilienDe_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);

lib/provider/immowelt.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immowelt_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);

lib/provider/kleinanzeigen.js

@@ -128,7 +128,7 @@ async function enrichListingFromDetails(listing, browser) {
   if (!absoluteLink) return listing;
 
   try {
-    const html = await puppeteerExtractor(absoluteLink, null, { browser });
+    const html = await puppeteerExtractor(absoluteLink, null, { browser, name: 'kleinanzeigen_details' });
     if (!html) return { ...listing, link: absoluteLink };
 
     const { detailAddress, detailDescription } = extractDetailFromHtml(html);
@@ -196,8 +196,8 @@ const config = {
     id: '.aditem@data-adid',
     price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
     tags: '.aditem-main--middle--tags | removeNewline | trim',
-    title: '.aditem-main .text-module-begin a | removeNewline | trim',
-    link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
+    title: '.aditem-main .text-module-begin | removeNewline | trim',
+    link: '.aditem@data-href',
     description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
     address: '.aditem-main--top--left | trim | removeNewline',
     image: 'img@src',

lib/provider/sparkasse.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, 'body', { browser });
+    const html = await puppeteerExtractor(listing.link, 'body', { browser, name: 'sparkasse_details' });
 
     const $ = cheerio.load(html);
     const nextDataRaw = $('#__NEXT_DATA__').text;

lib/provider/wgGesucht.js

@@ -16,7 +16,7 @@ let appliedBlackList = [];
 
 async function fetchDetails(listing, browser) {
   try {
-    const html = await puppeteerExtractor(listing.link, null, { browser });
+    const html = await puppeteerExtractor(listing.link, null, { browser, name: 'wgGesucht_details' });
     if (!html) return listing;
 
     const $ = cheerio.load(html);

lib/services/extractor/extractor.js

@@ -29,11 +29,12 @@ export default class Extractor {
    * your response will never contain what you are really looking for
    * @param url
    * @param waitForSelector
+   * @param jobKey
    */
-  execute = async (url, waitForSelector = null) => {
+  execute = async (url, waitForSelector = null, jobKey = null) => {
     this.responseText = null;
     try {
-      this.responseText = await puppeteerExtractor(url, waitForSelector, this.options);
+      this.responseText = await puppeteerExtractor(url, waitForSelector, { ...this.options, name: jobKey });
       if (this.responseText != null) {
         loadParser(this.responseText);
       }

lib/services/extractor/puppeteerExtractor.js

@@ -4,9 +4,11 @@
  */
 
 import { launch } from 'cloakbrowser/puppeteer';
-import { debug, botDetected } from './utils.js';
+import { botDetected, debug } from './utils.js';
 import { getPreLaunchConfig } from './botPrevention.js';
 import logger from '../logger.js';
+import { trackPoi } from '../tracking/Tracker.js';
+import { TRACKING_POIS } from '../../TRACKING_POIS.js';
 
 /**
  * Launch a CloakBrowser/Puppeteer browser instance with stealth and humanizer enabled.
@@ -48,7 +50,7 @@ export async function launchBrowser(url, options) {
     preCfg.windowSizeArg,
   ];
 
-  const browser = await launch({
+  return await launch({
     headless: options?.puppeteerHeadless ?? true,
     humanize: true,
     args,
@@ -57,8 +59,6 @@ export async function launchBrowser(url, options) {
     ...(options?.proxyUrl ? { proxy: options.proxyUrl } : {}),
     ...(preCfg.timezone ? { timezone: preCfg.timezone } : {}),
   });
-
-  return browser;
 }
 
 /**
@@ -145,6 +145,13 @@ export default async function execute(url, waitForSelector, options) {
 
     if (botDetected(pageSource, statusCode)) {
       logger.warn('We have been detected as a bot :-/ Tried url: => ', url);
+
+      if (options != null && options.name != null) {
+        await trackPoi(TRACKING_POIS.DETECTED_AS_BOT + '_' + options.name);
+      } else {
+        await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
+      }
+
       result = null;
     } else {
       result = pageSource || (await page.content());

lib/services/immoscout/immoscout-web-translator.js

@@ -141,6 +141,43 @@ const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
   'barrierefreie-wohnung-mieten': { equipment: ['handicappedaccessible'] },
 };
 
+// SEO-optimized rental paths used by the ImmoScout web UI when the user
+// configures a maximum warmrent. Example: "wohnung-bis-800-euro-warm" means
+// "apartment for rent up to 800 EUR warmrent". The web UI generates these
+// paths instead of explicit `price` / `pricetype` query parameters.
+// Note: only the warmrent variant uses an SEO slug; max coldrent searches
+// use the regular "wohnung-mieten" path with explicit `price` and
+// `pricetype=rentpermonth` query params, which the existing translator
+// already handles.
+const SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE = {
+  wohnung: 'apartmentrent',
+  haus: 'houserent',
+};
+const SEO_MAX_WARMRENT_PATH_PATTERN = /^(?<type>wohnung|haus)-bis-(?<price>\d+)-euro-warm$/;
+
+/**
+ * Parses SEO-optimized ImmoScout web paths that encode a maximum warmrent, such
+ * as "wohnung-bis-800-euro-warm". Returns the corresponding mobile API real
+ * estate type and the implicit price/pricetype parameters, or null if the path
+ * does not match the known SEO max-warmrent pattern.
+ *
+ * @param {string} realTypeKey The last segment of the URL path.
+ * @returns {{ realType: string, additionalParams: Record<string, string> } | null}
+ */
+function parseSeoMaxWarmrentPath(realTypeKey) {
+  const match = realTypeKey.match(SEO_MAX_WARMRENT_PATH_PATTERN);
+  if (!match) return null;
+
+  const { type, price } = match.groups;
+  return {
+    realType: SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE[type],
+    additionalParams: {
+      price: `-${price}`,
+      pricetype: 'calculatedtotalrent',
+    },
+  };
+}
+
 export function convertWebToMobile(webUrl) {
   let url;
   try {
@@ -164,7 +201,14 @@ export function convertWebToMobile(webUrl) {
       additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey];
       realType = REAL_ESTATE_TYPE['wohnung-mieten'];
     } else {
-      throw new Error(`Real estate type not found: ${realTypeKey}`);
+      // Test for SEO max-warmrent path, e.g. "wohnung-bis-800-euro-warm"
+      const seoMaxWarmrent = parseSeoMaxWarmrentPath(realTypeKey);
+      if (seoMaxWarmrent) {
+        realType = seoMaxWarmrent.realType;
+        additionalParamsFromWebPath = seoMaxWarmrent.additionalParams;
+      } else {
+        throw new Error(`Real estate type not found: ${realTypeKey}`);
+      }
     }
   }
 

lib/services/jobs/jobExecutionService.js

@@ -14,6 +14,7 @@ import * as similarityCache from '../similarity-check/similarityCache.js';
 import { isRunning, markFinished, markRunning } from './run-state.js';
 import { sendToUsers } from '../sse/sse-broker.js';
 import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
+import { getSettings } from '../storage/settingsStorage.js';
 
 /**
  * Initializes the job execution service.
@@ -160,6 +161,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
     }
     let browser;
     try {
+      // Read the proxy live (not from the startup snapshot) so changing it in the
+      // UI takes effect on the next run without a backend restart. An empty value
+      // disables the proxy. Routing the headless browser through a (German
+      // residential) proxy avoids datacenter-IP based bot detection on the
+      // Puppeteer-based providers (immowelt, immonet, kleinanzeigen, ...).
+      const liveSettings = await getSettings();
+      const proxyUrl = typeof liveSettings?.proxyUrl === 'string' ? liveSettings.proxyUrl.trim() : '';
+
       const jobProviders = job.provider.filter(
         (p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
       );
@@ -168,14 +177,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
           const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
           matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
 
-          if (browser && !browser.isConnected()) {
+          if (browser && !browser.connected) {
             logger.debug('Browser is disconnected, nullifying to launch a new one.');
             await puppeteerExtractor.closeBrowser(browser);
             browser = null;
           }
 
           if (!browser && matchedProvider.config.getListings == null) {
-            browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
+            browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, proxyUrl ? { proxyUrl } : {});
           }
 
           await new FredyPipelineExecutioner(matchedProvider.config, job, prov.id, similarityCache, browser).execute();

lib/services/storage/listingsStorage.js

@@ -214,6 +214,8 @@ export const storeListings = (jobId, providerId, listings) => {
         longitude: item.longitude || null,
       };
       stmt.run(params);
+      // Propagate the DB primary key back so downstream pipeline steps use the correct id
+      item.id = params.id;
     }
   });
 
@@ -417,9 +419,10 @@ export const deleteListingsByJobId = (jobId, hardDelete = false) => {
 };
 
 /**
- * Delete listings by a list of listing IDs.
+ * Delete listings by a list of listing IDs (the nanoid primary key stored in the `id` column).
+ * Used by API routes that receive row IDs from the client.
  *
- * @param {string[]} ids - Array of listing IDs to delete.
+ * @param {string[]} ids - Array of DB row IDs to delete.
  * @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
  * @returns {any} The result from SqliteConnection.execute.
  */

package.json

@@ -1,6 +1,6 @@
 {
   "name": "fredy",
-  "version": "22.0.0",
+  "version": "22.2.2",
   "description": "[F]ind [R]eal [E]states [d]amn eas[y].",
   "scripts": {
     "prepare": "husky",
@@ -62,9 +62,9 @@
     "Firefox ESR"
   ],
   "dependencies": {
-    "@douyinfe/semi-icons": "^2.96.1",
-    "@douyinfe/semi-ui": "2.96.1",
-    "@douyinfe/semi-ui-19": "^2.96.1",
+    "@douyinfe/semi-icons": "^2.99.3",
+    "@douyinfe/semi-ui": "2.99.3",
+    "@douyinfe/semi-ui-19": "^2.99.3",
     "@fastify/cookie": "^11.0.2",
     "@fastify/helmet": "^13.0.2",
     "@fastify/session": "^11.1.1",
@@ -73,12 +73,12 @@
     "@modelcontextprotocol/sdk": "^1.29.0",
     "@sendgrid/mail": "8.1.6",
     "@turf/boolean-point-in-polygon": "^7.3.5",
-    "@vitejs/plugin-react": "6.0.1",
+    "@vitejs/plugin-react": "6.0.2",
     "adm-zip": "^0.5.17",
-    "better-sqlite3": "^12.9.0",
+    "better-sqlite3": "^12.10.0",
     "chart.js": "^4.5.1",
     "cheerio": "^1.2.0",
-    "cloakbrowser": "^0.3.27",
+    "cloakbrowser": "^0.3.31",
     "fastify": "^5.8.5",
     "handlebars": "4.7.9",
     "maplibre-gl": "^5.24.0",
@@ -86,41 +86,41 @@
     "node-cron": "^4.2.1",
     "node-fetch": "3.3.2",
     "node-mailjet": "6.0.11",
-    "nodemailer": "^8.0.7",
+    "nodemailer": "^8.0.10",
     "p-throttle": "^8.1.0",
     "package-up": "^5.0.0",
-    "puppeteer-core": "^24.43.0",
-    "query-string": "9.3.1",
-    "react": "19.2.6",
+    "puppeteer-core": "^25.1.0",
+    "query-string": "9.4.0",
+    "react": "19.2.7",
     "react-chartjs-2": "^5.3.1",
-    "react-dom": "19.2.6",
+    "react-dom": "19.2.7",
     "react-range-slider-input": "^3.3.5",
-    "react-router": "7.15.0",
-    "react-router-dom": "7.15.0",
-    "resend": "^6.12.3",
-    "semver": "^7.7.4",
+    "react-router": "7.16.0",
+    "react-router-dom": "7.16.0",
+    "resend": "^6.12.4",
+    "semver": "^7.8.1",
     "slack": "11.0.2",
-    "vite": "8.0.11",
+    "vite": "8.0.16",
     "x-var": "^3.0.1",
-    "zustand": "^5.0.13"
+    "zustand": "^5.0.14"
   },
   "devDependencies": {
-    "@babel/core": "7.29.0",
-    "@babel/eslint-parser": "7.28.6",
-    "@babel/preset-env": "7.29.5",
-    "@babel/preset-react": "7.28.5",
+    "@babel/core": "7.29.7",
+    "@babel/eslint-parser": "7.29.7",
+    "@babel/preset-env": "7.29.7",
+    "@babel/preset-react": "7.29.7",
     "@eslint/js": "^10.0.1",
     "chalk": "^5.6.2",
-    "eslint": "10.3.0",
+    "eslint": "10.4.1",
     "eslint-config-prettier": "10.1.8",
     "eslint-plugin-react": "7.37.5",
     "globals": "^17.6.0",
     "history": "5.3.0",
     "husky": "9.1.7",
     "less": "4.6.4",
-    "lint-staged": "16.4.0",
+    "lint-staged": "17.0.7",
     "nodemon": "^3.1.14",
     "prettier": "3.8.3",
-    "vitest": "^4.1.5"
+    "vitest": "^4.1.8"
   }
 }

test/mocks/mockStore.js

@@ -32,4 +32,7 @@ export const deletedIds = [];
 export const deleteListingsById = (ids) => {
   deletedIds.push(...ids);
 };
+export const deleteListingsByHash = (hashes) => {
+  deletedIds.push(...hashes);
+};
 /* eslint-enable no-unused-vars */

test/notification/telegram.test.js

@@ -0,0 +1,360 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+
+// Mock external deps BEFORE importing the module under test.
+vi.mock('node-fetch', () => ({ default: vi.fn() }));
+vi.mock('../../lib/services/storage/jobStorage.js', () => ({
+  getJob: (jobKey) => ({ id: jobKey, name: jobKey }),
+}));
+vi.mock('../../lib/services/markdown.js', () => ({
+  markdown2Html: () => '',
+}));
+
+// Helpers to build mock fetch responses.
+function jsonOk(body = { ok: true }) {
+  return {
+    ok: true,
+    status: 200,
+    text: async () => JSON.stringify(body),
+  };
+}
+
+function jsonErr(status, body) {
+  return {
+    ok: false,
+    status,
+    text: async () => JSON.stringify(body),
+  };
+}
+
+function imageOk(bytes = new Uint8Array([0xff, 0xd8, 0xff])) {
+  return {
+    ok: true,
+    status: 200,
+    headers: {
+      get: (h) => {
+        const k = h.toLowerCase();
+        if (k === 'content-type') return 'image/jpeg';
+        if (k === 'content-length') return String(bytes.byteLength);
+        return null;
+      },
+    },
+    arrayBuffer: async () => bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength),
+  };
+}
+
+// Globals are mocked too so buildPhotoFormData (which uses global fetch) can be
+// intercepted by the same single mock.
+let mockNodeFetch;
+let mockGlobalFetch;
+let send;
+
+beforeEach(async () => {
+  // Reset modules to get a fresh import with our mocks applied.
+  vi.resetModules();
+  const nodeFetchMod = await import('node-fetch');
+  mockNodeFetch = nodeFetchMod.default;
+  mockNodeFetch.mockReset();
+
+  mockGlobalFetch = vi.fn();
+  vi.stubGlobal('fetch', mockGlobalFetch);
+
+  ({ send } = await import('../../lib/notification/adapter/telegram.js'));
+});
+
+afterEach(() => {
+  vi.unstubAllGlobals();
+  vi.useRealTimers();
+});
+
+const baseConfig = {
+  id: 'telegram',
+  fields: { token: 'TKN', chatId: '999' },
+};
+
+describe('telegram send() - HTTP URL path (default for .jpg / .png)', () => {
+  it('POSTs JSON to sendPhoto for a .jpg image URL', async () => {
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 'Listing',
+          link: 'https://example.com/a',
+          address: 'Addr',
+          price: '500€',
+          size: '50m²',
+          image: 'https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    const [url, opts] = mockNodeFetch.mock.calls[0];
+    expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
+    expect(opts.method).toBe('post');
+    expect(opts.headers?.['Content-Type']).toBe('application/json');
+    const body = JSON.parse(opts.body);
+    expect(body.chat_id).toBe('999');
+    expect(body.photo).toBe('https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394');
+    expect(body.parse_mode).toBe('HTML');
+  });
+
+  it('does NOT pre-fetch the image when using HTTP URL path', async () => {
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/x.jpg',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    // global fetch (used by buildPhotoFormData) must not be called
+    expect(mockGlobalFetch).not.toHaveBeenCalled();
+  });
+
+  it('falls back to sendMessage when sendPhoto fails', async () => {
+    mockNodeFetch
+      .mockResolvedValueOnce(jsonErr(400, { ok: false, description: 'boom' }))
+      .mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/x.jpg',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(2);
+    expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendPhoto');
+    expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+  });
+});
+
+describe('telegram send() - multipart path (.webp URLs)', () => {
+  it('pre-fetches the image then POSTs FormData to sendPhoto for a .webp URL', async () => {
+    // 1st: GET image via global fetch
+    mockGlobalFetch.mockResolvedValueOnce(imageOk());
+    // 2nd: POST sendPhoto via node-fetch
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 'Listing',
+          link: 'https://example.com/a',
+          address: 'Addr',
+          price: '500€',
+          size: '50m²',
+          image: 'https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    // image was fetched
+    expect(mockGlobalFetch).toHaveBeenCalledTimes(1);
+    expect(mockGlobalFetch.mock.calls[0][0]).toBe('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394');
+
+    // sendPhoto called via node-fetch with FormData
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    const [url, opts] = mockNodeFetch.mock.calls[0];
+    expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
+    expect(opts.method).toBe('post');
+    expect(opts.body).toBeInstanceOf(FormData);
+    // No explicit Content-Type header - fetch sets multipart boundary itself
+    expect(opts.headers).toBeUndefined();
+    expect(opts.body.get('chat_id')).toBe('999');
+    expect(opts.body.get('parse_mode')).toBe('HTML');
+    const photo = opts.body.get('photo');
+    expect(photo).toBeTruthy();
+    expect(photo.size).toBeGreaterThan(0);
+  });
+
+  it('falls back to sendMessage when the image pre-fetch fails for a .webp URL', async () => {
+    // image fetch fails (404 from CDN)
+    mockGlobalFetch.mockResolvedValueOnce({
+      ok: false,
+      status: 404,
+      headers: { get: () => null },
+      arrayBuffer: async () => new ArrayBuffer(0),
+    });
+    // then sendMessage succeeds via node-fetch
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/gone.webp',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+  });
+
+  it('falls back to sendMessage when multipart sendPhoto returns a Telegram error', async () => {
+    mockGlobalFetch.mockResolvedValueOnce(imageOk());
+    mockNodeFetch
+      .mockResolvedValueOnce(jsonErr(400, { description: 'broke' })) // multipart sendPhoto
+      .mockResolvedValueOnce(jsonOk()); // sendMessage fallback
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/x.webp',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(2);
+    expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+  });
+});
+
+describe('telegram send() - mixed batch (regression-safety)', () => {
+  it('handles a batch with both .jpg and .webp - jpg uses URL, webp uses multipart', async () => {
+    // .webp image fetch
+    mockGlobalFetch.mockResolvedValueOnce(imageOk());
+    // both sendPhoto calls succeed
+    mockNodeFetch
+      .mockResolvedValueOnce(jsonOk()) // could be either listing first
+      .mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'jpg-listing',
+          title: 'a',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/a.jpg',
+        },
+        {
+          id: 'webp-listing',
+          title: 'b',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: 'https://example.com/b.webp',
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockGlobalFetch).toHaveBeenCalledTimes(1); // only webp pre-fetches
+    expect(mockNodeFetch).toHaveBeenCalledTimes(2);
+
+    // Verify one call had FormData and one had JSON body
+    const bodies = mockNodeFetch.mock.calls.map((c) => c[1].body);
+    const hasFormData = bodies.some((b) => b instanceof FormData);
+    const hasJson = bodies.some((b) => typeof b === 'string' && b.startsWith('{'));
+    expect(hasFormData).toBe(true);
+    expect(hasJson).toBe(true);
+  });
+
+  it('uses sendMessage (not sendPhoto) when image is null', async () => {
+    mockNodeFetch.mockResolvedValueOnce(jsonOk());
+
+    await send({
+      serviceName: 'immowelt',
+      newListings: [
+        {
+          id: 'a',
+          title: 't',
+          link: 'l',
+          address: 'a',
+          price: '',
+          size: '',
+          image: null,
+        },
+      ],
+      notificationConfig: [baseConfig],
+      jobKey: 'Berlin',
+    });
+
+    expect(mockNodeFetch).toHaveBeenCalledTimes(1);
+    expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
+    expect(mockGlobalFetch).not.toHaveBeenCalled();
+  });
+});
+
+describe('telegram send() - config validation', () => {
+  it('throws when telegram adapter config is missing', () => {
+    expect(() =>
+      send({
+        serviceName: 's',
+        newListings: [],
+        notificationConfig: [],
+        jobKey: 'k',
+      }),
+    ).toThrow(/configuration missing/);
+  });
+
+  it('throws when token or chatId is missing', () => {
+    expect(() =>
+      send({
+        serviceName: 's',
+        newListings: [],
+        notificationConfig: [{ id: 'telegram', fields: { token: '' } }],
+        jobKey: 'k',
+      }),
+    ).toThrow(/token.*chatId/);
+  });
+});

test/notification/telegramPhotoUploader.test.js

@@ -0,0 +1,287 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
+import { shouldUseMultipart, buildPhotoFormData } from '../../lib/notification/adapter/telegramPhotoUploader.js';
+
+describe('shouldUseMultipart', () => {
+  it('returns true for .webp URL with query string', () => {
+    expect(shouldUseMultipart('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394')).toBe(true);
+  });
+
+  it('returns true for .webp URL without query string', () => {
+    expect(shouldUseMultipart('https://example.com/photo.webp')).toBe(true);
+  });
+
+  it('returns true for uppercase .WEBP extension', () => {
+    expect(shouldUseMultipart('https://example.com/IMG.WEBP?x=1')).toBe(true);
+  });
+
+  it('returns false for .jpg URL with query string', () => {
+    expect(shouldUseMultipart('https://mms.immowelt.de/a/b/c/d/xyz.jpg?ci_seal=hash&w=525&h=394')).toBe(false);
+  });
+
+  it('returns false for .jpeg URL', () => {
+    expect(shouldUseMultipart('https://example.com/photo.jpeg')).toBe(false);
+  });
+
+  it('returns false for .png URL with query string', () => {
+    expect(shouldUseMultipart('https://example.com/photo.png?w=100')).toBe(false);
+  });
+
+  it('returns false for .gif URL', () => {
+    expect(shouldUseMultipart('https://example.com/photo.gif')).toBe(false);
+  });
+
+  it('returns false for null', () => {
+    expect(shouldUseMultipart(null)).toBe(false);
+  });
+
+  it('returns false for undefined', () => {
+    expect(shouldUseMultipart(undefined)).toBe(false);
+  });
+
+  it('returns false for empty string', () => {
+    expect(shouldUseMultipart('')).toBe(false);
+  });
+
+  it('returns false for malformed URL', () => {
+    expect(shouldUseMultipart('not a url')).toBe(false);
+  });
+
+  it('returns false for URL where webp is in the query but not the path', () => {
+    expect(shouldUseMultipart('https://example.com/photo.jpg?format=webp')).toBe(false);
+  });
+
+  it('returns false for URL with no extension at all', () => {
+    expect(shouldUseMultipart('https://example.com/photo')).toBe(false);
+  });
+
+  it('returns false for non-https schemes', () => {
+    // file/data/ftp URLs should not be relevant; safer to skip multipart
+    expect(shouldUseMultipart('http://example.com/photo.webp')).toBe(false);
+  });
+});
+
+describe('buildPhotoFormData', () => {
+  let mockFetch;
+
+  beforeEach(() => {
+    mockFetch = vi.fn();
+    vi.stubGlobal('fetch', mockFetch);
+  });
+
+  afterEach(() => {
+    vi.unstubAllGlobals();
+  });
+
+  function makeImageResponse({ contentType = 'image/jpeg', bytes = new Uint8Array([0xff, 0xd8, 0xff]) } = {}) {
+    const buf = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
+    return {
+      ok: true,
+      status: 200,
+      headers: {
+        get: (h) =>
+          h.toLowerCase() === 'content-type'
+            ? contentType
+            : h.toLowerCase() === 'content-length'
+              ? String(bytes.byteLength)
+              : null,
+      },
+      arrayBuffer: async () => buf,
+    };
+  }
+
+  it('fetches image with Accept header that excludes webp so the CDN transcodes to JPEG', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    await buildPhotoFormData({
+      chatId: '123',
+      imageUrl: 'https://example.com/photo.webp',
+      caption: 'hi',
+      parseMode: 'HTML',
+    });
+
+    expect(mockFetch).toHaveBeenCalledTimes(1);
+    const [url, opts] = mockFetch.mock.calls[0];
+    expect(url).toBe('https://example.com/photo.webp');
+    expect(opts?.headers?.Accept || opts?.headers?.accept).toMatch(/image\/jpeg/);
+    expect(opts?.headers?.Accept || opts?.headers?.accept).not.toMatch(/image\/webp/);
+  });
+
+  it('returns FormData containing chat_id, caption, parse_mode, and photo fields', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '12345',
+      imageUrl: 'https://example.com/abc.webp',
+      caption: 'My caption',
+      parseMode: 'HTML',
+    });
+
+    expect(fd).toBeInstanceOf(FormData);
+    expect(fd.get('chat_id')).toBe('12345');
+    expect(fd.get('caption')).toBe('My caption');
+    expect(fd.get('parse_mode')).toBe('HTML');
+    const photo = fd.get('photo');
+    expect(photo).toBeTruthy();
+    // File-like (Blob); has a name and a size
+    expect(typeof photo.name).toBe('string');
+    expect(photo.size).toBeGreaterThan(0);
+  });
+
+  it('uses a .jpg filename (Telegram uses URL/filename extension for type detection)', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    const photo = fd.get('photo');
+    expect(photo.name).toMatch(/\.jpg$/i);
+  });
+
+  it('includes message_thread_id when provided', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+      messageThreadId: 42,
+    });
+
+    expect(fd.get('message_thread_id')).toBe('42');
+  });
+
+  it('omits message_thread_id when not provided', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    expect(fd.get('message_thread_id')).toBeNull();
+  });
+
+  it('omits parse_mode when not provided (plain text mode)', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/source.webp',
+      caption: 'c',
+    });
+
+    expect(fd.get('parse_mode')).toBeNull();
+  });
+
+  it('throws when the image fetch returns non-200', async () => {
+    mockFetch.mockResolvedValueOnce({
+      ok: false,
+      status: 404,
+      headers: { get: () => null },
+      arrayBuffer: async () => new ArrayBuffer(0),
+    });
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/gone.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/404/);
+  });
+
+  it('throws when the image fetch throws (network error)', async () => {
+    mockFetch.mockRejectedValueOnce(new Error('ECONNREFUSED'));
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/x.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/ECONNREFUSED/);
+  });
+
+  it('throws when the image exceeds 10 MB (Telegram multipart limit)', async () => {
+    // 11 MB
+    const big = new Uint8Array(11 * 1024 * 1024);
+    mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes: big }));
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/huge.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/size|large|10/i);
+  });
+
+  it('rejects early when content-length header advertises > 10 MB (avoids download)', async () => {
+    mockFetch.mockResolvedValueOnce({
+      ok: true,
+      status: 200,
+      headers: {
+        get: (h) => {
+          const k = h.toLowerCase();
+          if (k === 'content-type') return 'image/jpeg';
+          if (k === 'content-length') return String(50 * 1024 * 1024);
+          return null;
+        },
+      },
+      arrayBuffer: async () => {
+        throw new Error('should not be called - size check should reject first');
+      },
+    });
+
+    await expect(
+      buildPhotoFormData({
+        chatId: '1',
+        imageUrl: 'https://example.com/huge.webp',
+        caption: 'c',
+        parseMode: 'HTML',
+      }),
+    ).rejects.toThrow(/size|large|10/i);
+  });
+
+  it('accepts exactly 10 MB images (boundary)', async () => {
+    const bytes = new Uint8Array(10 * 1024 * 1024);
+    mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes }));
+
+    const fd = await buildPhotoFormData({
+      chatId: '1',
+      imageUrl: 'https://example.com/exact.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    expect(fd.get('photo').size).toBe(10 * 1024 * 1024);
+  });
+
+  it('coerces non-string chatId (number) to string in form data', async () => {
+    mockFetch.mockResolvedValueOnce(makeImageResponse());
+
+    const fd = await buildPhotoFormData({
+      chatId: 999,
+      imageUrl: 'https://example.com/x.webp',
+      caption: 'c',
+      parseMode: 'HTML',
+    });
+
+    expect(fd.get('chat_id')).toBe('999');
+  });
+});

test/offlineFixtures.js

@@ -38,6 +38,20 @@ async function tryReadFile(filepath) {
   }
 }
 
+function withRealEstateType(data, realEstateType) {
+  if (!realEstateType?.length || !Array.isArray(data?.resultListItems)) {
+    return data;
+  }
+
+  const cloned = typeof structuredClone === 'function' ? structuredClone(data) : JSON.parse(JSON.stringify(data));
+  for (const item of cloned.resultListItems) {
+    if (item?.type === 'EXPOSE_RESULT' && item?.item) {
+      item.item.realEstateType = realEstateType;
+    }
+  }
+  return cloned;
+}
+
 /**
  * Returns fixture HTML for the given URL by mapping hostname → provider name,
  * then distinguishing list vs detail pages by comparing the URL path against
@@ -83,7 +97,10 @@ export function buildFetchMock() {
         const raw = await tryReadFile(path.join(FIXTURES_DIR, 'immoscout_list.json'));
         listData = raw ? JSON.parse(raw) : { resultListItems: [] };
       }
-      return { ok: true, status: 200, json: () => Promise.resolve(listData) };
+
+      const requestedType = new URL(urlStr).searchParams.get('realestatetype');
+      const responseData = withRealEstateType(listData, requestedType);
+      return { ok: true, status: 200, json: () => Promise.resolve(responseData) };
     }
 
     if (urlStr.includes('api.mobile.immobilienscout24.de/expose/')) {

test/services/extractor/puppeteerExtractor.test.js

@@ -0,0 +1,37 @@
+/*
+ * Copyright (c) 2026 by Christian Kellner.
+ * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
+ */
+
+import { vi, describe, it, expect, beforeEach } from 'vitest';
+
+// Mock the CloakBrowser launcher so no real Chromium binary is needed and we can
+// assert which options get forwarded to it.
+const { launchMock } = vi.hoisted(() => ({ launchMock: vi.fn() }));
+
+vi.mock('cloakbrowser/puppeteer', () => ({
+  launch: launchMock,
+}));
+
+const { launchBrowser } = await import('../../../lib/services/extractor/puppeteerExtractor.js');
+
+describe('launchBrowser proxy forwarding', () => {
+  beforeEach(() => {
+    launchMock.mockReset();
+    launchMock.mockResolvedValue({ close: async () => {} });
+  });
+
+  it('forwards proxyUrl to CloakBrowser as the proxy option', async () => {
+    await launchBrowser('https://www.immowelt.de/', { proxyUrl: 'http://user:pass@host:8080' });
+
+    expect(launchMock).toHaveBeenCalledTimes(1);
+    expect(launchMock.mock.calls[0][0]).toMatchObject({ proxy: 'http://user:pass@host:8080' });
+  });
+
+  it('does not set a proxy when no proxyUrl is given', async () => {
+    await launchBrowser('https://www.immowelt.de/', {});
+
+    expect(launchMock).toHaveBeenCalledTimes(1);
+    expect(launchMock.mock.calls[0][0].proxy).toBeUndefined();
+  });
+});

test/services/immoscout/immoscout-web-translator.test.js

@@ -4,11 +4,16 @@
  */
 
 import { convertWebToMobile } from '../../../lib/services/immoscout/immoscout-web-translator.js';
-import { expect } from 'vitest';
+import { expect, vi } from 'vitest';
 import { readFile } from 'fs/promises';
+import { buildFetchMock } from '../../offlineFixtures.js';
 
 export const testData = JSON.parse(await readFile(new URL('./testdata.json', import.meta.url)));
 
+if (process.env.TEST_MODE === 'offline') {
+  vi.stubGlobal('fetch', buildFetchMock());
+}
+
 describe('#immoscout-mobile URL conversion', () => {
   // Test shape URL conversion
   it('should convert a full web URL with shape to mobile URL', () => {
@@ -41,6 +46,60 @@ describe('#immoscout-mobile URL conversion', () => {
     expect(queryParams.get('equipment').split(',')).toEqual(expect.arrayContaining(['garden', 'balcony']));
   });
 
+  // Test URL conversion of SEO web path for max warmrent. The ImmoScout web UI
+  // generates this special SEO slug instead of explicit price/pricetype params
+  // when the user configures a "Warmmiete" filter (real-world URL).
+  it('should convert a SEO apartment max warmrent path to rent + price + pricetype', () => {
+    const webUrl =
+      'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-bis-800-euro-warm?livingspace=-800.0&enteredFrom=result_list';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('apartmentrent');
+    expect(queryParams.get('price')).toBe('-800');
+    expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
+    expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
+    expect(queryParams.get('livingspace')).toBe('-800.0');
+  });
+
+  // Same SEO pattern for houses ("haus-bis-X-euro-warm" → houserent).
+  it('should convert a SEO house max warmrent path to rent + price + pricetype', () => {
+    const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/haus-bis-1500-euro-warm';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('houserent');
+    expect(queryParams.get('price')).toBe('-1500');
+    expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
+  });
+
+  // Sanity check: max coldrent ("Kaltmiete") does NOT use an SEO slug. The web
+  // UI keeps the regular "wohnung-mieten" path and passes explicit
+  // price + pricetype query params, which the existing translator already
+  // handles (real-world URL).
+  it('should convert a max coldrent search via the regular wohnung-mieten path', () => {
+    const webUrl =
+      'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-mieten?price=-800.0&livingspace=-800.0&pricetype=rentpermonth&enteredFrom=result_list';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('apartmentrent');
+    expect(queryParams.get('price')).toBe('-800.0');
+    expect(queryParams.get('pricetype')).toBe('rentpermonth');
+    expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
+  });
+
+  // Explicit query params win over the SEO slug's implicit defaults.
+  it('should let explicit query params override SEO path price defaults', () => {
+    const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-bis-800-euro-warm?price=100-500';
+
+    const converted = convertWebToMobile(webUrl);
+    const queryParams = new URL(converted).searchParams;
+    expect(queryParams.get('realestatetype')).toBe('apartmentrent');
+    expect(queryParams.get('price')).toBe('100-500');
+    expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
+  });
+
   // Test URL conversion with unsupported query parameters
   it('should remove unsupported query parameters', () => {
     const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?minimuminternetspeed=100000';

test/services/jobs/jobExecutionService.test.js

@@ -18,6 +18,7 @@ describe('services/jobs/jobExecutionService', () => {
     const busPath = root + '/lib/services/events/event-bus.js';
     const jobStoragePath = root + '/lib/services/storage/jobStorage.js';
     const userStoragePath = root + '/lib/services/storage/userStorage.js';
+    const settingsStoragePath = root + '/lib/services/storage/settingsStorage.js';
     const brokerPath = root + '/lib/services/sse/sse-broker.js';
     const utilsPath = root + '/lib/utils.js';
     const loggerPath = root + '/lib/services/logger.js';
@@ -33,11 +34,15 @@ describe('services/jobs/jobExecutionService', () => {
       getUsers: () => state.users.slice(),
       getUser: (id) => state.users.find((u) => u.id === id) || null,
     }));
+    vi.doMock(settingsStoragePath, () => ({
+      getSettings: async () => ({}),
+    }));
     vi.doMock(brokerPath, () => ({
       sendToUsers: (...args) => calls.sent.push(args),
     }));
     vi.doMock(utilsPath, () => ({
       duringWorkingHoursOrNotSet: () => false,
+      getPackageVersion: async () => '0.0.0-test',
     }));
     vi.doMock(loggerPath, () => {
       const m = { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} };

tools/devMock.js

@@ -155,6 +155,7 @@ const routes = {
   'GET /api/dashboard': dashboard,
   'GET /api/demo': { demoMode: false },
   'POST /api/user/settings/news-hash': {},
+  'POST /api/user/settings/listing-deletion-preference': {},
 };
 
 const server = http.createServer((req, res) => {

tools/testFixtures/downloadFixtures.js

@@ -95,7 +95,10 @@ async function downloadHtmlProvider(name, providerConfig, launchBrowser, closeBr
 
   const browser = await launchBrowser(providerConfig.url, {});
   try {
-    const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, { browser });
+    const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, {
+      browser,
+      name: 'dowload_fixtures',
+    });
 
     if (!html) {
       console.warn(`  Failed to download ${name}`);

ui/src/components/ListingDeletionModal.jsx

@@ -3,8 +3,8 @@
  * Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
  */
 
-import { useState } from 'react';
-import { Modal, Radio, RadioGroup, Typography } from '@douyinfe/semi-ui-19';
+import { useState, useEffect } from 'react';
+import { Modal, Radio, RadioGroup, Typography, Checkbox } from '@douyinfe/semi-ui-19';
 
 const { Text } = Typography;
 
@@ -15,11 +15,24 @@ const ListingDeletionModal = ({
   title = 'Delete Listings',
   showOptions = true,
   message = 'How would you like to delete the selected listing(s)?',
+  defaultDeleteType = 'soft',
 }) => {
   const [deleteType, setDeleteType] = useState('soft');
+  const [remember, setRemember] = useState(false);
+
+  useEffect(() => {
+    if (visible) {
+      setDeleteType(defaultDeleteType);
+      setRemember(false);
+    }
+  }, [visible, defaultDeleteType]);
 
   const handleOk = () => {
-    onConfirm(!showOptions || deleteType === 'hard');
+    if (showOptions) {
+      onConfirm(deleteType === 'hard', remember);
+    } else {
+      onConfirm(true);
+    }
   };
 
   return (
@@ -36,32 +49,37 @@ const ListingDeletionModal = ({
         <Text>{message}</Text>
       </div>
       {showOptions && (
-        <RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
-          <Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
-            <div style={{ marginLeft: 8 }}>
-              <Text strong>Mark as deleted (Soft Delete)</Text>
-              <br />
-              <Text type="secondary">
-                Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
-                scraping session.
-              </Text>
-            </div>
-          </Radio>
-          <Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
-            <div style={{ marginLeft: 8 }}>
-              <Text strong>Remove from database (Hard Delete)</Text>
-              <br />
-              <Text type="secondary">
-                Listings are completely removed from the database.
+        <>
+          <RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
+            <Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
+              <div style={{ marginLeft: 8 }}>
+                <Text strong>Mark as deleted (Soft Delete)</Text>
                 <br />
-                <Text type="warning">
-                  Consequence: They might re-appear when scraping the next time because Fredy won't know they were
-                  previously found.
+                <Text type="secondary">
+                  Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
+                  scraping session.
                 </Text>
-              </Text>
-            </div>
-          </Radio>
-        </RadioGroup>
+              </div>
+            </Radio>
+            <Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
+              <div style={{ marginLeft: 8 }}>
+                <Text strong>Remove from database (Hard Delete)</Text>
+                <br />
+                <Text type="secondary">
+                  Listings are completely removed from the database.
+                  <br />
+                  <Text type="warning">
+                    Consequence: They might re-appear when scraping the next time because Fredy won't know they were
+                    previously found.
+                  </Text>
+                </Text>
+              </div>
+            </Radio>
+          </RadioGroup>
+          <Checkbox checked={remember} onChange={(e) => setRemember(e.target.checked)} style={{ marginTop: 16 }}>
+            Remember my choice and skip this dialog next time
+          </Checkbox>
+        </>
       )}
     </Modal>
   );

ui/src/components/grid/jobs/JobGrid.jsx

@@ -60,6 +60,8 @@ const JobGrid = () => {
 
   const userSettings = useSelector((state) => state.userSettings.settings);
   const viewMode = userSettings?.jobs_view_mode ?? 'grid';
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
 
   const [page, setPage] = useState(1);
   const pageSize = 12;
@@ -142,13 +144,21 @@ const JobGrid = () => {
   };
 
   const onListingRemoval = (jobId) => {
-    setPendingDeletion({ type: 'listings', jobId });
+    const deletion = { type: 'listings', jobId };
+    if (listingDeletionPref?.skipPrompt) {
+      confirmDeletion(listingDeletionPref.hardDelete, false, deletion);
+      return;
+    }
+    setPendingDeletion(deletion);
     setDeleteModalVisible(true);
   };
 
-  const confirmDeletion = async (hardDelete) => {
-    const { type, jobId } = pendingDeletion;
+  const confirmDeletion = async (hardDelete, remember, deletion = pendingDeletion) => {
+    const { type, jobId } = deletion;
     try {
+      if (remember && type === 'listings') {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
       if (type === 'job') {
         await xhrDelete('/api/jobs', { jobId });
         Toast.success('Job and listings successfully removed');
@@ -174,7 +184,7 @@ const JobGrid = () => {
       Toast.success('Job status successfully changed');
       loadData();
     } catch (error) {
-      Toast.error(error);
+      Toast.error(error.error);
     }
   };
 
@@ -425,6 +435,7 @@ const JobGrid = () => {
         visible={deleteModalVisible}
         title={pendingDeletion?.type === 'job' ? 'Delete Job' : 'Delete Listings'}
         showOptions={pendingDeletion?.type !== 'job'}
+        defaultDeleteType={defaultDeleteType}
         message={
           pendingDeletion?.type === 'job'
             ? 'Are you sure you want to delete this job? All associated listings will be removed from the database.'

ui/src/components/listings/ListingsOverview.jsx

@@ -33,6 +33,8 @@ const ListingsOverview = () => {
   const sp = useSearchParams();
 
   const viewMode = userSettings?.listings_view_mode ?? 'grid';
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
 
   const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
   const pageSize = 40;
@@ -91,15 +93,22 @@ const ListingsOverview = () => {
   };
 
   const handleDelete = (id) => {
+    if (listingDeletionPref?.skipPrompt) {
+      confirmDeletion(listingDeletionPref.hardDelete, false, id);
+      return;
+    }
     setListingToDelete(id);
     setDeleteModalVisible(true);
   };
 
   const handleNavigate = (id) => navigate(`/listings/listing/${id}`);
 
-  const confirmDeletion = async (hardDelete) => {
+  const confirmDeletion = async (hardDelete, remember, id = listingToDelete) => {
     try {
-      await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
+      if (remember) {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
+      await xhrDelete('/api/listings/', { ids: [id], hardDelete });
       Toast.success('Listing successfully removed');
       loadData();
     } catch (error) {
@@ -251,6 +260,7 @@ const ListingsOverview = () => {
 
       <ListingDeletionModal
         visible={deleteModalVisible}
+        defaultDeleteType={defaultDeleteType}
         onConfirm={confirmDeletion}
         onCancel={() => {
           setDeleteModalVisible(false);

ui/src/components/map/Map.jsx

@@ -8,6 +8,7 @@ import maplibregl from 'maplibre-gl';
 import 'maplibre-gl/dist/maplibre-gl.css';
 import '@mapbox/mapbox-gl-draw/dist/mapbox-gl-draw.css';
 import { fixMapboxDrawCompatibility, addDrawingControl, setupAreaFilterEventListeners } from './MapDrawingExtension.js';
+import { getBoundsFromCoords } from '../../views/listings/mapUtils.js';
 import './Map.less';
 
 export const GERMANY_BOUNDS = [
@@ -66,6 +67,7 @@ export default function Map({
   const mapContainerRef = useRef(null);
   const mapRef = useRef(null);
   const drawRef = useRef(null);
+  const hasFittedToInitialAreaRef = useRef(false);
 
   // Initialize map - ONLY when container changes, never reinitialize
   useEffect(() => {
@@ -128,6 +130,17 @@ export default function Map({
       } catch (error) {
         console.error('Error loading spatial filter:', error);
       }
+
+      if (!hasFittedToInitialAreaRef.current) {
+        const coords = initialSpatialFilter.features.flatMap((feature) =>
+          feature.geometry?.type === 'Polygon' ? feature.geometry.coordinates.flat() : [],
+        );
+        const bounds = getBoundsFromCoords(coords);
+        if (bounds) {
+          mapRef.current.fitBounds(bounds, { padding: 50, maxZoom: 15, duration: 0 });
+          hasFittedToInitialAreaRef.current = true;
+        }
+      }
     }
 
     // Setup drawing event listeners

ui/src/services/state/store.js

@@ -349,6 +349,20 @@ export const useFredyState = create(
               throw Exception;
             }
           },
+          async setListingDeletionPreference(listing_deletion_preference) {
+            try {
+              await xhrPost('/api/user/settings/listing-deletion-preference', { listing_deletion_preference });
+              set((state) => ({
+                userSettings: {
+                  ...state.userSettings,
+                  settings: { ...state.userSettings.settings, listing_deletion_preference },
+                },
+              }));
+            } catch (Exception) {
+              console.error('Error while trying to update listing deletion preference. Error:', Exception);
+              throw Exception;
+            }
+          },
         },
       };
 

ui/src/views/generalSettings/GeneralSettings.jsx

@@ -18,6 +18,9 @@ import {
   AutoComplete,
   Select,
   Banner,
+  Radio,
+  RadioGroup,
+  Typography,
 } from '@douyinfe/semi-ui-19';
 import { InputNumber } from '@douyinfe/semi-ui-19';
 import { xhrPost, xhrGet } from '../../services/xhr';
@@ -33,6 +36,8 @@ import { debounce } from '../../utils';
 import Headline from '../../components/headline/Headline.jsx';
 import './GeneralSettings.less';
 
+const { Text } = Typography;
+
 function formatFromTimestamp(ts) {
   const date = new Date(ts);
   return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
@@ -54,8 +59,10 @@ const GeneralSettings = function GeneralSettings() {
   const [loading, setLoading] = React.useState(true);
 
   const settings = useSelector((state) => state.generalSettings.settings);
+  const currentUser = useSelector((state) => state.user.currentUser);
 
   const [interval, setInterval] = React.useState('');
+  const [proxyUrl, setProxyUrl] = React.useState('');
   const [port, setPort] = React.useState('');
   const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
   const [workingHourTo, setWorkingHourTo] = React.useState(null);
@@ -72,9 +79,12 @@ const GeneralSettings = function GeneralSettings() {
   // User settings state
   const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
   const providerDetails = useSelector((state) => state.userSettings.settings.provider_details);
+  const listingDeletionPreference = useSelector((state) => state.userSettings.settings.listing_deletion_preference);
   const allProviders = useSelector((state) => state.provider);
   const [address, setAddress] = useState(homeAddress?.address || '');
   const [coords, setCoords] = useState(homeAddress?.coords || null);
+  const [listingDeleteHard, setListingDeleteHard] = useState(false);
+  const [listingDeleteSkipPrompt, setListingDeleteSkipPrompt] = useState(false);
   const saving = useIsLoading(actions.userSettings.setHomeAddress);
   const [dataSource, setDataSource] = useState([]);
 
@@ -90,6 +100,7 @@ const GeneralSettings = function GeneralSettings() {
   React.useEffect(() => {
     async function init() {
       setInterval(settings?.interval);
+      setProxyUrl(settings?.proxyUrl ?? '');
       setPort(settings?.port);
       setWorkingHourFrom(settings?.workingHours?.from);
       setWorkingHourTo(settings?.workingHours?.to);
@@ -107,6 +118,11 @@ const GeneralSettings = function GeneralSettings() {
     setCoords(homeAddress?.coords || null);
   }, [homeAddress]);
 
+  useEffect(() => {
+    setListingDeleteHard(listingDeletionPreference?.hardDelete ?? false);
+    setListingDeleteSkipPrompt(listingDeletionPreference?.skipPrompt ?? false);
+  }, [listingDeletionPreference]);
+
   const nullOrEmpty = (val) => val == null || val.length === 0;
 
   const handleStore = async () => {
@@ -132,6 +148,7 @@ const GeneralSettings = function GeneralSettings() {
     try {
       await xhrPost('/api/admin/generalSettings', {
         interval,
+        proxyUrl: proxyUrl?.trim() ?? '',
         port,
         workingHours: {
           from: workingHourFrom,
@@ -214,6 +231,10 @@ const GeneralSettings = function GeneralSettings() {
     try {
       const responseJson = await actions.userSettings.setHomeAddress(address);
       setCoords(responseJson.coords);
+      await actions.userSettings.setListingDeletionPreference({
+        skipPrompt: listingDeleteSkipPrompt,
+        hardDelete: listingDeleteHard,
+      });
       await actions.userSettings.getUserSettings();
       Toast.success('Settings saved. Distance calculations are running in the background.');
     } catch (error) {
@@ -375,6 +396,18 @@ const GeneralSettings = function GeneralSettings() {
                   </div>
                 </SegmentPart>
 
+                <SegmentPart
+                  name="Proxy URL"
+                  helpText="Optional. Routes the scraping browser through a proxy. Server/datacenter IPs are frequently blocked by providers (e.g. immowelt) regardless of browser fingerprint, a German residential proxy makes requests look like a normal household and is the most effective fix. Format: http://user:pass@host:port or socks5://user:pass@host:port. Leave empty to disable."
+                >
+                  <Input
+                    type="text"
+                    placeholder="http://user:pass@host:port"
+                    value={proxyUrl}
+                    onChange={(value) => setProxyUrl(value)}
+                  />
+                </SegmentPart>
+
                 <div className="generalSettings__save-row">
                   <Button type="primary" theme="solid" onClick={handleStore} icon={<IconSave />}>
                     Save
@@ -443,6 +476,48 @@ const GeneralSettings = function GeneralSettings() {
                   />
                 </SegmentPart>
 
+                <SegmentPart
+                  name="Listing deletion"
+                  helpText="Choose the default deletion mode. Soft delete hides them without re-scraping; hard delete removes them from the database."
+                >
+                  <RadioGroup
+                    value={listingDeleteHard ? 'hard' : 'soft'}
+                    onChange={(e) => setListingDeleteHard(e.target.value === 'hard')}
+                  >
+                    <Radio value="soft">
+                      <div>
+                        <Text strong>Mark as deleted (Soft Delete)</Text>
+                        <br />
+                        <Text type="secondary">
+                          Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during
+                          the next scraping session.
+                        </Text>
+                      </div>
+                    </Radio>
+                    <Radio value="hard">
+                      <div>
+                        <Text strong>Remove from database (Hard Delete)</Text>
+                        <br />
+                        <Text type="secondary">
+                          Listings are completely removed from the database.
+                          <br />
+                          <Text type="warning">
+                            Consequence: They might re-appear when scraping the next time because Fredy won't know they
+                            were previously found.
+                          </Text>
+                        </Text>
+                      </div>
+                    </Radio>
+                  </RadioGroup>
+                  <Checkbox
+                    checked={listingDeleteSkipPrompt}
+                    onChange={(e) => setListingDeleteSkipPrompt(e.target.checked)}
+                    style={{ marginTop: 12 }}
+                  >
+                    Skip confirmation dialog
+                  </Checkbox>
+                </SegmentPart>
+
                 <div className="generalSettings__save-row">
                   <Button
                     icon={<IconSave />}
@@ -467,12 +542,26 @@ const GeneralSettings = function GeneralSettings() {
               itemKey="backup"
             >
               <div className="generalSettings__tab-content">
+                {demoMode && !currentUser?.isAdmin && (
+                  <Banner
+                    fullMode={false}
+                    type="warning"
+                    closeIcon={null}
+                    style={{ marginBottom: '12px' }}
+                    description="Backup and restore are not available in demo mode."
+                  />
+                )}
                 <SegmentPart
                   name="Backup & Restore"
                   helpText="Download a zipped backup of your database or restore from a backup zip."
                 >
                   <div style={{ display: 'flex', gap: '0.5rem', alignItems: 'center', flexWrap: 'wrap' }}>
-                    <Button theme="solid" icon={<IconSave />} onClick={handleDownloadBackup}>
+                    <Button
+                      theme="solid"
+                      icon={<IconSave />}
+                      onClick={handleDownloadBackup}
+                      disabled={demoMode && !currentUser?.isAdmin}
+                    >
                       Download Backup
                     </Button>
                     <input
@@ -482,7 +571,12 @@ const GeneralSettings = function GeneralSettings() {
                       style={{ display: 'none' }}
                       onChange={handleSelectRestoreFile}
                     />
-                    <Button onClick={handleOpenFilePicker} theme="light" icon={<IconFolder />}>
+                    <Button
+                      onClick={handleOpenFilePicker}
+                      theme="light"
+                      icon={<IconFolder />}
+                      disabled={demoMode && !currentUser?.isAdmin}
+                    >
                       Restore from Zip
                     </Button>
                   </div>

ui/src/views/jobs/mutation/components/notificationAdapter/NotificationAdapterMutator.jsx

@@ -73,7 +73,9 @@ export default function NotificationAdapterMutator({
   const adapter = useSelector((state) => state.notificationAdapter);
 
   const preFilledSelectedAdapter =
-    editNotificationAdapter == null ? null : adapter.find((a) => a.id === editNotificationAdapter.id);
+    editNotificationAdapter == null
+      ? null
+      : adapter.filter((a) => a != null).find((a) => a.id === editNotificationAdapter.id);
 
   spreadPrefilledAdapterWithValues(preFilledSelectedAdapter, editNotificationAdapter?.fields);
 
@@ -227,9 +229,9 @@ export default function NotificationAdapterMutator({
         className="providerMutator__fields"
         value={selectedAdapter == null ? '' : selectedAdapter.id}
         optionList={adapter
+          .filter((a) => a != null)
           .map((a) => {
             return {
-              otherKey: a.id,
               value: a.id,
               label: a.name,
             };
@@ -238,7 +240,7 @@ export default function NotificationAdapterMutator({
           .filter((option) =>
             editNotificationAdapter != null
               ? true
-              : selected.find((selectedOption) => selectedOption.id === option.key) == null,
+              : selected.find((selectedOption) => selectedOption.id === option.value) == null,
           )
           .sort(sortAdapter)}
         onChange={(value) => {

ui/src/views/listings/ListingDetail.jsx

@@ -57,7 +57,10 @@ export default function ListingDetail() {
   const navigate = useNavigate();
   const actions = useActions();
   const listing = useSelector((state) => state.listingsData.currentListing);
-  const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
+  const userSettings = useSelector((state) => state.userSettings.settings);
+  const homeAddress = userSettings?.home_address;
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
   const mapContainer = useRef(null);
   const map = useRef(null);
   const [loading, setLoading] = useState(true);
@@ -242,8 +245,11 @@ export default function ListingDetail() {
     };
   }, [listing, loading, homeAddress]);
 
-  const confirmDeletion = async (hardDelete) => {
+  const confirmDeletion = async (hardDelete, remember) => {
     try {
+      if (remember) {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
       await xhrDelete('/api/listings/', { ids: [listing.id], hardDelete });
       Toast.success('Listing successfully removed');
       navigate('/listings');
@@ -347,7 +353,13 @@ export default function ListingDetail() {
             </a>
             <Button
               icon={<IconDelete />}
-              onClick={() => setDeleteModalVisible(true)}
+              onClick={() => {
+                if (listingDeletionPref?.skipPrompt) {
+                  confirmDeletion(listingDeletionPref.hardDelete);
+                  return;
+                }
+                setDeleteModalVisible(true);
+              }}
               theme="light"
               type="danger"
             >
@@ -423,6 +435,7 @@ export default function ListingDetail() {
 
       <ListingDeletionModal
         visible={deleteModalVisible}
+        defaultDeleteType={defaultDeleteType}
         onConfirm={confirmDeletion}
         onCancel={() => setDeleteModalVisible(false)}
       />

ui/src/views/listings/Map.jsx

@@ -37,7 +37,10 @@ export default function MapView() {
   const sp = useSearchParams();
   const [searchParams, setSearchParams] = sp;
   const listings = useSelector((state) => state.listingsData.mapListings);
-  const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
+  const userSettings = useSelector((state) => state.userSettings.settings);
+  const homeAddress = userSettings?.home_address;
+  const listingDeletionPref = userSettings?.listing_deletion_preference;
+  const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
 
   const jobs = useSelector((state) => state.jobsData.jobs);
   const [jobId, setJobId] = useSearchParamState(sp, 'job', null, parseString);
@@ -52,10 +55,14 @@ export default function MapView() {
 
   const [deleteModalVisible, setDeleteModalVisible] = useState(false);
   const [listingToDelete, setListingToDelete] = useState(null);
+  const deleteListingRef = useRef(null);
 
-  const confirmListingDeletion = async (hardDelete) => {
+  const confirmListingDeletion = async (hardDelete, remember, id = listingToDelete) => {
     try {
-      await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
+      if (remember) {
+        await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
+      }
+      await xhrDelete('/api/listings/', { ids: [id], hardDelete });
       Toast.success('Listing successfully removed');
       fetchListings();
     } catch (error) {
@@ -66,6 +73,15 @@ export default function MapView() {
     }
   };
 
+  deleteListingRef.current = (id) => {
+    if (listingDeletionPref?.skipPrompt) {
+      confirmListingDeletion(listingDeletionPref.hardDelete, false, id);
+      return;
+    }
+    setListingToDelete(id);
+    setDeleteModalVisible(true);
+  };
+
   useEffect(() => {
     // Only reset to full range when no URL override is set
     if (urlPriceMax === null) {
@@ -88,10 +104,7 @@ export default function MapView() {
   };
 
   useEffect(() => {
-    window.deleteListing = (id) => {
-      setListingToDelete(id);
-      setDeleteModalVisible(true);
-    };
+    window.deleteListing = (id) => deleteListingRef.current(id);
 
     window.viewDetails = (id) => {
       navigate(`/listings/listing/${id}`);
@@ -472,6 +485,7 @@ export default function MapView() {
 
         <ListingDeletionModal
           visible={deleteModalVisible}
+          defaultDeleteType={defaultDeleteType}
           onConfirm={confirmListingDeletion}
           onCancel={() => {
             setDeleteModalVisible(false);

ui/src/views/user/Users.jsx

@@ -37,7 +37,7 @@ const Users = function Users() {
       await actions.jobsData.getJobs();
       await actions.user.getUsers();
     } catch (error) {
-      Toast.error(error);
+      Toast.error(error.error);
       setUserIdToBeRemoved(null);
     }
   };