Compare commits

..

37 Commits

Author SHA1 Message Date
orangecoding
019b9ac87b next release version 2026-06-03 17:36:40 +02:00
Ramin
0d23d43e79 fix: multiple small style fixes/improvements (#316)
* fix(ui-nav): use selected ring and remove item margin override

* fix(listings): format prices as 1.234,50 € with tabular numerals

* fix(listings): align watchlist star button styles

* fix(version-banner): set margin-bottom to 16px

* fix(ui-nav): keep selected ring with focus reset

* fix(listings): right-align price values

* fix(listings): remove cart icon from price display

* fix(listings): format detail price with shared formatter

* revert(listings): restore grid tile layout from master

Drop watchlist/actions restyle on grid cards.
2026-06-03 17:34:37 +02:00
orangecoding
324afee483 next release version 2026-06-03 10:20:04 +02:00
orangecoding
e95ebb9624 more housekeeping 2026-06-03 10:19:50 +02:00
orangecoding
c29387c85d housekeeping 2026-06-03 09:59:32 +02:00
orangecoding
322ae199b0 allowing multiple chat id's for telegram 2026-06-03 09:46:56 +02:00
orangecoding
b3300169fa merged branch 2026-06-03 08:13:49 +02:00
orangecoding
9296bcdc86 fixing 'open in fredy' link 2026-06-03 08:12:30 +02:00
Christian Kellner
44edf47393 Improved Listing Management (#317)
* adding ability to tag listings eg if you have applied to it / adding ability to add notes to a listing

* storing the date when a status was set
2026-06-02 21:10:08 +02:00
orangecoding
bbebc2a1a2 storing the date when a status was set 2026-06-02 21:09:35 +02:00
orangecoding
d2978c14db merged master 2026-06-02 20:41:43 +02:00
orangecoding
5ceac25aa6 fixing #319 & #318 2026-06-02 20:11:43 +02:00
orangecoding
34b68e1f52 fixing notification adapter 2026-06-02 19:53:06 +02:00
orangecoding
696ae451d3 fixing tooltip issues 2026-06-02 12:57:12 +02:00
orangecoding
317ef79336 adding ability to tag listings eg if you have applied to it / adding ability to add notes to a listing 2026-06-02 12:48:01 +02:00
Christian Kellner
6428e7ad78 Ghost commit for rebuilding due to ci interruption 2026-06-02 11:02:05 +02:00
orangecoding
2bcec04d55 next release version 2026-06-02 10:56:05 +02:00
orangecoding
ee2112a24d fixing tests harder 2026-06-02 10:55:16 +02:00
orangecoding
5a54448288 fixing tests 2026-06-02 10:49:06 +02:00
Ramin
f1b8709ab7 feat: remember listing delete preference (#314)
* feat: remember listing delete preference

Persist soft/hard choice and skip-confirm in user settings.
2026-06-02 10:23:45 +02:00
orangecoding
b56e13aa16 upgrading dependencies 2026-06-02 09:26:46 +02:00
Christian Kellner
a834abc31c fixing filtering of lists (#311)
* fixing listing filtering by applying the correct id
2026-06-02 09:24:45 +02:00
Ramin
573868eccb feat(ui): zoom map to saved area when editing a job (#313)
Fit the job edit map to existing polygon areas on init.
2026-06-02 08:54:56 +02:00
orangecoding
1a210d7c1c poi for proxies 2026-05-25 11:54:49 +02:00
orangecoding
996b841cfb adding ability to add proxies for cloak 2026-05-24 20:49:27 +02:00
orangecoding
b2e294e38c next release version 2026-05-21 21:46:13 +02:00
orangecoding
8afeaa05d9 fixing cloakbrowser connection issue 2026-05-21 21:45:57 +02:00
orangecoding
ec47137b89 upgrading dependencies 2026-05-21 21:40:35 +02:00
orangecoding
33161de087 upgrading dependencies 2026-05-19 09:15:12 +02:00
Stephan
acab23207e fix: kleinanzige listing might have a different structure when it was with no image created (#308) 2026-05-16 14:16:31 +02:00
orangecoding
2896d531e4 upgrading pois 2026-05-13 08:14:57 +02:00
orangecoding
0cbfa25062 upgrading pois 2026-05-12 19:28:58 +02:00
orangecoding
bcd3042026 fixing error messages not being shown properly in user table 2026-05-12 13:24:13 +02:00
orangecoding
0ce93acaf6 more demo fixes 2026-05-12 13:12:26 +02:00
orangecoding
cabef973a2 forbid backuo/restore in demo mode 2026-05-12 12:42:25 +02:00
orangecoding
3d0fa87d19 upgrading dependencies 2026-05-12 09:23:52 +02:00
orangecoding
8b012ef2f1 upgrading dependencies / new pois 2026-05-11 09:18:32 +02:00
78 changed files with 3627 additions and 1658 deletions

View File

@@ -23,7 +23,7 @@
# Fredy 🏡 Your Self-Hosted Real Estate Finder for Germany
# Fredy 🏡 - Your Self-Hosted Real Estate Finder for Germany
Finding an apartment or house in Germany can be stressful and
time-consuming.\
@@ -167,6 +167,40 @@ For more information on how to set it up and use it, please refer to the [MCP Re
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
## 🛡️ Bot Detection & Proxies
Most browser-based providers (immowelt, immonet, kleinanzeigen, ...) are scraped through a hardened headless browser ([CloakBrowser](https://www.npmjs.com/package/cloakbrowser)). It makes the **browser fingerprint** indistinguishable from a real Chrome, which is enough when you run Fredy on a normal home connection.
On a **server / VPS the requests usually originate from a datacenter IP**, and providers behind anti-bot systems (e.g. AWS CloudFront/WAF) block those based on **IP reputation alone**, no matter how perfect the fingerprint is. The typical symptom: it works locally but you get `We have been detected as a bot :-/` on the server.
### The fix: a residential proxy
A **residential proxy** routes Fredy's browser through the internet connection of a real household, so the provider sees a "normal user" IP instead of a datacenter. For German portals, use a **German (DE) residential** (or mobile/4G) proxy. Plain VPNs and **datacenter proxies do not help** here, they share the same bad reputation as your server.
**Configure it** under **Settings → Execution → Proxy URL**. Supported formats:
```
http://user:pass@host:port
socks5://user:pass@host:port
```
Leave the field empty to disable. The proxy applies to all headless-browser providers and takes effect on the next job run (no restart needed). Immoscout uses a separate mobile API and is not affected.
### Where to get a residential proxy
Residential proxies are a paid service (usually billed per GB, Fredy's traffic is small). Well-known providers offering German residential IPs include:
| Provider | Notes |
|---|---|
| [IPRoyal](https://iproyal.com) | Pay-as-you-go, no monthly minimum, good for low volume |
| [Webshare](https://www.webshare.io) | Cheap entry tier, has a small free plan to test with |
| [Decodo (formerly Smartproxy)](https://decodo.com) | Easy setup, country/city targeting |
| [SOAX](https://soax.com) | Residential + mobile, fine-grained geo-targeting |
| [Bright Data](https://brightdata.com) | Largest pool, most features, higher complexity/price |
| [Oxylabs](https://oxylabs.io) | Enterprise-grade, larger plans |
This is not an endorsement, pick whatever fits your budget. For low-volume use like Fredy, a pay-as-you-go plan (e.g. IPRoyal) or a cheap entry tier (e.g. Webshare) is usually plenty. Make sure to select **Germany** as the proxy location and keep the search interval reasonable (the higher the interval, the less you look like a bot).
## Analytics
Fredy is completely free (and will always remain free). However, it would be a huge help if youd allow me to collect some analytical data.

View File

@@ -11,6 +11,8 @@
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<title>Fredy || Real Estate Finder</title>
<link rel="icon" type="image/png" href="/ui/src/assets/heart.png" />
<link rel="apple-touch-icon" href="/ui/src/assets/heart.png" />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />

View File

@@ -5,9 +5,10 @@
import { NoNewListingsWarning } from './errors.js';
import {
storeListings,
getKnownListingHashesForJobAndProvider,
deleteListingsById,
getKnownListingHashesForJobAndProvider,
storeListings,
updateListingDistance,
} from './services/storage/listingsStorage.js';
import { getJob } from './services/storage/jobStorage.js';
import * as notify from './notification/notify.js';
@@ -16,8 +17,7 @@ import urlModifier from './services/queryStringMutator.js';
import logger from './services/logger.js';
import { geocodeAddress } from './services/geocoding/geoCodingService.js';
import { distanceMeters } from './services/listings/distanceCalculator.js';
import { getUserSettings, getSettings } from './services/storage/settingsStorage.js';
import { updateListingDistance } from './services/storage/listingsStorage.js';
import { getSettings, getUserSettings } from './services/storage/settingsStorage.js';
import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
import { formatListing } from './utils/formatListing.js';
@@ -97,10 +97,10 @@ class FredyPipelineExecutioner {
}
/**
* Optionally enrich new listings with data from their detail pages.
* Optionally, enrich new listings with data from their detail pages.
* Only called when the provider config defines a `fetchDetails` function.
* Runs all fetches in parallel. Each individual fetch must handle its own errors
* and always resolve (never reject) to avoid aborting other listings.
* Fetches are performed sequentially to avoid overloading the provider or
* the shared browser instance.
*
* @param {Listing[]} newListings New listings to enrich.
* @returns {Promise<Listing[]>} Resolves with enriched listings.
@@ -132,7 +132,7 @@ class FredyPipelineExecutioner {
for (const listing of newListings) {
if (listing.address) {
const coords = await geocodeAddress(listing.address);
if (coords) {
if (coords && coords.lat !== -1 && coords.lng !== -1) {
listing.latitude = coords.lat;
listing.longitude = coords.lng;
}
@@ -199,9 +199,9 @@ class FredyPipelineExecutioner {
const toDeleteListingByIds = [];
const keptListings = newListings.filter((listing) => {
const filterOut =
(minRooms && listing.rooms && listing.rooms < minRooms) ||
(minSize && listing.size && listing.size < minSize) ||
(maxPrice && listing.price && listing.price > maxPrice);
(minRooms && listing.rooms != null && listing.rooms < minRooms) ||
(minSize && listing.size != null && listing.size < minSize) ||
(maxPrice && listing.price != null && listing.price > maxPrice);
if (filterOut) {
toDeleteListingByIds.push(listing.id);
@@ -223,24 +223,15 @@ class FredyPipelineExecutioner {
* @param {string} url The provider URL to fetch from.
* @returns {Promise<ParsedListing[]>} Resolves with an array of listings (empty when none found).
*/
_getListings(url) {
async _getListings(url) {
const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
return new Promise((resolve, reject) => {
extractor
.execute(url, this._providerConfig.waitForSelector)
.then(() => {
const listings = extractor.parseResponseText(
this._providerConfig.crawlContainer,
this._providerConfig.crawlFields,
url,
);
resolve(listings == null ? [] : listings);
})
.catch((err) => {
reject(err);
logger.error(err);
});
});
await extractor.execute(url, this._providerConfig.waitForSelector, this._providerId);
const listings = extractor.parseResponseText(
this._providerConfig.crawlContainer,
this._providerConfig.crawlFields,
url,
);
return listings == null ? [] : listings;
}
/**
@@ -264,15 +255,15 @@ class FredyPipelineExecutioner {
const requiredKeys = this._providerConfig.requiredFieldNames;
const requireValues = ['id', 'link', 'title'];
const filteredListings = listings
// this should never filter some listings out, because the normalize function should always extract all fields.
.filter((item) => requiredKeys.every((key) => key in item))
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
.filter(this._providerConfig.filter)
// filter out listings that are missing required fields
.filter((item) => requireValues.every((key) => item[key] != null));
return filteredListings;
return (
listings
// this should never filter some listings out, because the normalize function should always extract all fields.
.filter((item) => requiredKeys.every((key) => key in item))
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
.filter(this._providerConfig.filter)
// filter out listings that are missing required fields
.filter((item) => requireValues.every((key) => item[key] != null))
);
}
/**

View File

@@ -7,4 +7,11 @@ export const TRACKING_POIS = {
DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
WELCOME_FINISHED: 'WELCOME_FINISHED',
WELCOME_SKIPPED: 'WELCOME_SKIPPED',
JOBS_TABLE_VIEW: 'JOBS_TABLE_VIEW',
LISTING_TABLE_VIEW: 'LISTING_TABLE_VIEW',
BASE_URL_SETTING: 'BASE_URL_SETTING',
SET_PROXY_SETTING: 'SET_PROXY_SETTING',
DETECTED_AS_BOT: 'DETECTED_AS_BOT',
NOTES_CREATE: 'NOTES_CREATE',
USING_LISTING_STATUS: 'USING_LISTING_STATUS',
};

View File

@@ -76,13 +76,13 @@ fastify.register(async (app) => {
app.register(dashboardPlugin, { prefix: '/api/dashboard' });
app.register(userSettingsPlugin, { prefix: '/api/user/settings' });
app.register(trackingPlugin, { prefix: '/api/tracking' });
app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
});
// Admin-only routes
fastify.register(async (app) => {
app.addHook('preHandler', authHook);
app.addHook('preHandler', adminHook);
app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
app.register(backupPlugin, { prefix: '/api/admin/backup' });
app.register(userPlugin, { prefix: '/api/admin/users' });
});

View File

@@ -9,6 +9,10 @@ import {
precheckRestore,
restoreFromZip,
} from '../../services/storage/backupRestoreService.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin } from '../security.js';
const DEMO_MODE_ERROR = 'Backup and restore are not available in demo mode.';
/**
* @param {import('fastify').FastifyInstance} fastify
@@ -21,7 +25,11 @@ export default async function backupPlugin(fastify) {
(req, body, done) => done(null, body),
);
fastify.get('/', async (_request, reply) => {
fastify.get('/', async (request, reply) => {
const settings = await getSettings();
if (settings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: DEMO_MODE_ERROR });
}
const zipBuffer = await createBackupZip();
const fileName = await buildBackupFileName();
reply.header('Content-Type', 'application/zip');
@@ -30,6 +38,10 @@ export default async function backupPlugin(fastify) {
});
fastify.post('/restore', async (request, reply) => {
const settings = await getSettings();
if (settings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: DEMO_MODE_ERROR });
}
const { dryRun = 'false', force = 'false' } = request.query || {};
const doDryRun = String(dryRun) === 'true';
const doForce = String(force) === 'true';

View File

@@ -9,6 +9,8 @@ import { ensureDemoUserExists } from '../../services/storage/userStorage.js';
import logger from '../../services/logger.js';
import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin } from '../security.js';
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
/**
* @param {import('fastify').FastifyInstance} fastify
@@ -25,16 +27,26 @@ export default async function generalSettingsPlugin(fastify) {
}
const localSettings = await getSettings();
if (localSettings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change these settings.' });
if (!isAdmin(request)) {
const reason = localSettings.demoMode
? 'In demo mode, it is not allowed to change these settings.'
: 'Only admins can change these settings.';
return reply.code(403).send({ error: reason });
}
try {
if (typeof sqlitepath !== 'undefined') {
fs.writeFileSync(`${getDirName()}/../conf/config.json`, JSON.stringify({ sqlitepath }));
}
upsertSettings(appSettings);
ensureDemoUserExists();
if (appSettings.baseUrl != null) {
await trackPoi(TRACKING_POIS.BASE_URL_SETTING);
}
if (appSettings.proxyUrl != null) {
await trackPoi(TRACKING_POIS.SET_PROXY_SETTING);
}
} catch (err) {
logger.error(err);
return reply.code(500).send({ error: 'Error while trying to write settings.' });

View File

@@ -195,6 +195,9 @@ export default async function jobPlugin(fastify) {
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (!job) {
return reply.code(404).send({ error: 'Job not found' });
}
if (settings.demoMode && !isAdmin(request) && job.name === DEMO_JOB_NAME) {
return reply.code(403).send({ error: 'Sorry, but you cannot remove the Demo Job ;)' });
}
@@ -216,6 +219,9 @@ export default async function jobPlugin(fastify) {
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (!job) {
return reply.code(404).send({ error: 'Job not found' });
}
if (settings.demoMode && !isAdmin(request) && job.name === DEMO_JOB_NAME) {
return reply.code(403).send({ error: 'Sorry, but you cannot change the Status of our Demo Job ;)' });

View File

@@ -8,8 +8,10 @@ import * as watchListStorage from '../../services/storage/watchListStorage.js';
import { isAdmin as isAdminFn } from '../security.js';
import logger from '../../services/logger.js';
import { nullOrEmpty } from '../../utils.js';
import { getJobs } from '../../services/storage/jobStorage.js';
import { getJob } from '../../services/storage/jobStorage.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
/**
* @param {import('fastify').FastifyInstance} fastify
@@ -23,6 +25,7 @@ export default async function listingsPlugin(fastify) {
jobNameFilter,
providerFilter,
watchListFilter,
statusFilter,
sortfield = null,
sortdir = 'asc',
freeTextFilter,
@@ -35,12 +38,16 @@ export default async function listingsPlugin(fastify) {
};
const normalizedActivity = toBool(activityFilter);
const normalizedWatch = toBool(watchListFilter);
const allowedStatuses = ['applied', 'rejected', 'accepted', 'none'];
const normalizedStatus =
typeof statusFilter === 'string' && allowedStatuses.includes(statusFilter.toLowerCase())
? statusFilter.toLowerCase()
: undefined;
let jobFilter = null;
let jobIdFilter = null;
const jobs = getJobs();
if (!nullOrEmpty(jobNameFilter)) {
const job = jobs.find((j) => j.id === jobNameFilter);
const job = getJob(jobNameFilter);
jobFilter = job != null ? job.name : null;
jobIdFilter = job != null ? job.id : null;
}
@@ -54,6 +61,7 @@ export default async function listingsPlugin(fastify) {
jobIdFilter: jobIdFilter,
providerFilter,
watchListFilter: normalizedWatch,
statusFilter: normalizedStatus,
sortField: sortfield || null,
sortDir: sortdir === 'desc' ? 'desc' : 'asc',
userId: request.session.currentUser,
@@ -94,6 +102,55 @@ export default async function listingsPlugin(fastify) {
return reply.send();
});
fastify.post('/:listingId/notes', async (request, reply) => {
const { listingId } = request.params || {};
const { notes } = request.body || {};
const userId = request.session?.currentUser;
if (!listingId || !userId) {
return reply.code(400).send({ message: 'listingId or user not provided' });
}
try {
const changes = listingStorage.setListingNotes(listingId, typeof notes === 'string' ? notes : null);
if (changes === 0) {
return reply.code(404).send({ message: 'Listing not found' });
}
} catch (error) {
logger.error(error);
return reply.code(500).send({ message: 'Failed to update listing notes' });
}
await trackPoi(TRACKING_POIS.NOTES_CREATE);
return reply.send();
});
fastify.post('/:listingId/status', async (request, reply) => {
const { listingId } = request.params || {};
const { status } = request.body || {};
const userId = request.session?.currentUser;
if (!listingId || !userId) {
return reply.code(400).send({ message: 'listingId or user not provided' });
}
const allowed = ['applied', 'rejected', 'accepted'];
const normalized = status == null ? null : String(status).toLowerCase();
if (normalized != null && !allowed.includes(normalized)) {
return reply.code(400).send({ message: `Invalid status: ${status}` });
}
try {
const changes = listingStorage.setListingStatus(listingId, normalized);
await trackPoi(TRACKING_POIS.USING_LISTING_STATUS);
if (changes === 0) {
return reply.code(404).send({ message: 'Listing not found' });
}
if (normalized != null) {
watchListStorage.ensureWatch(listingId, userId);
}
} catch (error) {
logger.error(error);
return reply.code(500).send({ message: 'Failed to update listing status' });
}
return reply.send();
});
fastify.delete('/job', async (request, reply) => {
const { jobId, hardDelete = false } = request.body;
const settings = await getSettings();
@@ -101,6 +158,16 @@ export default async function listingsPlugin(fastify) {
if (settings.demoMode && !isAdminFn(request)) {
return reply.code(403).send({ error: 'Sorry, but you cannot remove listings in demo mode ;)' });
}
const job = getJob(jobId);
if (!job) {
return reply.code(404).send({ error: 'Job not found' });
}
const userId = request.session.currentUser;
if (!isAdminFn(request) && job.userId !== userId && !job.shared_with_user.includes(userId)) {
return reply
.code(403)
.send({ error: 'You are trying to remove listings for a job that is not associated to your user' });
}
listingStorage.deleteListingsByJobId(jobId, hardDelete);
} catch (error) {
logger.error(error);
@@ -111,7 +178,11 @@ export default async function listingsPlugin(fastify) {
fastify.delete('/', async (request, reply) => {
const { ids, hardDelete = false } = request.body;
const settings = await getSettings();
try {
if (settings.demoMode && !isAdminFn(request)) {
return reply.code(403).send({ error: 'Sorry, but you cannot remove listings in demo mode ;)' });
}
if (Array.isArray(ids) && ids.length > 0) {
listingStorage.deleteListingsById(ids, hardDelete);
}

View File

@@ -20,6 +20,9 @@ function getClientIp(request) {
function isRateLimited(ip) {
const now = Date.now();
for (const [key, rec] of loginAttempts) {
if (now - rec.firstAttempt > LOGIN_WINDOW_MS) loginAttempts.delete(key);
}
const record = loginAttempts.get(ip);
if (!record || now - record.firstAttempt > LOGIN_WINDOW_MS) {
loginAttempts.set(ip, { count: 1, firstAttempt: now });

View File

@@ -18,7 +18,7 @@ const notificationAdapter = await Promise.all(
*/
export default async function notificationAdapterPlugin(fastify) {
fastify.get('/', async () => {
return notificationAdapter.map((adapter) => adapter.config);
return notificationAdapter.map((adapter) => adapter.config).filter(Boolean);
});
fastify.post('/try', async (request, reply) => {

View File

@@ -3,13 +3,11 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import SqliteConnection from '../../services/storage/SqliteConnection.js';
import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
import { getSettings, getUserSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin } from '../security.js';
import { resetGeocoordinatesAndDistanceForUser } from '../../services/storage/listingsStorage.js';
import { geocodeAddress } from '../../services/geocoding/geoCodingService.js';
import { autocompleteAddress } from '../../services/geocoding/autocompleteService.js';
import { fromJson } from '../../utils.js';
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
import logger from '../../services/logger.js';
@@ -21,12 +19,7 @@ import { runGeoCordTask } from '../../services/crons/geocoding-cron.js';
export default async function userSettingsPlugin(fastify) {
fastify.get('/', async (request) => {
const userId = request.session.currentUser;
const rows = SqliteConnection.query('SELECT name, value FROM settings WHERE user_id = @userId', { userId });
const settings = {};
for (const r of rows) {
settings[r.name] = fromJson(r.value, null);
}
return settings;
return getUserSettings(userId);
});
fastify.get('/autocomplete', async (request, reply) => {
@@ -118,6 +111,10 @@ export default async function userSettingsPlugin(fastify) {
return reply.code(400).send({ error: 'listings_view_mode must be "grid" or "table".' });
}
if (listings_view_mode === 'table') {
await trackPoi(TRACKING_POIS.LISTING_TABLE_VIEW);
}
try {
upsertSettings({ listings_view_mode }, userId);
return { success: true };
@@ -135,6 +132,10 @@ export default async function userSettingsPlugin(fastify) {
return reply.code(400).send({ error: 'jobs_view_mode must be "grid" or "table".' });
}
if (jobs_view_mode === 'table') {
await trackPoi(TRACKING_POIS.JOBS_TABLE_VIEW);
}
try {
upsertSettings({ jobs_view_mode }, userId);
return { success: true };
@@ -143,4 +144,28 @@ export default async function userSettingsPlugin(fastify) {
return reply.code(500).send({ error: error.message });
}
});
fastify.post('/listing-deletion-preference', async (request, reply) => {
const userId = request.session.currentUser;
const { listing_deletion_preference } = request.body;
const globalSettings = await getSettings();
if (globalSettings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
}
if (listing_deletion_preference == null) {
return reply.code(400).send({ error: 'listing_deletion_preference is required.' });
}
const { skipPrompt, hardDelete } = listing_deletion_preference;
try {
upsertSettings({ listing_deletion_preference: { skipPrompt, hardDelete } }, userId);
return { success: true };
} catch (error) {
logger.error('Error updating listing deletion preference', error);
return reply.code(500).send({ error: error.message });
}
});
}

View File

@@ -155,6 +155,12 @@ export function createMcpServer() {
),
sortField: z.string().optional().describe('Sort by: created_at, price, size, provider, title, is_active'),
sortDir: z.string().optional().describe('Sort direction: asc or desc'),
status: z
.enum(['applied', 'rejected', 'accepted', 'none'])
.optional()
.describe(
'Filter by user-set status. "applied", "rejected", or "accepted" return only listings with that status; "none" returns only listings without a status set.',
),
},
async (
{
@@ -170,6 +176,7 @@ export function createMcpServer() {
maxPrice,
sortField,
sortDir,
status,
},
extra,
) => {
@@ -192,6 +199,7 @@ export function createMcpServer() {
maxPrice: maxPrice ?? null,
sortField: sortField ?? null,
sortDir: sortDir ?? 'desc',
statusFilter: status,
userId: user.id,
isAdmin: user.isAdmin,
});

View File

@@ -124,10 +124,10 @@ export function normalizeListListings(queryResult, { page, pageSize }) {
md += '\n\n';
if (listings.length > 0) {
md += `| ID | Title | Address | Price | Size | Provider | Active | Created | Job |\n`;
md += `|----|-------|---------|-------|------|----------|--------|---------|-----|\n`;
md += `| ID | Title | Address | Price | Size | Provider | Active | Status | Created | Job |\n`;
md += `|----|-------|---------|-------|------|----------|--------|--------|---------|-----|\n`;
for (const l of listings) {
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${cell(l.status?.status)} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
}
md += `\nUse **get_listing** with an ID for full details (description, link, image).\n`;
} else {
@@ -156,6 +156,10 @@ export function normalizeGetListing(listing) {
md += `- **Link:** ${listing.link || ''}\n`;
md += `- **Image:** ${listing.image_url || ''}\n`;
md += `- **Active:** ${listing.is_active ? 'yes' : 'no'}\n`;
md += `- **Status:** ${listing.status?.status || ''}\n`;
if (listing.status?.setAt) {
md += `- **Status set at:** ${formatDate(listing.status.setAt)}\n`;
}
md += `- **Created:** ${formatDate(listing.created_at)}\n`;
md += `- **Job:** ${listing.job_name || ''}\n`;
if (listing.latitude != null && listing.longitude != null) {

View File

@@ -13,7 +13,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
const jobName = job == null ? jobKey : job.name;
const promises = newListings.map((newListing) => {
const title = `${jobName} at ${serviceName}: ${newListing.title}`;
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
const message = `Address: ${newListing.address}\nSize: ${newListing.size}\nPrice: ${newListing.price}\nLink: ${newListing.link}${fredyLine}`;
return fetch(server, {
method: 'POST',

View File

@@ -7,7 +7,7 @@ import { markdown2Html } from '../../services/markdown.js';
export const send = ({ serviceName, newListings, jobKey, baseUrl }) => {
/* eslint-disable no-console */
const fredyLinks = baseUrl ? newListings.map((l) => `${baseUrl}/listings/listing/${l.id}`).join(', ') : null;
const fredyLinks = baseUrl ? newListings.map((l) => `${baseUrl}/#/listings/listing/${l.id}`).join(', ') : null;
return [
Promise.resolve(
console.info(

View File

@@ -7,6 +7,7 @@ import fetch from 'node-fetch';
import { getJob } from '../../services/storage/jobStorage.js';
import { markdown2Html } from '../../services/markdown.js';
import { normalizeImageUrl } from '../../utils.js';
import logger from '../../services/logger.js';
/**
* Generates an idempotent decimal color code. The input string-based color code is
@@ -67,11 +68,19 @@ const buildEmbed = (jobKey, listing, baseUrl) => {
},
];
if (baseUrl && listing.id) {
fields.push({
name: 'Open in Fredy',
value: `[Open in Fredy](${baseUrl}/#/listings/listing/${listing.id})`,
inline: false,
});
}
const embed = {
title: title,
color: generateColorFromString(jobKey),
url: listing.link,
fields: fields,
fields,
};
if (listing.image) {
@@ -80,14 +89,6 @@ const buildEmbed = (jobKey, listing, baseUrl) => {
};
}
if (baseUrl && listing.id) {
fields.push({
name: 'Open in Fredy',
value: `[Open in Fredy](${baseUrl}/listings/listing/${listing.id})`,
inline: false,
});
}
return embed;
};
@@ -119,7 +120,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
headers: { 'Content-Type': 'application/json' },
body,
}).catch((error) => {
console.error(`Error sending Discord webhook for chunk starting at ${i}:`, error);
logger.error(`Error sending Discord webhook for chunk starting at ${i}:`, error);
return Promise.reject(new Error(`Webhook failed: ${error.message}`));
});

View File

@@ -13,7 +13,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
let message = `### *${jobName}* (${serviceName}) found **${newListings.length}** new listings:\n\n`;
message += `| Title | Address | Size | Price |${baseUrl ? ' Open in Fredy |' : ''}\n|:----|:----|:----|:----|${baseUrl ? ':----|\n' : '\n'}`;
message += newListings.map((o) => {
const fredyCell = baseUrl && o.id ? ` [Open in Fredy](${baseUrl}/listings/listing/${o.id}) |` : '';
const fredyCell = baseUrl && o.id ? ` [Open in Fredy](${baseUrl}/#/listings/listing/${o.id}) |` : '';
return (
`| [${o.title}](${o.link}) | ` +
[o.address, o.size.replace(/2m/g, '$m^2$'), o.price].join(' | ') +

View File

@@ -14,7 +14,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
const jobName = job == null ? jobKey : job.name;
const promises = newListings.map((newListing) => {
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
const message = `
Address: ${newListing.address}
Size: ${newListing.size == null ? 'N/A' : newListing.size.replace(/2m/g, '$m^2$')}

View File

@@ -15,7 +15,8 @@ export const send = async ({ serviceName, newListings, notificationConfig, jobKe
const results = await Promise.all(
newListings.map(async (newListing) => {
const title = `${jobName} at ${serviceName}: ${newListing.title}`;
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
const fredyLine =
baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
const message = `Address: ${newListing.address}\nSize: ${newListing.size}\nPrice: ${newListing.price}\nLink: ${newListing.link}${fredyLine}`;
const form = new FormData();

View File

@@ -39,7 +39,7 @@ const buildBlocks = (serviceName, jobKey, p, baseUrl) => {
if (baseUrl && p.id) {
blocks.push({
type: 'section',
text: { type: 'mrkdwn', text: `<${baseUrl}/listings/listing/${p.id}|Open in Fredy>` },
text: { type: 'mrkdwn', text: `<${baseUrl}/#/listings/listing/${p.id}|Open in Fredy>` },
});
}

View File

@@ -39,7 +39,7 @@ const buildBlocks = (serviceName, jobKey, p, baseUrl) => {
if (baseUrl && p.id) {
blocks.push({
type: 'section',
text: { type: 'mrkdwn', text: `<${baseUrl}/listings/listing/${p.id}|Open in Fredy>` },
text: { type: 'mrkdwn', text: `<${baseUrl}/#/listings/listing/${p.id}|Open in Fredy>` },
});
}

View File

@@ -9,43 +9,48 @@ import fetch from 'node-fetch';
import pThrottle from 'p-throttle';
import { normalizeImageUrl } from '../../utils.js';
import logger from '../../services/logger.js';
import { shouldUseMultipart, buildPhotoFormData } from './telegramPhotoUploader.js';
const RATE_LIMIT_INTERVAL = 1000;
const THROTTLE_MAX_IDLE_MS = RATE_LIMIT_INTERVAL + 2000;
const chatThrottleMap = new Map();
/**
* Removes stale throttled call entries to keep memory bounded.
* An entry is stale when no API call has fired for longer than THROTTLE_MAX_IDLE_MS.
*/
function cleanupOldThrottles() {
const now = Date.now();
const maxAge = RATE_LIMIT_INTERVAL + 1000;
const toBeDeleted = [];
for (const [chatId, chatThrottle] of chatThrottleMap.entries()) {
if (now - chatThrottle.lastUsedAt > maxAge) toBeDeleted.push(chatId);
if (now - chatThrottle.lastUsedAt > THROTTLE_MAX_IDLE_MS) chatThrottleMap.delete(chatId);
}
for (const chatId of toBeDeleted) chatThrottleMap.delete(chatId);
}
/**
* Return a throttled wrapper for a chatId to limit Telegram API calls.
* Uses p-throttle with 1 request per RATE_LIMIT_INTERVAL per chat.
* `lastUsedAt` is refreshed on every actual API call so that the idle window
* starts from the last fired call, not from when send() was invoked.
*
* @template {Function} T
* @param {string|number} chatId
* @param {T} call - async function (endpoint: string, body: any) => Promise<Response>
* @returns {T}
* @param {Function} call - async function (endpoint: string, body: any) => Promise<Response>
* @returns {Function}
*/
function getThrottled(chatId, call) {
cleanupOldThrottles();
const now = Date.now();
const chatThrottle = chatThrottleMap.get(chatId);
if (chatThrottle) {
chatThrottle.lastUsedAt = now;
return chatThrottle.throttled;
const existing = chatThrottleMap.get(chatId);
if (existing) {
existing.lastUsedAt = Date.now();
return existing.throttled;
}
const throttled = pThrottle({ limit: 1, interval: RATE_LIMIT_INTERVAL })(call);
chatThrottleMap.set(chatId, { lastUsedAt: now, throttled });
return throttled;
const entry = { lastUsedAt: Date.now(), throttled: null };
chatThrottleMap.set(chatId, entry);
entry.throttled = pThrottle({ limit: 1, interval: RATE_LIMIT_INTERVAL })(async (endpoint, body) => {
const e = chatThrottleMap.get(chatId);
if (e) e.lastUsedAt = Date.now();
return call(endpoint, body);
});
return entry.throttled;
}
/**
@@ -69,39 +74,20 @@ function escapeHtml(s = '') {
}
/**
* Build a Telegram photo caption (max 1024 characters) using HTML parse mode.
* Build a Telegram HTML-formatted message body.
* Suitable for both sendMessage (uncapped) and sendPhoto captions (caller must slice to 1024).
*
* @param {string} jobName
* @param {string} serviceName
* @param {Object} o - Listing object
* @param {string} [o.title]
* @param {string} [o.address]
* @param {string|number} [o.price]
* @param {string|number} [o.size]
* @param {string} [o.link]
* @param {string} [baseUrl]
* @returns {string}
*/
function buildCaption(jobName, serviceName, o, baseUrl) {
function buildHtmlBody(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLink =
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
return `<i>${escapeHtml(jobName)}</i> (${escapeHtml(serviceName)})\n<a href='${escapeHtml(
o.link || '',
)}'><b>${escapeHtml(title)}</b></a>\n${escapeHtml(meta)}${fredyLink}`.slice(0, 1024);
}
/**
* Build a Telegram message text using HTML parse mode.
* @param {string} jobName
* @param {string} serviceName
* @param {Object} o - Listing object
* @returns {string}
*/
function buildText(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLink =
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/#/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
return (
`<i>${escapeHtml(jobName)}</i> (${escapeHtml(serviceName)})\n` +
`<a href='${escapeHtml(o.link || '')}'><b>${escapeHtml(title)}</b></a>\n` +
@@ -110,34 +96,128 @@ function buildText(jobName, serviceName, o, baseUrl) {
}
/**
* Build a plain text Telegram photo caption (max 4096 characters).
* Build a plain-text Telegram photo caption (max 4096 characters).
* Meta appears before the link so the most relevant info is visible within the cap.
*
* @param {string} jobName
* @param {string} serviceName
* @param {Object} o - Listing object
* @param baseUrl
* @param {string} [baseUrl]
* @returns {string}
*/
function buildCaptionPlain(jobName, serviceName, o, baseUrl) {
function buildPlainCaption(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${o.id}` : '';
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${o.id}` : '';
return `${jobName} (${serviceName})\n${title}\n${meta}\n\n${o.link || ''}${fredyLine}`.slice(0, 4096);
}
/**
* Build a plain text Telegram message.
* Build a plain-text Telegram message body.
* Link appears early so it is tappable without scrolling.
*
* @param {string} jobName
* @param {string} serviceName
* @param {Object} o - Listing object
* @param {string} [baseUrl]
* @returns {string}
*/
function buildTextPlain(jobName, serviceName, o, baseUrl) {
function buildPlainText(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${o.id}` : '';
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${o.id}` : '';
return `${jobName} (${serviceName})\n${title}\n${o.link || ''}\n${meta}${fredyLine}`;
}
/**
* Create the raw Telegram API caller for a given bot token.
* Handles JSON and multipart (FormData) bodies.
*
* @param {string} token - Telegram bot token.
* @param {string} jobName - Used in error messages.
* @returns {(endpoint: string, body: object|FormData) => Promise<Response>}
*/
function makeTelegramCaller(token, jobName) {
return async function (endpoint, body) {
const isFormData = body instanceof FormData;
const opts = isFormData
? { method: 'post', body }
: { method: 'post', body: JSON.stringify(body), headers: { 'Content-Type': 'application/json' } };
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, opts);
if (!res.ok) {
const errorBody = await res.text();
throw new Error(`API error for '${jobName}'. '${endpoint}' returned ${errorBody}`);
}
return res;
};
}
/**
* Send a single listing to a single Telegram chat, with photo-then-text fallback.
*
* @param {Function} throttledCall - Throttled Telegram API caller for this chat.
* @param {Object} listing - Listing object.
* @param {string|number} chatId
* @param {Object} opts
* @param {string} opts.jobName
* @param {string} opts.serviceName
* @param {string} opts.baseUrl
* @param {boolean} opts.plainText
* @param {number|undefined} opts.message_thread_id
* @returns {Promise<void>}
*/
async function sendListingToChat(
throttledCall,
listing,
chatId,
{ jobName, serviceName, baseUrl, plainText, message_thread_id },
) {
const img = normalizeImageUrl(listing.image);
const textPayload = {
chat_id: chatId,
text: plainText
? buildPlainText(jobName, serviceName, listing, baseUrl)
: buildHtmlBody(jobName, serviceName, listing, baseUrl),
...(plainText ? {} : { parse_mode: 'HTML' }),
disable_web_page_preview: true,
...(message_thread_id ? { message_thread_id } : {}),
};
if (!img) {
return throttledCall('sendMessage', textPayload).catch((e) => {
logger.error(`Error sending message to Telegram: ${e.message}`);
});
}
const caption = plainText
? buildPlainCaption(jobName, serviceName, listing, baseUrl)
: buildHtmlBody(jobName, serviceName, listing, baseUrl).slice(0, 1024);
const parseMode = plainText ? undefined : 'HTML';
// .webp URLs (Immowelt/Cloudimage) fail Telegram's URL-based sendPhoto with
// "failed to get HTTP URL content". Upload the bytes via multipart instead.
const photoCall = shouldUseMultipart(img)
? buildPhotoFormData({ chatId, imageUrl: img, caption, parseMode, messageThreadId: message_thread_id }).then((fd) =>
throttledCall('sendPhoto', fd),
)
: throttledCall('sendPhoto', {
chat_id: chatId,
photo: img,
caption,
...(parseMode ? { parse_mode: parseMode } : {}),
...(message_thread_id ? { message_thread_id } : {}),
});
return photoCall.catch(async (e) => {
logger.warn(`Error sending photo to Telegram and use a fallback: ${e.message}`);
return throttledCall('sendMessage', textPayload).catch((e) => {
logger.error(`Error sending message to Telegram: ${e.message}`);
throw e;
});
});
}
/**
* Send new listings to Telegram.
* - Respects per-chat Telegram rate limits using a lightweight throttle cache.
@@ -160,6 +240,11 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
throw new Error("Telegram 'token' and 'chatId' must be provided in notification config");
}
const chatIds = String(chatId)
.split(',')
.map((s) => s.trim())
.filter(Boolean);
// Optional Telegram topic/thread support (supergroups)
let message_thread_id;
if (messageThreadId !== undefined && messageThreadId !== null && `${messageThreadId}`.trim() !== '') {
@@ -176,56 +261,16 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
const job = getJob(jobKey);
const jobName = job == null ? jobKey : job.name;
const throttledCall = getThrottled(chatId, async function (endpoint, body) {
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, {
method: 'post',
body: JSON.stringify(body),
headers: { 'Content-Type': 'application/json' },
});
if (!res.ok) {
const errorBody = await res.text();
throw new Error(`API error for '${jobName}'. '${endpoint}' returned ${errorBody}`);
}
return res;
});
if (!Array.isArray(newListings) || newListings.length === 0) return Promise.resolve([]);
const promises = newListings.map(async (o) => {
const img = normalizeImageUrl(o.image);
const textPayload = {
chat_id: chatId,
text: plainText ? buildTextPlain(jobName, serviceName, o, baseUrl) : buildText(jobName, serviceName, o, baseUrl),
...(plainText ? {} : { parse_mode: 'HTML' }),
disable_web_page_preview: true,
...(message_thread_id ? { message_thread_id } : {}),
};
if (!img) {
return await throttledCall('sendMessage', textPayload).catch(async (e) => {
logger.error(`Error sending message to Telegram: ${e.message}`);
});
}
return await throttledCall('sendPhoto', {
chat_id: chatId,
photo: img,
caption: plainText
? buildCaptionPlain(jobName, serviceName, o, baseUrl)
: buildCaption(jobName, serviceName, o, baseUrl),
...(plainText ? {} : { parse_mode: 'HTML' }),
...(message_thread_id ? { message_thread_id } : {}),
}).catch(async (e) => {
logger.error(`Error sending photo to Telegram and use a fallback: ${e.message}`);
return await throttledCall('sendMessage', textPayload).catch((e) => {
logger.error(`Error sending message to Telegram: ${e.message}`);
throw e;
});
});
const allPromises = chatIds.flatMap((id) => {
const caller = makeTelegramCaller(token, jobName);
const throttledCall = getThrottled(id, caller);
const opts = { jobName, serviceName, baseUrl, plainText, message_thread_id };
return newListings.map((listing) => sendListingToChat(throttledCall, listing, id, opts));
});
return Promise.all(promises);
return Promise.all(allPromises);
};
/**
@@ -246,7 +291,8 @@ export const config = {
chatId: {
type: 'chatId',
label: 'Chat Id',
description: 'The chat id to send messages to you.',
description:
'The chat ID to send messages to. Separate multiple IDs with commas to notify several recipients (e.g. 123456789, 987654321).',
},
messageThreadId: {
type: 'text',

View File

@@ -21,6 +21,8 @@ Steps:
- Private chats: `chat.id` is a positive number
- Groups/supergroups: `chat.id` is a negative number
**Multiple recipients:** To notify several users individually, enter a comma-separated list of chat IDs in the Chat Id field, e.g. `123456789, 987654321`. Each recipient receives the same messages and gets its own independent rate-limit window. This avoids having to create a group and add the bot to it.
Keep your bot token secret. If `getUpdates` returns an empty list, send a new message and try again, or make sure your bots privacy settings allow it to see group messages when used in groups.
#### Getting the thread ID (this is optional to be used for forum topics)

View File

@@ -0,0 +1,106 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
/**
* Helpers for sending photos to Telegram via `multipart/form-data` instead of
* the HTTP-URL path. Used when the URL is one that Telegram's URL-fetcher will
* reject - notably `.webp` images from Cloudimage (mms.immowelt.de), which
* Telegram refuses with "Bad Request: failed to get HTTP URL content".
*
* The HTTP-URL path is faster and is still the default in telegram.js; this
* module is the fallback for URLs whose extension makes Telegram fail.
*/
/** Telegram's sendPhoto limit when uploading bytes via multipart/form-data. */
const TELEGRAM_MULTIPART_MAX_BYTES = 10 * 1024 * 1024;
/** Accept header used when re-fetching the image ourselves.
* Deliberately excludes `image/webp` so CDNs that content-negotiate
* (like Cloudimage on mms.immowelt.de) transcode WEBP to JPEG. */
const NON_WEBP_ACCEPT = 'image/jpeg,image/png,image/*;q=0.8';
/**
* Returns true if the URL's path ends in a `.webp` extension. Such URLs need
* multipart upload because Telegram identifies media types from the URL path
* and rejects `.webp` in sendPhoto via HTTP URL.
*
* Conservative: returns false for null/empty/non-string input, malformed URLs,
* and non-https schemes.
*
* @param {string|null|undefined} url
* @returns {boolean}
*/
export function shouldUseMultipart(url) {
if (typeof url !== 'string' || url.length === 0) return false;
let parsed;
try {
parsed = new URL(url);
} catch {
return false;
}
if (parsed.protocol !== 'https:') return false;
return /\.webp$/i.test(parsed.pathname);
}
/**
* Fetch an image from `imageUrl` and build a `FormData` body suitable for
* POSTing to `https://api.telegram.org/bot<token>/sendPhoto`.
*
* - Sends an `Accept` header that excludes `image/webp` so origin/CDN servers
* that content-negotiate return JPEG bytes.
* - Rejects images larger than Telegram's 10 MB multipart limit, both
* advertised via `Content-Length` and (defensively) after download.
* - The `photo` field is named with a `.jpg` extension because Telegram
* identifies file type from the filename.
*
* Throws if the image fetch fails, the size limit is exceeded, or the URL is
* unreachable. The caller is responsible for catching and falling back.
*
* @param {Object} args
* @param {string|number} args.chatId
* @param {string} args.imageUrl
* @param {string} args.caption
* @param {string} [args.parseMode] - Telegram parse_mode, e.g. 'HTML'.
* @param {number} [args.messageThreadId] - Telegram supergroup topic id.
* @returns {Promise<FormData>}
*/
export async function buildPhotoFormData({ chatId, imageUrl, caption, parseMode, messageThreadId }) {
const res = await fetch(imageUrl, {
method: 'GET',
headers: { Accept: NON_WEBP_ACCEPT },
});
if (!res.ok) {
throw new Error(`Failed to fetch image for multipart upload (${res.status}): ${imageUrl}`);
}
const advertised = Number(res.headers.get('content-length'));
if (Number.isFinite(advertised) && advertised > TELEGRAM_MULTIPART_MAX_BYTES) {
throw new Error(
`Image exceeds Telegram multipart size limit (advertised ${advertised} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
);
}
const buf = await res.arrayBuffer();
if (buf.byteLength > TELEGRAM_MULTIPART_MAX_BYTES) {
throw new Error(
`Image exceeds Telegram multipart size limit (downloaded ${buf.byteLength} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
);
}
// Telegram identifies the media type from the filename extension. We always
// upload as .jpg because the Accept header forces JPEG bytes from CDNs that
// honor it; for the rare CDN that ignores Accept and still returns WEBP, the
// .jpg filename is a small lie but Telegram's image pipeline accepts it.
const blob = new Blob([buf], { type: 'image/jpeg' });
const fd = new FormData();
fd.append('chat_id', String(chatId));
fd.append('caption', caption);
if (parseMode) fd.append('parse_mode', parseMode);
if (messageThreadId != null) fd.append('message_thread_id', String(messageThreadId));
fd.append('photo', blob, 'photo.jpg');
return fd;
}

View File

@@ -4,6 +4,7 @@
*/
import fs from 'fs';
import logger from '../services/logger.js';
const path = './adapter';
/** Read every integration existing in ./adapter **/
@@ -23,7 +24,13 @@ const findAdapter = (notificationAdapter) => {
export const send = (serviceName, newListings, notificationConfig, jobKey, baseUrl) => {
//this is not being used in tests, therefore adapter are always set
return notificationConfig
.filter((notificationAdapter) => findAdapter(notificationAdapter) != null)
.map((notificationAdapter) => findAdapter(notificationAdapter))
.map((notificationAdapter) => {
const found = findAdapter(notificationAdapter);
if (!found) {
logger.warn(`Notification adapter '${notificationAdapter.id}' not found for job '${jobKey || ''}'`);
}
return found;
})
.filter(Boolean)
.map((a) => a.send({ serviceName, newListings, notificationConfig, jobKey, baseUrl }));
};

View File

@@ -26,7 +26,7 @@ function parseId(shortenedLink) {
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, null, { browser });
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immobilienDe_details' });
if (!html) return listing;
const $ = cheerio.load(html);

View File

@@ -16,7 +16,7 @@ let appliedBlackList = [];
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, null, { browser });
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immowelt_details' });
if (!html) return listing;
const $ = cheerio.load(html);

View File

@@ -128,7 +128,7 @@ async function enrichListingFromDetails(listing, browser) {
if (!absoluteLink) return listing;
try {
const html = await puppeteerExtractor(absoluteLink, null, { browser });
const html = await puppeteerExtractor(absoluteLink, null, { browser, name: 'kleinanzeigen_details' });
if (!html) return { ...listing, link: absoluteLink };
const { detailAddress, detailDescription } = extractDetailFromHtml(html);
@@ -196,8 +196,8 @@ const config = {
id: '.aditem@data-adid',
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
tags: '.aditem-main--middle--tags | removeNewline | trim',
title: '.aditem-main .text-module-begin a | removeNewline | trim',
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
title: '.aditem-main .text-module-begin | removeNewline | trim',
link: '.aditem@data-href',
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
address: '.aditem-main--top--left | trim | removeNewline',
image: 'img@src',

View File

@@ -16,7 +16,7 @@ let appliedBlackList = [];
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, 'body', { browser });
const html = await puppeteerExtractor(listing.link, 'body', { browser, name: 'sparkasse_details' });
const $ = cheerio.load(html);
const nextDataRaw = $('#__NEXT_DATA__').text;

View File

@@ -16,7 +16,7 @@ let appliedBlackList = [];
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, null, { browser });
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'wgGesucht_details' });
if (!html) return listing;
const $ = cheerio.load(html);

View File

@@ -29,11 +29,12 @@ export default class Extractor {
* your response will never contain what you are really looking for
* @param url
* @param waitForSelector
* @param jobKey
*/
execute = async (url, waitForSelector = null) => {
execute = async (url, waitForSelector = null, jobKey = null) => {
this.responseText = null;
try {
this.responseText = await puppeteerExtractor(url, waitForSelector, this.options);
this.responseText = await puppeteerExtractor(url, waitForSelector, { ...this.options, name: jobKey });
if (this.responseText != null) {
loadParser(this.responseText);
}

View File

@@ -4,9 +4,11 @@
*/
import { launch } from 'cloakbrowser/puppeteer';
import { debug, botDetected } from './utils.js';
import { botDetected, debug } from './utils.js';
import { getPreLaunchConfig } from './botPrevention.js';
import logger from '../logger.js';
import { trackPoi } from '../tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
/**
* Launch a CloakBrowser/Puppeteer browser instance with stealth and humanizer enabled.
@@ -48,7 +50,7 @@ export async function launchBrowser(url, options) {
preCfg.windowSizeArg,
];
const browser = await launch({
return await launch({
headless: options?.puppeteerHeadless ?? true,
humanize: true,
args,
@@ -57,8 +59,6 @@ export async function launchBrowser(url, options) {
...(options?.proxyUrl ? { proxy: options.proxyUrl } : {}),
...(preCfg.timezone ? { timezone: preCfg.timezone } : {}),
});
return browser;
}
/**
@@ -145,6 +145,13 @@ export default async function execute(url, waitForSelector, options) {
if (botDetected(pageSource, statusCode)) {
logger.warn('We have been detected as a bot :-/ Tried url: => ', url);
if (options != null && options.name != null) {
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT + '_' + options.name);
} else {
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
}
result = null;
} else {
result = pageSource || (await page.content());

View File

@@ -141,6 +141,43 @@ const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
'barrierefreie-wohnung-mieten': { equipment: ['handicappedaccessible'] },
};
// SEO-optimized rental paths used by the ImmoScout web UI when the user
// configures a maximum warmrent. Example: "wohnung-bis-800-euro-warm" means
// "apartment for rent up to 800 EUR warmrent". The web UI generates these
// paths instead of explicit `price` / `pricetype` query parameters.
// Note: only the warmrent variant uses an SEO slug; max coldrent searches
// use the regular "wohnung-mieten" path with explicit `price` and
// `pricetype=rentpermonth` query params, which the existing translator
// already handles.
const SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE = {
wohnung: 'apartmentrent',
haus: 'houserent',
};
const SEO_MAX_WARMRENT_PATH_PATTERN = /^(?<type>wohnung|haus)-bis-(?<price>\d+)-euro-warm$/;
/**
* Parses SEO-optimized ImmoScout web paths that encode a maximum warmrent, such
* as "wohnung-bis-800-euro-warm". Returns the corresponding mobile API real
* estate type and the implicit price/pricetype parameters, or null if the path
* does not match the known SEO max-warmrent pattern.
*
* @param {string} realTypeKey The last segment of the URL path.
* @returns {{ realType: string, additionalParams: Record<string, string> } | null}
*/
function parseSeoMaxWarmrentPath(realTypeKey) {
const match = realTypeKey.match(SEO_MAX_WARMRENT_PATH_PATTERN);
if (!match) return null;
const { type, price } = match.groups;
return {
realType: SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE[type],
additionalParams: {
price: `-${price}`,
pricetype: 'calculatedtotalrent',
},
};
}
export function convertWebToMobile(webUrl) {
let url;
try {
@@ -164,7 +201,14 @@ export function convertWebToMobile(webUrl) {
additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey];
realType = REAL_ESTATE_TYPE['wohnung-mieten'];
} else {
throw new Error(`Real estate type not found: ${realTypeKey}`);
// Test for SEO max-warmrent path, e.g. "wohnung-bis-800-euro-warm"
const seoMaxWarmrent = parseSeoMaxWarmrentPath(realTypeKey);
if (seoMaxWarmrent) {
realType = seoMaxWarmrent.realType;
additionalParamsFromWebPath = seoMaxWarmrent.additionalParams;
} else {
throw new Error(`Real estate type not found: ${realTypeKey}`);
}
}
}

View File

@@ -14,6 +14,7 @@ import * as similarityCache from '../similarity-check/similarityCache.js';
import { isRunning, markFinished, markRunning } from './run-state.js';
import { sendToUsers } from '../sse/sse-broker.js';
import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
import { getSettings } from '../storage/settingsStorage.js';
/**
* Initializes the job execution service.
@@ -104,14 +105,11 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
return;
}
settings.lastRun = now;
const jobs = jobStorage
.getJobs()
.filter((job) => job.enabled)
.filter((job) => {
if (!context) return true; // startup/cron → all
if (context.isAdmin) return true; // admin → all
return context.userId ? job.userId === context.userId : false; // user → own
});
const jobs = jobStorage.getJobs().filter((job) => {
if (!context) return true; // startup/cron → all
if (context.isAdmin) return true; // admin → all
return context.userId ? job.userId === context.userId : false; // user → own
});
for (const job of jobs) {
await executeJob(job);
@@ -160,6 +158,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
}
let browser;
try {
// Read the proxy live (not from the startup snapshot) so changing it in the
// UI takes effect on the next run without a backend restart. An empty value
// disables the proxy. Routing the headless browser through a (German
// residential) proxy avoids datacenter-IP based bot detection on the
// Puppeteer-based providers (immowelt, immonet, kleinanzeigen, ...).
const liveSettings = await getSettings();
const proxyUrl = typeof liveSettings?.proxyUrl === 'string' ? liveSettings.proxyUrl.trim() : '';
const jobProviders = job.provider.filter(
(p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
);
@@ -168,14 +174,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
if (browser && !browser.isConnected()) {
if (browser && !browser.connected) {
logger.debug('Browser is disconnected, nullifying to launch a new one.');
await puppeteerExtractor.closeBrowser(browser);
browser = null;
}
if (!browser && matchedProvider.config.getListings == null) {
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, proxyUrl ? { proxyUrl } : {});
}
await new FredyPipelineExecutioner(matchedProvider.config, job, prov.id, similarityCache, browser).execute();

View File

@@ -16,7 +16,7 @@ import logger from '../../services/logger.js';
* Concurrency: network-bound checks are executed with a configurable concurrency limit.
*
* @param {object} [opts]
* @param {number} [opts.concurrency=8] Max number of parallel activeTester calls.
* @param {number} [opts.concurrency=4] Max number of parallel activeTester calls.
* @returns {Promise<void>}
*/
export default async function runActiveChecker(opts = {}) {

View File

@@ -3,10 +3,27 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { nullOrEmpty } from '../../utils.js';
import { nullOrEmpty, fromJson } from '../../utils.js';
import SqliteConnection from './SqliteConnection.js';
import { nanoid } from 'nanoid';
/**
* Parse the JSON `status` column of a listing row in place.
*
* The DB stores status as a JSON payload `{ status, setAt }` (or NULL).
* Consumers expect an object/null, so we normalize before returning.
*
* @param {Object|null|undefined} row - A raw row from the listings table.
* @returns {Object|null|undefined} The same row with `status` parsed.
*/
const parseListingStatus = (row) => {
if (row == null) return row;
if (typeof row.status === 'string') {
row.status = fromJson(row.status, null);
}
return row;
};
/**
* Return a list of known listing hashes for a given job and provider.
* Useful to de-duplicate before inserting new listings.
@@ -43,18 +60,14 @@ export const getListingsKpisForJobIds = (jobIds = []) => {
const placeholders = jobIds.map(() => '?').join(',');
const rows = SqliteConnection.query(
`SELECT
SUM(CASE WHEN is_active = 1 THEN 1 ELSE 0 END) OVER() AS active_count,
price
FROM listings
WHERE job_id IN (${placeholders})
AND manually_deleted = 0
GROUP BY
id`,
`SELECT is_active, price
FROM listings
WHERE job_id IN (${placeholders})
AND manually_deleted = 0`,
jobIds,
);
const activeCount = rows[0]?.active_count ?? 0;
const activeCount = rows.filter((r) => r.is_active === 1).length;
const prices = rows
.map((r) => r.price)
@@ -214,6 +227,8 @@ export const storeListings = (jobId, providerId, listings) => {
longitude: item.longitude || null,
};
stmt.run(params);
// Propagate the DB primary key back so downstream pipeline steps use the correct id
item.id = params.id;
}
});
@@ -242,6 +257,7 @@ export const storeListings = (jobId, providerId, listings) => {
* @param {object} [params.jobNameFilter]
* @param {object} [params.providerFilter]
* @param {object} [params.watchListFilter]
* @param {('applied'|'rejected'|'accepted'|'none')} [params.statusFilter] - Filter by listing status. 'none' matches NULL.
* @param {string|null} [params.sortField=null] - One of: 'created_at','price','size','provider','title'.
* @param {('asc'|'desc')} [params.sortDir='asc']
* @param {number} [params.createdAfter] - Only include listings created at or after this unix timestamp (ms).
@@ -258,6 +274,7 @@ export const queryListings = ({
jobIdFilter,
providerFilter,
watchListFilter,
statusFilter,
freeTextFilter,
sortField = null,
sortDir = 'asc',
@@ -287,13 +304,15 @@ export const queryListings = ({
}
if (freeTextFilter && String(freeTextFilter).trim().length > 0) {
params.filter = `%${String(freeTextFilter).trim()}%`;
whereParts.push(`(title LIKE @filter OR address LIKE @filter OR provider LIKE @filter OR link LIKE @filter)`);
whereParts.push(
`(l.title LIKE @filter OR l.address LIKE @filter OR l.provider LIKE @filter OR l.link LIKE @filter)`,
);
}
// activityFilter: when true -> only active listings (is_active = 1), false -> only inactive
if (activityFilter === true) {
whereParts.push('(is_active = 1)');
whereParts.push('(l.is_active = 1)');
} else if (activityFilter === false) {
whereParts.push('(is_active = 0)');
whereParts.push('(l.is_active = 0)');
}
// Prefer filtering by job id when provided (unambiguous and robust)
if (jobIdFilter && String(jobIdFilter).trim().length > 0) {
@@ -307,7 +326,7 @@ export const queryListings = ({
// providerFilter: when provided as string (assumed provider name), filter listings where provider equals that name (exact match)
if (providerFilter && String(providerFilter).trim().length > 0) {
params.providerName = String(providerFilter).trim();
whereParts.push('(provider = @providerName)');
whereParts.push('(l.provider = @providerName)');
}
// watchListFilter: when true -> only watched listings, false -> only unwatched
if (watchListFilter === true) {
@@ -315,14 +334,26 @@ export const queryListings = ({
} else if (watchListFilter === false) {
whereParts.push('(wl.id IS NULL)');
}
// statusFilter: 'applied'|'rejected'|'accepted' -> equality on JSON status field; 'none' -> NULL.
// The status column is a JSON payload `{ status, setAt }`, so we extract the inner
// status string for comparison instead of matching the raw text.
if (statusFilter === 'none') {
whereParts.push('(l.status IS NULL)');
} else if (
typeof statusFilter === 'string' &&
['applied', 'rejected', 'accepted'].includes(statusFilter.toLowerCase())
) {
params.statusValue = statusFilter.toLowerCase();
whereParts.push(`(json_extract(l.status, '$.status') = @statusValue)`);
}
// Time range filters (unix timestamps in milliseconds)
if (Number.isFinite(createdAfter) && createdAfter > 0) {
params.createdAfter = createdAfter;
whereParts.push('(created_at >= @createdAfter)');
whereParts.push('(l.created_at >= @createdAfter)');
}
if (Number.isFinite(createdBefore) && createdBefore > 0) {
params.createdBefore = createdBefore;
whereParts.push('(created_at <= @createdBefore)');
whereParts.push('(l.created_at <= @createdBefore)');
}
// Price range filters
if (Number.isFinite(minPrice) && minPrice >= 0) {
@@ -337,32 +368,22 @@ export const queryListings = ({
// Build whereSql (filtering by manually_deleted = 0)
whereParts.push('(l.manually_deleted = 0)');
const whereSql = whereParts.length ? `WHERE ${whereParts.join(' AND ')}` : '';
const whereSqlWithAlias = whereSql
.replace(/\btitle\b/g, 'l.title')
.replace(/\bdescription\b/g, 'l.description')
.replace(/\baddress\b/g, 'l.address')
.replace(/\bprovider\b/g, 'l.provider')
.replace(/\blink\b/g, 'l.link')
.replace(/\bis_active\b/g, 'l.is_active')
.replace(/\bj\.user_id\b/g, 'j.user_id')
.replace(/\bj\.name\b/g, 'j.name')
.replace(/\bwl\.id\b/g, 'wl.id');
const whereSqlWithAlias = whereParts.length ? `WHERE ${whereParts.join(' AND ')}` : '';
// whitelist sortable fields to avoid SQL injection
const sortable = new Set(['created_at', 'price', 'size', 'provider', 'title', 'job_name', 'is_active', 'isWatched']);
const safeSortField = sortField && sortable.has(sortField) ? sortField : null;
// whitelist sortable fields to avoid SQL injection; map to fully-qualified expressions
const sortableMap = {
created_at: 'l.created_at',
price: 'l.price',
size: 'l.size',
provider: 'l.provider',
title: 'l.title',
job_name: 'j.name',
is_active: 'l.is_active',
isWatched: 'CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END',
};
const safeSortExpr = sortField && sortableMap[sortField] ? sortableMap[sortField] : null;
const safeSortDir = String(sortDir).toLowerCase() === 'desc' ? 'DESC' : 'ASC';
const orderSql = safeSortField ? `ORDER BY ${safeSortField} ${safeSortDir}` : 'ORDER BY created_at DESC';
const orderSqlWithAlias = orderSql
.replace(/\bcreated_at\b/g, 'l.created_at')
.replace(/\bprice\b/g, 'l.price')
.replace(/\bsize\b/g, 'l.size')
.replace(/\bprovider\b/g, 'l.provider')
.replace(/\btitle\b/g, 'l.title')
.replace(/\bjob_name\b/g, 'j.name')
// Sort by computed watch flag when requested
.replace(/\bisWatched\b/g, 'CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END');
const orderSqlWithAlias = safeSortExpr ? `ORDER BY ${safeSortExpr} ${safeSortDir}` : 'ORDER BY l.created_at DESC';
// count total with same WHERE
const countRow = SqliteConnection.query(
@@ -389,7 +410,7 @@ export const queryListings = ({
params,
);
return { totalNumber, page: safePage, result: rows };
return { totalNumber, page: safePage, result: rows.map(parseListingStatus) };
};
/**
@@ -417,9 +438,10 @@ export const deleteListingsByJobId = (jobId, hardDelete = false) => {
};
/**
* Delete listings by a list of listing IDs.
* Delete listings by a list of listing IDs (the nanoid primary key stored in the `id` column).
* Used by API routes that receive row IDs from the client.
*
* @param {string[]} ids - Array of listing IDs to delete.
* @param {string[]} ids - Array of DB row IDs to delete.
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
* @returns {any} The result from SqliteConnection.execute.
*/
@@ -482,7 +504,7 @@ export const updateListingGeocoordinates = (id, latitude, longitude) => {
* @param {string} [params.jobId]
* @param {string} [params.userId]
* @param {boolean} [params.isAdmin=false]
* @returns {{listings: Object[], maxPrice: number}} Object containing listings and maxPrice.
* @returns {{listings: Object[]}} Object containing listings.
*/
export const getListingsForMap = ({ jobId, userId = null, isAdmin = false } = {}) => {
const baseWhereParts = [
@@ -623,7 +645,7 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
if (!isAdmin) {
whereScoping = `AND (j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`;
}
return (
return parseListingStatus(
SqliteConnection.query(
`SELECT l.*, j.name AS job_name, CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END AS isWatched
FROM listings l
@@ -631,10 +653,57 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
LEFT JOIN watch_list wl ON wl.listing_id = l.id AND wl.user_id = @userId
WHERE l.id = @id AND l.manually_deleted = 0 ${whereScoping}`,
params,
)[0] || null
)[0] || null,
);
};
/**
* Set or clear the notes attached to a single listing.
*
* Empty strings are normalized to NULL so the DB doesn't keep meaningless
* whitespace and queries can filter "has notes" with a simple IS NOT NULL.
*
* @param {string} id - The listing ID.
* @param {string|null} notes - The note text to store, or null/empty to clear.
* @returns {number} Number of rows affected (0 if listing not found).
*/
export const setListingNotes = (id, notes) => {
if (!id) return 0;
const trimmed = typeof notes === 'string' ? notes.trim() : null;
const value = trimmed && trimmed.length > 0 ? trimmed : null;
const res = SqliteConnection.execute(`UPDATE listings SET notes = @notes WHERE id = @id`, {
id,
notes: value,
});
return res?.changes ?? 0;
};
/**
* Set or clear the status of a single listing.
*
* The status column stores a JSON payload `{ status, setAt }` so consumers
* can show both the user's decision and when it was made. Passing `null`
* clears the column.
*
* @param {string} id - The listing ID.
* @param {('applied'|'rejected'|'accepted'|null)} status - New status, or null to clear.
* @returns {number} Number of rows affected (0 if listing not found).
*/
export const setListingStatus = (id, status) => {
if (!id) return 0;
const allowed = ['applied', 'rejected', 'accepted'];
const normalized = status == null ? null : String(status).toLowerCase();
if (normalized != null && !allowed.includes(normalized)) {
throw new Error(`Invalid listing status: ${status}`);
}
const payload = normalized == null ? null : JSON.stringify({ status: normalized, setAt: Date.now() });
const res = SqliteConnection.execute(`UPDATE listings SET status = @status WHERE id = @id`, {
id,
status: payload,
});
return res?.changes ?? 0;
};
/**
* Resets geocoordinates and distance for all listings related to a user.
*

View File

@@ -0,0 +1,11 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
export function up(db) {
db.exec(`
ALTER TABLE listings ADD COLUMN status JSON;
CREATE INDEX IF NOT EXISTS idx_listings_status ON listings (json_extract(status, '$.status'));
`);
}

View File

@@ -0,0 +1,10 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
export function up(db) {
db.exec(`
ALTER TABLE listings ADD COLUMN notes TEXT;
`);
}

View File

@@ -123,8 +123,11 @@ export function upsertSettings(settingsMapOrEntry, userId = null) {
);
}
}
// keep cache in sync (only for global settings)
// Invalidate cache synchronously so the next getSettings() call rebuilds it.
// refreshSettingsCache() is async (reads config.json), so we cannot await it
// here without making upsertSettings async everywhere. Nulling is safe because
// getSettings() will call refreshSettingsCache() on the next invocation.
if (userId == null) {
refreshSettingsCache();
cachedSettingsConfig = null;
}
}

View File

@@ -47,6 +47,25 @@ export const deleteWatch = (listingId, userId) => {
return { deleted: Boolean(res?.changes) };
};
/**
* Ensure a watch entry exists. Does not toggle; safe to call when row may already exist.
* Used by the status endpoint to auto-watch a listing when a status is set.
* @param {string} listingId
* @param {string} userId
* @returns {{watched:boolean}}
*/
export const ensureWatch = (listingId, userId) => {
if (!listingId || !userId) return { watched: false };
const { created } = createWatch(listingId, userId);
if (created) return { watched: true };
const exists =
SqliteConnection.query(
`SELECT 1 AS ok FROM watch_list WHERE listing_id = @listing_id AND user_id = @user_id LIMIT 1`,
{ listing_id: listingId, user_id: userId },
).length > 0;
return { watched: exists };
};
/**
* Toggle a watch entry. If exists -> delete, otherwise create.
* @param {string} listingId

View File

@@ -1,6 +1,6 @@
{
"name": "fredy",
"version": "22.0.0",
"version": "22.3.3",
"description": "[F]ind [R]eal [E]states [d]amn eas[y].",
"scripts": {
"prepare": "husky",
@@ -62,9 +62,9 @@
"Firefox ESR"
],
"dependencies": {
"@douyinfe/semi-icons": "^2.96.1",
"@douyinfe/semi-ui": "2.96.1",
"@douyinfe/semi-ui-19": "^2.96.1",
"@douyinfe/semi-icons": "^2.99.3",
"@douyinfe/semi-ui": "2.99.3",
"@douyinfe/semi-ui-19": "^2.99.3",
"@fastify/cookie": "^11.0.2",
"@fastify/helmet": "^13.0.2",
"@fastify/session": "^11.1.1",
@@ -73,12 +73,12 @@
"@modelcontextprotocol/sdk": "^1.29.0",
"@sendgrid/mail": "8.1.6",
"@turf/boolean-point-in-polygon": "^7.3.5",
"@vitejs/plugin-react": "6.0.1",
"@vitejs/plugin-react": "6.0.2",
"adm-zip": "^0.5.17",
"better-sqlite3": "^12.9.0",
"better-sqlite3": "^12.10.0",
"chart.js": "^4.5.1",
"cheerio": "^1.2.0",
"cloakbrowser": "^0.3.27",
"cloakbrowser": "^0.3.31",
"fastify": "^5.8.5",
"handlebars": "4.7.9",
"maplibre-gl": "^5.24.0",
@@ -86,41 +86,41 @@
"node-cron": "^4.2.1",
"node-fetch": "3.3.2",
"node-mailjet": "6.0.11",
"nodemailer": "^8.0.7",
"nodemailer": "^8.0.10",
"p-throttle": "^8.1.0",
"package-up": "^5.0.0",
"puppeteer-core": "^24.43.0",
"query-string": "9.3.1",
"react": "19.2.6",
"puppeteer-core": "^25.1.0",
"query-string": "9.4.0",
"react": "19.2.7",
"react-chartjs-2": "^5.3.1",
"react-dom": "19.2.6",
"react-dom": "19.2.7",
"react-range-slider-input": "^3.3.5",
"react-router": "7.15.0",
"react-router-dom": "7.15.0",
"resend": "^6.12.3",
"semver": "^7.7.4",
"react-router": "7.16.0",
"react-router-dom": "7.16.0",
"resend": "^6.12.4",
"semver": "^7.8.1",
"slack": "11.0.2",
"vite": "8.0.11",
"vite": "8.0.16",
"x-var": "^3.0.1",
"zustand": "^5.0.13"
"zustand": "^5.0.14"
},
"devDependencies": {
"@babel/core": "7.29.0",
"@babel/eslint-parser": "7.28.6",
"@babel/preset-env": "7.29.5",
"@babel/preset-react": "7.28.5",
"@babel/core": "7.29.7",
"@babel/eslint-parser": "7.29.7",
"@babel/preset-env": "7.29.7",
"@babel/preset-react": "7.29.7",
"@eslint/js": "^10.0.1",
"chalk": "^5.6.2",
"eslint": "10.3.0",
"eslint": "10.4.1",
"eslint-config-prettier": "10.1.8",
"eslint-plugin-react": "7.37.5",
"globals": "^17.6.0",
"history": "5.3.0",
"husky": "9.1.7",
"less": "4.6.4",
"lint-staged": "16.4.0",
"lint-staged": "17.0.7",
"nodemon": "^3.1.14",
"prettier": "3.8.3",
"vitest": "^4.1.5"
"vitest": "^4.1.8"
}
}

View File

@@ -32,4 +32,7 @@ export const deletedIds = [];
export const deleteListingsById = (ids) => {
deletedIds.push(...ids);
};
export const deleteListingsByHash = (hashes) => {
deletedIds.push(...hashes);
};
/* eslint-enable no-unused-vars */

View File

@@ -0,0 +1,415 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
// Mock external deps BEFORE importing the module under test.
vi.mock('node-fetch', () => ({ default: vi.fn() }));
vi.mock('../../lib/services/storage/jobStorage.js', () => ({
getJob: (jobKey) => ({ id: jobKey, name: jobKey }),
}));
vi.mock('../../lib/services/markdown.js', () => ({
markdown2Html: () => '',
}));
// Helpers to build mock fetch responses.
function jsonOk(body = { ok: true }) {
return {
ok: true,
status: 200,
text: async () => JSON.stringify(body),
};
}
function jsonErr(status, body) {
return {
ok: false,
status,
text: async () => JSON.stringify(body),
};
}
function imageOk(bytes = new Uint8Array([0xff, 0xd8, 0xff])) {
return {
ok: true,
status: 200,
headers: {
get: (h) => {
const k = h.toLowerCase();
if (k === 'content-type') return 'image/jpeg';
if (k === 'content-length') return String(bytes.byteLength);
return null;
},
},
arrayBuffer: async () => bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength),
};
}
// Globals are mocked too so buildPhotoFormData (which uses global fetch) can be
// intercepted by the same single mock.
let mockNodeFetch;
let mockGlobalFetch;
let send;
beforeEach(async () => {
// Reset modules to get a fresh import with our mocks applied.
vi.resetModules();
const nodeFetchMod = await import('node-fetch');
mockNodeFetch = nodeFetchMod.default;
mockNodeFetch.mockReset();
mockGlobalFetch = vi.fn();
vi.stubGlobal('fetch', mockGlobalFetch);
({ send } = await import('../../lib/notification/adapter/telegram.js'));
});
afterEach(() => {
vi.unstubAllGlobals();
vi.useRealTimers();
});
const baseConfig = {
id: 'telegram',
fields: { token: 'TKN', chatId: '999' },
};
describe('telegram send() - HTTP URL path (default for .jpg / .png)', () => {
it('POSTs JSON to sendPhoto for a .jpg image URL', async () => {
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 'Listing',
link: 'https://example.com/a',
address: 'Addr',
price: '500€',
size: '50m²',
image: 'https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
const [url, opts] = mockNodeFetch.mock.calls[0];
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
expect(opts.method).toBe('post');
expect(opts.headers?.['Content-Type']).toBe('application/json');
const body = JSON.parse(opts.body);
expect(body.chat_id).toBe('999');
expect(body.photo).toBe('https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394');
expect(body.parse_mode).toBe('HTML');
});
it('does NOT pre-fetch the image when using HTTP URL path', async () => {
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/x.jpg',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
// global fetch (used by buildPhotoFormData) must not be called
expect(mockGlobalFetch).not.toHaveBeenCalled();
});
it('falls back to sendMessage when sendPhoto fails', async () => {
mockNodeFetch
.mockResolvedValueOnce(jsonErr(400, { ok: false, description: 'boom' }))
.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/x.jpg',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendPhoto');
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
});
});
describe('telegram send() - multipart path (.webp URLs)', () => {
it('pre-fetches the image then POSTs FormData to sendPhoto for a .webp URL', async () => {
// 1st: GET image via global fetch
mockGlobalFetch.mockResolvedValueOnce(imageOk());
// 2nd: POST sendPhoto via node-fetch
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 'Listing',
link: 'https://example.com/a',
address: 'Addr',
price: '500€',
size: '50m²',
image: 'https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
// image was fetched
expect(mockGlobalFetch).toHaveBeenCalledTimes(1);
expect(mockGlobalFetch.mock.calls[0][0]).toBe('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394');
// sendPhoto called via node-fetch with FormData
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
const [url, opts] = mockNodeFetch.mock.calls[0];
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
expect(opts.method).toBe('post');
expect(opts.body).toBeInstanceOf(FormData);
// No explicit Content-Type header - fetch sets multipart boundary itself
expect(opts.headers).toBeUndefined();
expect(opts.body.get('chat_id')).toBe('999');
expect(opts.body.get('parse_mode')).toBe('HTML');
const photo = opts.body.get('photo');
expect(photo).toBeTruthy();
expect(photo.size).toBeGreaterThan(0);
});
it('falls back to sendMessage when the image pre-fetch fails for a .webp URL', async () => {
// image fetch fails (404 from CDN)
mockGlobalFetch.mockResolvedValueOnce({
ok: false,
status: 404,
headers: { get: () => null },
arrayBuffer: async () => new ArrayBuffer(0),
});
// then sendMessage succeeds via node-fetch
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/gone.webp',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
});
it('falls back to sendMessage when multipart sendPhoto returns a Telegram error', async () => {
mockGlobalFetch.mockResolvedValueOnce(imageOk());
mockNodeFetch
.mockResolvedValueOnce(jsonErr(400, { description: 'broke' })) // multipart sendPhoto
.mockResolvedValueOnce(jsonOk()); // sendMessage fallback
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/x.webp',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
});
});
describe('telegram send() - mixed batch (regression-safety)', () => {
it('handles a batch with both .jpg and .webp - jpg uses URL, webp uses multipart', async () => {
// .webp image fetch
mockGlobalFetch.mockResolvedValueOnce(imageOk());
// both sendPhoto calls succeed
mockNodeFetch
.mockResolvedValueOnce(jsonOk()) // could be either listing first
.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'jpg-listing',
title: 'a',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/a.jpg',
},
{
id: 'webp-listing',
title: 'b',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/b.webp',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockGlobalFetch).toHaveBeenCalledTimes(1); // only webp pre-fetches
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
// Verify one call had FormData and one had JSON body
const bodies = mockNodeFetch.mock.calls.map((c) => c[1].body);
const hasFormData = bodies.some((b) => b instanceof FormData);
const hasJson = bodies.some((b) => typeof b === 'string' && b.startsWith('{'));
expect(hasFormData).toBe(true);
expect(hasJson).toBe(true);
});
it('uses sendMessage (not sendPhoto) when image is null', async () => {
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: null,
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
expect(mockGlobalFetch).not.toHaveBeenCalled();
});
});
describe('telegram send() - multiple chat IDs', () => {
const listing = {
id: '1',
title: 'Flat',
link: 'https://ex.com',
address: 'Berlin',
price: '800',
size: '50',
image: 'https://ex.com/img.jpg',
};
it('sends to every chat ID in a comma-separated list', async () => {
mockNodeFetch.mockResolvedValue(jsonOk());
await send({
serviceName: 'immoscout',
newListings: [listing],
notificationConfig: [{ id: 'telegram', fields: { token: 'TKN', chatId: '111, 222' } }],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
const bodies = mockNodeFetch.mock.calls.map((c) => JSON.parse(c[1].body));
expect(bodies.map((b) => b.chat_id)).toEqual(expect.arrayContaining(['111', '222']));
});
it('trims whitespace around each chat ID', async () => {
mockNodeFetch.mockResolvedValue(jsonOk());
await send({
serviceName: 'immoscout',
newListings: [listing],
notificationConfig: [{ id: 'telegram', fields: { token: 'TKN', chatId: ' 333 , 444 ' } }],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
const bodies = mockNodeFetch.mock.calls.map((c) => JSON.parse(c[1].body));
expect(bodies.map((b) => b.chat_id)).toEqual(expect.arrayContaining(['333', '444']));
});
it('sends each listing to each chat ID (N listings × M chats)', async () => {
mockNodeFetch.mockResolvedValue(jsonOk());
await send({
serviceName: 'immoscout',
newListings: [listing, { ...listing, id: '2' }],
notificationConfig: [{ id: 'telegram', fields: { token: 'TKN', chatId: '555, 666' } }],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(4);
});
});
describe('telegram send() - config validation', () => {
it('throws when telegram adapter config is missing', () => {
expect(() =>
send({
serviceName: 's',
newListings: [],
notificationConfig: [],
jobKey: 'k',
}),
).toThrow(/configuration missing/);
});
it('throws when token or chatId is missing', () => {
expect(() =>
send({
serviceName: 's',
newListings: [],
notificationConfig: [{ id: 'telegram', fields: { token: '' } }],
jobKey: 'k',
}),
).toThrow(/token.*chatId/);
});
});

View File

@@ -0,0 +1,287 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { shouldUseMultipart, buildPhotoFormData } from '../../lib/notification/adapter/telegramPhotoUploader.js';
describe('shouldUseMultipart', () => {
it('returns true for .webp URL with query string', () => {
expect(shouldUseMultipart('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394')).toBe(true);
});
it('returns true for .webp URL without query string', () => {
expect(shouldUseMultipart('https://example.com/photo.webp')).toBe(true);
});
it('returns true for uppercase .WEBP extension', () => {
expect(shouldUseMultipart('https://example.com/IMG.WEBP?x=1')).toBe(true);
});
it('returns false for .jpg URL with query string', () => {
expect(shouldUseMultipart('https://mms.immowelt.de/a/b/c/d/xyz.jpg?ci_seal=hash&w=525&h=394')).toBe(false);
});
it('returns false for .jpeg URL', () => {
expect(shouldUseMultipart('https://example.com/photo.jpeg')).toBe(false);
});
it('returns false for .png URL with query string', () => {
expect(shouldUseMultipart('https://example.com/photo.png?w=100')).toBe(false);
});
it('returns false for .gif URL', () => {
expect(shouldUseMultipart('https://example.com/photo.gif')).toBe(false);
});
it('returns false for null', () => {
expect(shouldUseMultipart(null)).toBe(false);
});
it('returns false for undefined', () => {
expect(shouldUseMultipart(undefined)).toBe(false);
});
it('returns false for empty string', () => {
expect(shouldUseMultipart('')).toBe(false);
});
it('returns false for malformed URL', () => {
expect(shouldUseMultipart('not a url')).toBe(false);
});
it('returns false for URL where webp is in the query but not the path', () => {
expect(shouldUseMultipart('https://example.com/photo.jpg?format=webp')).toBe(false);
});
it('returns false for URL with no extension at all', () => {
expect(shouldUseMultipart('https://example.com/photo')).toBe(false);
});
it('returns false for non-https schemes', () => {
// file/data/ftp URLs should not be relevant; safer to skip multipart
expect(shouldUseMultipart('http://example.com/photo.webp')).toBe(false);
});
});
describe('buildPhotoFormData', () => {
let mockFetch;
beforeEach(() => {
mockFetch = vi.fn();
vi.stubGlobal('fetch', mockFetch);
});
afterEach(() => {
vi.unstubAllGlobals();
});
function makeImageResponse({ contentType = 'image/jpeg', bytes = new Uint8Array([0xff, 0xd8, 0xff]) } = {}) {
const buf = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
return {
ok: true,
status: 200,
headers: {
get: (h) =>
h.toLowerCase() === 'content-type'
? contentType
: h.toLowerCase() === 'content-length'
? String(bytes.byteLength)
: null,
},
arrayBuffer: async () => buf,
};
}
it('fetches image with Accept header that excludes webp so the CDN transcodes to JPEG', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
await buildPhotoFormData({
chatId: '123',
imageUrl: 'https://example.com/photo.webp',
caption: 'hi',
parseMode: 'HTML',
});
expect(mockFetch).toHaveBeenCalledTimes(1);
const [url, opts] = mockFetch.mock.calls[0];
expect(url).toBe('https://example.com/photo.webp');
expect(opts?.headers?.Accept || opts?.headers?.accept).toMatch(/image\/jpeg/);
expect(opts?.headers?.Accept || opts?.headers?.accept).not.toMatch(/image\/webp/);
});
it('returns FormData containing chat_id, caption, parse_mode, and photo fields', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '12345',
imageUrl: 'https://example.com/abc.webp',
caption: 'My caption',
parseMode: 'HTML',
});
expect(fd).toBeInstanceOf(FormData);
expect(fd.get('chat_id')).toBe('12345');
expect(fd.get('caption')).toBe('My caption');
expect(fd.get('parse_mode')).toBe('HTML');
const photo = fd.get('photo');
expect(photo).toBeTruthy();
// File-like (Blob); has a name and a size
expect(typeof photo.name).toBe('string');
expect(photo.size).toBeGreaterThan(0);
});
it('uses a .jpg filename (Telegram uses URL/filename extension for type detection)', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
parseMode: 'HTML',
});
const photo = fd.get('photo');
expect(photo.name).toMatch(/\.jpg$/i);
});
it('includes message_thread_id when provided', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
parseMode: 'HTML',
messageThreadId: 42,
});
expect(fd.get('message_thread_id')).toBe('42');
});
it('omits message_thread_id when not provided', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
parseMode: 'HTML',
});
expect(fd.get('message_thread_id')).toBeNull();
});
it('omits parse_mode when not provided (plain text mode)', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
});
expect(fd.get('parse_mode')).toBeNull();
});
it('throws when the image fetch returns non-200', async () => {
mockFetch.mockResolvedValueOnce({
ok: false,
status: 404,
headers: { get: () => null },
arrayBuffer: async () => new ArrayBuffer(0),
});
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/gone.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/404/);
});
it('throws when the image fetch throws (network error)', async () => {
mockFetch.mockRejectedValueOnce(new Error('ECONNREFUSED'));
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/x.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/ECONNREFUSED/);
});
it('throws when the image exceeds 10 MB (Telegram multipart limit)', async () => {
// 11 MB
const big = new Uint8Array(11 * 1024 * 1024);
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes: big }));
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/huge.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/size|large|10/i);
});
it('rejects early when content-length header advertises > 10 MB (avoids download)', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
status: 200,
headers: {
get: (h) => {
const k = h.toLowerCase();
if (k === 'content-type') return 'image/jpeg';
if (k === 'content-length') return String(50 * 1024 * 1024);
return null;
},
},
arrayBuffer: async () => {
throw new Error('should not be called - size check should reject first');
},
});
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/huge.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/size|large|10/i);
});
it('accepts exactly 10 MB images (boundary)', async () => {
const bytes = new Uint8Array(10 * 1024 * 1024);
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes }));
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/exact.webp',
caption: 'c',
parseMode: 'HTML',
});
expect(fd.get('photo').size).toBe(10 * 1024 * 1024);
});
it('coerces non-string chatId (number) to string in form data', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: 999,
imageUrl: 'https://example.com/x.webp',
caption: 'c',
parseMode: 'HTML',
});
expect(fd.get('chat_id')).toBe('999');
});
});

View File

@@ -38,6 +38,20 @@ async function tryReadFile(filepath) {
}
}
function withRealEstateType(data, realEstateType) {
if (!realEstateType?.length || !Array.isArray(data?.resultListItems)) {
return data;
}
const cloned = typeof structuredClone === 'function' ? structuredClone(data) : JSON.parse(JSON.stringify(data));
for (const item of cloned.resultListItems) {
if (item?.type === 'EXPOSE_RESULT' && item?.item) {
item.item.realEstateType = realEstateType;
}
}
return cloned;
}
/**
* Returns fixture HTML for the given URL by mapping hostname → provider name,
* then distinguishing list vs detail pages by comparing the URL path against
@@ -83,7 +97,10 @@ export function buildFetchMock() {
const raw = await tryReadFile(path.join(FIXTURES_DIR, 'immoscout_list.json'));
listData = raw ? JSON.parse(raw) : { resultListItems: [] };
}
return { ok: true, status: 200, json: () => Promise.resolve(listData) };
const requestedType = new URL(urlStr).searchParams.get('realestatetype');
const responseData = withRealEstateType(listData, requestedType);
return { ok: true, status: 200, json: () => Promise.resolve(responseData) };
}
if (urlStr.includes('api.mobile.immobilienscout24.de/expose/')) {

View File

@@ -0,0 +1,37 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { vi, describe, it, expect, beforeEach } from 'vitest';
// Mock the CloakBrowser launcher so no real Chromium binary is needed and we can
// assert which options get forwarded to it.
const { launchMock } = vi.hoisted(() => ({ launchMock: vi.fn() }));
vi.mock('cloakbrowser/puppeteer', () => ({
launch: launchMock,
}));
const { launchBrowser } = await import('../../../lib/services/extractor/puppeteerExtractor.js');
describe('launchBrowser proxy forwarding', () => {
beforeEach(() => {
launchMock.mockReset();
launchMock.mockResolvedValue({ close: async () => {} });
});
it('forwards proxyUrl to CloakBrowser as the proxy option', async () => {
await launchBrowser('https://www.immowelt.de/', { proxyUrl: 'http://user:pass@host:8080' });
expect(launchMock).toHaveBeenCalledTimes(1);
expect(launchMock.mock.calls[0][0]).toMatchObject({ proxy: 'http://user:pass@host:8080' });
});
it('does not set a proxy when no proxyUrl is given', async () => {
await launchBrowser('https://www.immowelt.de/', {});
expect(launchMock).toHaveBeenCalledTimes(1);
expect(launchMock.mock.calls[0][0].proxy).toBeUndefined();
});
});

View File

@@ -4,11 +4,16 @@
*/
import { convertWebToMobile } from '../../../lib/services/immoscout/immoscout-web-translator.js';
import { expect } from 'vitest';
import { expect, vi } from 'vitest';
import { readFile } from 'fs/promises';
import { buildFetchMock } from '../../offlineFixtures.js';
export const testData = JSON.parse(await readFile(new URL('./testdata.json', import.meta.url)));
if (process.env.TEST_MODE === 'offline') {
vi.stubGlobal('fetch', buildFetchMock());
}
describe('#immoscout-mobile URL conversion', () => {
// Test shape URL conversion
it('should convert a full web URL with shape to mobile URL', () => {
@@ -41,6 +46,60 @@ describe('#immoscout-mobile URL conversion', () => {
expect(queryParams.get('equipment').split(',')).toEqual(expect.arrayContaining(['garden', 'balcony']));
});
// Test URL conversion of SEO web path for max warmrent. The ImmoScout web UI
// generates this special SEO slug instead of explicit price/pricetype params
// when the user configures a "Warmmiete" filter (real-world URL).
it('should convert a SEO apartment max warmrent path to rent + price + pricetype', () => {
const webUrl =
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-bis-800-euro-warm?livingspace=-800.0&enteredFrom=result_list';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
expect(queryParams.get('price')).toBe('-800');
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
expect(queryParams.get('livingspace')).toBe('-800.0');
});
// Same SEO pattern for houses ("haus-bis-X-euro-warm" → houserent).
it('should convert a SEO house max warmrent path to rent + price + pricetype', () => {
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/haus-bis-1500-euro-warm';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('houserent');
expect(queryParams.get('price')).toBe('-1500');
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
});
// Sanity check: max coldrent ("Kaltmiete") does NOT use an SEO slug. The web
// UI keeps the regular "wohnung-mieten" path and passes explicit
// price + pricetype query params, which the existing translator already
// handles (real-world URL).
it('should convert a max coldrent search via the regular wohnung-mieten path', () => {
const webUrl =
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-mieten?price=-800.0&livingspace=-800.0&pricetype=rentpermonth&enteredFrom=result_list';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
expect(queryParams.get('price')).toBe('-800.0');
expect(queryParams.get('pricetype')).toBe('rentpermonth');
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
});
// Explicit query params win over the SEO slug's implicit defaults.
it('should let explicit query params override SEO path price defaults', () => {
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-bis-800-euro-warm?price=100-500';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
expect(queryParams.get('price')).toBe('100-500');
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
});
// Test URL conversion with unsupported query parameters
it('should remove unsupported query parameters', () => {
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?minimuminternetspeed=100000';

View File

@@ -18,6 +18,7 @@ describe('services/jobs/jobExecutionService', () => {
const busPath = root + '/lib/services/events/event-bus.js';
const jobStoragePath = root + '/lib/services/storage/jobStorage.js';
const userStoragePath = root + '/lib/services/storage/userStorage.js';
const settingsStoragePath = root + '/lib/services/storage/settingsStorage.js';
const brokerPath = root + '/lib/services/sse/sse-broker.js';
const utilsPath = root + '/lib/utils.js';
const loggerPath = root + '/lib/services/logger.js';
@@ -33,11 +34,15 @@ describe('services/jobs/jobExecutionService', () => {
getUsers: () => state.users.slice(),
getUser: (id) => state.users.find((u) => u.id === id) || null,
}));
vi.doMock(settingsStoragePath, () => ({
getSettings: async () => ({}),
}));
vi.doMock(brokerPath, () => ({
sendToUsers: (...args) => calls.sent.push(args),
}));
vi.doMock(utilsPath, () => ({
duringWorkingHoursOrNotSet: () => false,
getPackageVersion: async () => '0.0.0-test',
}));
vi.doMock(loggerPath, () => {
const m = { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} };

View File

@@ -0,0 +1,193 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { vi, describe, it, expect, beforeEach } from 'vitest';
// We mock SqliteConnection so we can assert which SQL the storage layer
// runs and with which params, without spinning up a real SQLite DB.
const calls = {
execute: [],
query: [],
};
const sqliteMock = {
execute: (sql, params) => {
calls.execute.push({ sql, params });
// Default: pretend 1 row was affected (so setListingStatus reports success).
return { changes: 1 };
},
query: (sql, params) => {
calls.query.push({ sql, params });
// Return shape varies by test — overridden via queryHandler when needed.
if (sqliteMock.__queryHandler) return sqliteMock.__queryHandler(sql, params);
return [];
},
__queryHandler: null,
};
vi.mock('../../lib/services/storage/SqliteConnection.js', () => ({
default: sqliteMock,
}));
describe('listingsStorage.setListingStatus', () => {
let listingsStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
sqliteMock.__queryHandler = null;
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
});
it('runs an UPDATE storing a JSON payload with status and setAt', () => {
const before = Date.now();
const changes = listingsStorage.setListingStatus('listing-1', 'Applied');
const after = Date.now();
expect(changes).toBe(1);
expect(calls.execute).toHaveLength(1);
expect(calls.execute[0].sql).toMatch(/UPDATE listings SET status = @status WHERE id = @id/);
expect(calls.execute[0].params.id).toBe('listing-1');
const parsed = JSON.parse(calls.execute[0].params.status);
expect(parsed.status).toBe('applied');
expect(parsed.setAt).toBeGreaterThanOrEqual(before);
expect(parsed.setAt).toBeLessThanOrEqual(after);
});
it('accepts null to clear the status (no JSON wrapping)', () => {
listingsStorage.setListingStatus('listing-2', null);
expect(calls.execute[0].params).toEqual({ id: 'listing-2', status: null });
});
it('rejects invalid statuses', () => {
expect(() => listingsStorage.setListingStatus('listing-3', 'maybe')).toThrow(/Invalid listing status/);
expect(calls.execute).toHaveLength(0);
});
it('returns 0 when no id is supplied (no SQL is run)', () => {
const result = listingsStorage.setListingStatus(null, 'applied');
expect(result).toBe(0);
expect(calls.execute).toHaveLength(0);
});
});
describe('listingsStorage.queryListings statusFilter', () => {
let listingsStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
// Return empty rows for both the count and the page-fetch queries.
sqliteMock.__queryHandler = (sql) => {
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 0 }];
return [];
};
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
});
it("adds 'l.status IS NULL' to WHERE when statusFilter is 'none'", () => {
listingsStorage.queryListings({ statusFilter: 'none', userId: 'u1', isAdmin: true });
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
expect(pageQuery.sql).toMatch(/\(l\.status IS NULL\)/);
});
it('extracts the inner status field via json_extract for a concrete status', () => {
listingsStorage.queryListings({ statusFilter: 'applied', userId: 'u1', isAdmin: true });
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
expect(pageQuery.sql).toMatch(/json_extract\(l\.status, '\$\.status'\) = @statusValue/);
expect(pageQuery.params.statusValue).toBe('applied');
});
it('ignores unknown statusFilter values silently', () => {
listingsStorage.queryListings({ statusFilter: 'bogus', userId: 'u1', isAdmin: true });
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
expect(pageQuery.sql).not.toMatch(/status/i);
});
it('parses the JSON status payload of returned rows into an object', () => {
sqliteMock.__queryHandler = (sql) => {
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 2 }];
return [
{ id: 'a', status: JSON.stringify({ status: 'applied', setAt: 1700000000000 }) },
{ id: 'b', status: null },
];
};
const result = listingsStorage.queryListings({ userId: 'u1', isAdmin: true });
expect(result.result[0].status).toEqual({ status: 'applied', setAt: 1700000000000 });
expect(result.result[1].status).toBeNull();
});
});
describe('listingsStorage.getListingById', () => {
let listingsStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
});
it('parses the JSON status payload of the returned row', () => {
sqliteMock.__queryHandler = () => [
{ id: 'a', status: JSON.stringify({ status: 'rejected', setAt: 1700000000001 }) },
];
const row = listingsStorage.getListingById('a', 'u1', true);
expect(row.status).toEqual({ status: 'rejected', setAt: 1700000000001 });
});
it('returns null status untouched', () => {
sqliteMock.__queryHandler = () => [{ id: 'a', status: null }];
const row = listingsStorage.getListingById('a', 'u1', true);
expect(row.status).toBeNull();
});
it('returns null when no row is found', () => {
sqliteMock.__queryHandler = () => [];
const row = listingsStorage.getListingById('missing', 'u1', true);
expect(row).toBeNull();
});
});
describe('watchListStorage.ensureWatch', () => {
let watchListStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
sqliteMock.__queryHandler = null;
watchListStorage = await import('../../lib/services/storage/watchListStorage.js');
});
it('inserts and reports watched=true on first call', () => {
// After INSERT, createWatch queries for existence and gets a row back.
sqliteMock.__queryHandler = () => [{ ok: 1 }];
const result = watchListStorage.ensureWatch('listing-1', 'user-1');
expect(result).toEqual({ watched: true });
// INSERT should have been issued.
expect(calls.execute.some((c) => /INSERT INTO watch_list/.test(c.sql))).toBe(true);
});
it('returns watched=true when an entry already exists', () => {
// Simulate ON CONFLICT being a no-op: execute reports no changes, then SELECT confirms row exists.
sqliteMock.execute = (sql, params) => {
calls.execute.push({ sql, params });
return { changes: 0 };
};
sqliteMock.__queryHandler = () => [{ ok: 1 }];
const result = watchListStorage.ensureWatch('listing-2', 'user-2');
expect(result).toEqual({ watched: true });
// Restore execute to default for subsequent tests.
sqliteMock.execute = (sql, params) => {
calls.execute.push({ sql, params });
return { changes: 1 };
};
});
it('returns watched=false when listingId or userId is missing', () => {
expect(watchListStorage.ensureWatch(null, 'u')).toEqual({ watched: false });
expect(watchListStorage.ensureWatch('l', null)).toEqual({ watched: false });
expect(calls.execute).toHaveLength(0);
});
});

View File

@@ -155,6 +155,7 @@ const routes = {
'GET /api/dashboard': dashboard,
'GET /api/demo': { demoMode: false },
'POST /api/user/settings/news-hash': {},
'POST /api/user/settings/listing-deletion-preference': {},
};
const server = http.createServer((req, res) => {

View File

@@ -95,7 +95,10 @@ async function downloadHtmlProvider(name, providerConfig, launchBrowser, closeBr
const browser = await launchBrowser(providerConfig.url, {});
try {
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, { browser });
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, {
browser,
name: 'dowload_fixtures',
});
if (!html) {
console.warn(` Failed to download ${name}`);

View File

@@ -97,6 +97,7 @@ export default function FredyApp() {
<Route path="/dashboard" element={<Dashboard />} />
<Route path="/jobs" element={<Jobs />} />
<Route path="/listings" element={<Listings />} />
<Route path="/listings/watchlist" element={<Listings mode="watchlist" />} />
<Route path="/listings/listing/:listingId" element={<ListingDetail />} />
<Route path="/map" element={<MapView />} />
<Route path="/watchlistManagement" element={<WatchlistManagement />} />

View File

@@ -3,8 +3,8 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { useState } from 'react';
import { Modal, Radio, RadioGroup, Typography } from '@douyinfe/semi-ui-19';
import { useState, useEffect } from 'react';
import { Modal, Radio, RadioGroup, Typography, Checkbox } from '@douyinfe/semi-ui-19';
const { Text } = Typography;
@@ -15,11 +15,24 @@ const ListingDeletionModal = ({
title = 'Delete Listings',
showOptions = true,
message = 'How would you like to delete the selected listing(s)?',
defaultDeleteType = 'soft',
}) => {
const [deleteType, setDeleteType] = useState('soft');
const [remember, setRemember] = useState(false);
useEffect(() => {
if (visible) {
setDeleteType(defaultDeleteType);
setRemember(false);
}
}, [visible, defaultDeleteType]);
const handleOk = () => {
onConfirm(!showOptions || deleteType === 'hard');
if (showOptions) {
onConfirm(deleteType === 'hard', remember);
} else {
onConfirm(true);
}
};
return (
@@ -36,32 +49,37 @@ const ListingDeletionModal = ({
<Text>{message}</Text>
</div>
{showOptions && (
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Mark as deleted (Soft Delete)</Text>
<br />
<Text type="secondary">
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
scraping session.
</Text>
</div>
</Radio>
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Remove from database (Hard Delete)</Text>
<br />
<Text type="secondary">
Listings are completely removed from the database.
<>
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Mark as deleted (Soft Delete)</Text>
<br />
<Text type="warning">
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
previously found.
<Text type="secondary">
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
scraping session.
</Text>
</Text>
</div>
</Radio>
</RadioGroup>
</div>
</Radio>
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Remove from database (Hard Delete)</Text>
<br />
<Text type="secondary">
Listings are completely removed from the database.
<br />
<Text type="warning">
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
previously found.
</Text>
</Text>
</div>
</Radio>
</RadioGroup>
<Checkbox checked={remember} onChange={(e) => setRemember(e.target.checked)} style={{ marginTop: 16 }}>
Remember my choice and skip this dialog next time
</Checkbox>
</>
)}
</Modal>
);

View File

@@ -60,6 +60,8 @@ const JobGrid = () => {
const userSettings = useSelector((state) => state.userSettings.settings);
const viewMode = userSettings?.jobs_view_mode ?? 'grid';
const listingDeletionPref = userSettings?.listing_deletion_preference;
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
const [page, setPage] = useState(1);
const pageSize = 12;
@@ -142,13 +144,21 @@ const JobGrid = () => {
};
const onListingRemoval = (jobId) => {
setPendingDeletion({ type: 'listings', jobId });
const deletion = { type: 'listings', jobId };
if (listingDeletionPref?.skipPrompt) {
confirmDeletion(listingDeletionPref.hardDelete, false, deletion);
return;
}
setPendingDeletion(deletion);
setDeleteModalVisible(true);
};
const confirmDeletion = async (hardDelete) => {
const { type, jobId } = pendingDeletion;
const confirmDeletion = async (hardDelete, remember, deletion = pendingDeletion) => {
const { type, jobId } = deletion;
try {
if (remember && type === 'listings') {
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
}
if (type === 'job') {
await xhrDelete('/api/jobs', { jobId });
Toast.success('Job and listings successfully removed');
@@ -174,7 +184,7 @@ const JobGrid = () => {
Toast.success('Job status successfully changed');
loadData();
} catch (error) {
Toast.error(error);
Toast.error(error.error);
}
};
@@ -286,7 +296,7 @@ const JobGrid = () => {
</div>
<div style={{ display: 'flex', alignItems: 'center', gap: 6, flexShrink: 0 }}>
{job.isOnlyShared && (
<Popover content={getPopoverContent('This job has been shared with you read only.')}>
<Popover content={getPopoverContent('This job has been shared with you - read only.')}>
<div>
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
</div>
@@ -425,6 +435,7 @@ const JobGrid = () => {
visible={deleteModalVisible}
title={pendingDeletion?.type === 'job' ? 'Delete Job' : 'Delete Listings'}
showOptions={pendingDeletion?.type !== 'job'}
defaultDeleteType={defaultDeleteType}
message={
pendingDeletion?.type === 'job'
? 'Are you sure you want to delete this job? All associated listings will be removed from the database.'

View File

@@ -16,13 +16,14 @@ import {
} from '@douyinfe/semi-icons';
import no_image from '../../../assets/no_image.png';
import * as timeService from '../../../services/time/timeService.js';
import StatusControl from '../../listings/StatusControl.jsx';
import './ListingsGrid.less';
/**
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function }} props
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function, onStatusChange: Function }} props
*/
const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete }) => (
const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete, onStatusChange }) => (
<div className="listingsGrid__grid">
{listings.map((item) => (
<div
@@ -49,14 +50,16 @@ const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete }) => (
<span>Inactive</span>
</div>
)}
<button
type="button"
className="listingsGrid__card__star"
onClick={(e) => onWatch(e, item)}
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
>
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
</button>
<Tooltip content={item.isWatched === 1 ? 'Remove from Watchlist' : 'Add to Watchlist'}>
<button
type="button"
className="listingsGrid__card__star"
onClick={(e) => onWatch(e, item)}
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
>
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
</button>
</Tooltip>
</div>
<div className="listingsGrid__card__body">
@@ -83,6 +86,12 @@ const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete }) => (
</div>
<div className="listingsGrid__card__actions" onClick={(e) => e.stopPropagation()}>
<StatusControl
status={item.status?.status ?? null}
compact
onChange={(next) => onStatusChange?.(item, next)}
onTriggerClick={(e) => e.stopPropagation()}
/>
<Tooltip content="Original Listing">
<Button
size="small"

View File

@@ -11,12 +11,11 @@
border: 1px solid @color-border !important;
border-radius: @radius-card !important;
overflow: hidden;
transition: transform @transition-card, box-shadow @transition-card;
transition: box-shadow @transition-card;
display: flex;
flex-direction: column;
&:hover {
transform: translateY(-2px);
box-shadow: 0 6px 24px -4px rgba(0,0,0,0.6);
}

View File

@@ -23,7 +23,8 @@ import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-i
import './ListingsOverview.less';
const ListingsOverview = () => {
const ListingsOverview = ({ mode = 'all' }) => {
const isWatchlistMode = mode === 'watchlist';
const listingsData = useSelector((state) => state.listingsData);
const providers = useSelector((state) => state.provider);
const jobs = useSelector((state) => state.jobsData.jobs);
@@ -33,6 +34,8 @@ const ListingsOverview = () => {
const sp = useSearchParams();
const viewMode = userSettings?.listings_view_mode ?? 'grid';
const listingDeletionPref = userSettings?.listing_deletion_preference;
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
const pageSize = 40;
@@ -44,9 +47,13 @@ const ListingsOverview = () => {
const [jobNameFilter, setJobNameFilter] = useSearchParamState(sp, 'job', null, parseString);
const [activityFilter, setActivityFilter] = useSearchParamState(sp, 'active', null, parseNullableBoolean);
const [providerFilter, setProviderFilter] = useSearchParamState(sp, 'provider', null, parseString);
const [statusFilter, setStatusFilter] = useSearchParamState(sp, 'status', null, parseString);
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
const [listingToDelete, setListingToDelete] = useState(null);
// In watchlist mode the watch filter is forced to "watched only" — regardless of the URL.
const effectiveWatchListFilter = isWatchlistMode ? true : watchListFilter;
const loadData = () => {
actions.listingsData.getListingsData({
page,
@@ -54,13 +61,30 @@ const ListingsOverview = () => {
sortfield: sortField,
sortdir: sortDir,
freeTextFilter,
filter: { watchListFilter, jobNameFilter, activityFilter, providerFilter },
filter: {
watchListFilter: effectiveWatchListFilter,
jobNameFilter,
activityFilter,
providerFilter,
statusFilter,
},
});
};
useEffect(() => {
loadData();
}, [page, sortField, sortDir, freeTextFilter, providerFilter, activityFilter, jobNameFilter, watchListFilter]);
}, [
page,
sortField,
sortDir,
freeTextFilter,
providerFilter,
activityFilter,
jobNameFilter,
watchListFilter,
statusFilter,
isWatchlistMode,
]);
const handleFilterChange = useMemo(
() =>
@@ -90,16 +114,34 @@ const ListingsOverview = () => {
}
};
const handleStatusChange = async (item, nextStatus) => {
try {
await actions.listingsData.setListingStatus(item.id, nextStatus);
Toast.success(nextStatus ? `Marked as ${nextStatus}` : 'Status cleared');
loadData();
} catch (e) {
console.error(e);
Toast.error('Failed to update status');
}
};
const handleDelete = (id) => {
if (listingDeletionPref?.skipPrompt) {
confirmDeletion(listingDeletionPref.hardDelete, false, id);
return;
}
setListingToDelete(id);
setDeleteModalVisible(true);
};
const handleNavigate = (id) => navigate(`/listings/listing/${id}`);
const confirmDeletion = async (hardDelete) => {
const confirmDeletion = async (hardDelete, remember, id = listingToDelete) => {
try {
await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
if (remember) {
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
}
await xhrDelete('/api/listings/', { ids: [id], hardDelete });
Toast.success('Listing successfully removed');
loadData();
} catch (error) {
@@ -139,20 +181,38 @@ const ListingsOverview = () => {
<Radio value="false">Inactive</Radio>
</RadioGroup>
<RadioGroup
type="button"
buttonSize="middle"
value={watchListFilter === null ? 'all' : String(watchListFilter)}
onChange={(e) => {
const v = e.target.value;
setWatchListFilter(v === 'all' ? null : v === 'true');
{!isWatchlistMode && (
<RadioGroup
type="button"
buttonSize="middle"
value={watchListFilter === null ? 'all' : String(watchListFilter)}
onChange={(e) => {
const v = e.target.value;
setWatchListFilter(v === 'all' ? null : v === 'true');
setPage(1);
}}
>
<Radio value="all">All</Radio>
<Radio value="true">Watched</Radio>
<Radio value="false">Unwatched</Radio>
</RadioGroup>
)}
<Select
placeholder="Status"
showClear
onChange={(val) => {
setStatusFilter(val ?? null);
setPage(1);
}}
value={statusFilter}
style={{ width: 150 }}
>
<Radio value="all">All</Radio>
<Radio value="true">Watched</Radio>
<Radio value="false">Unwatched</Radio>
</RadioGroup>
<Select.Option value="applied">Applied</Select.Option>
<Select.Option value="rejected">Rejected</Select.Option>
<Select.Option value="accepted">Accepted</Select.Option>
<Select.Option value="none">No status</Select.Option>
</Select>
<Select
placeholder="Provider"
@@ -188,7 +248,13 @@ const ListingsOverview = () => {
))}
</Select>
<Select prefix="Sort by" style={{ width: 185 }} value={sortField} onChange={(val) => setSortField(val)}>
<Select
prefix="Sort by"
className="listingsOverview__topbar__sort"
style={{ width: 220 }}
value={sortField}
onChange={(val) => setSortField(val)}
>
<Select.Option value="job_name">Job Name</Select.Option>
<Select.Option value="created_at">Listing Date</Select.Option>
<Select.Option value="price">Price</Select.Option>
@@ -232,9 +298,21 @@ const ListingsOverview = () => {
)}
{viewMode === 'grid' ? (
<ListingsGrid listings={listings} onWatch={handleWatch} onNavigate={handleNavigate} onDelete={handleDelete} />
<ListingsGrid
listings={listings}
onWatch={handleWatch}
onNavigate={handleNavigate}
onDelete={handleDelete}
onStatusChange={handleStatusChange}
/>
) : (
<ListingsTable listings={listings} onWatch={handleWatch} onNavigate={handleNavigate} onDelete={handleDelete} />
<ListingsTable
listings={listings}
onWatch={handleWatch}
onNavigate={handleNavigate}
onDelete={handleDelete}
onStatusChange={handleStatusChange}
/>
)}
{listings.length > 0 && (
@@ -251,6 +329,7 @@ const ListingsOverview = () => {
<ListingDeletionModal
visible={deleteModalVisible}
defaultDeleteType={defaultDeleteType}
onConfirm={confirmDeletion}
onCancel={() => {
setDeleteModalVisible(false);

View File

@@ -19,6 +19,14 @@
flex-shrink: 0;
}
&__sort {
flex-shrink: 0;
.semi-select-prefix {
white-space: nowrap;
}
}
@media (max-width: 768px) {
.listingsOverview__topbar__search {
width: 100%;

View File

@@ -0,0 +1,115 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { useState } from 'react';
import { Dropdown, Button, Tooltip } from '@douyinfe/semi-ui-19';
import { IconChevronDown } from '@douyinfe/semi-icons';
import './StatusControl.less';
const STATUS_TOOLTIP =
'Track where you stand with this listing: Applied once you have reached out, Rejected if it did not work out, or Accepted if you got it.';
/**
* @typedef {('applied'|'rejected'|'accepted'|null)} ListingStatus
*/
const STATUS_OPTIONS = [
{ value: null, label: 'None' },
{ value: 'applied', label: 'Applied' },
{ value: 'rejected', label: 'Rejected' },
{ value: 'accepted', label: 'Accepted' },
];
/**
* Look up the option metadata for a status value.
* @param {ListingStatus} status
*/
const optionFor = (status) => STATUS_OPTIONS.find((o) => o.value === status) ?? STATUS_OPTIONS[0];
/**
* Shared control for setting a listing's user-decision status
* (Applied / Rejected / Accepted).
*
* Both compact (table/grid rows) and full (listing detail header) modes
* render a Button that picks up the project's CI tokens via the
* .status-btn classes, with a small size variant for compact contexts.
*
* @param {Object} props
* @param {ListingStatus} props.status - The current status value.
* @param {(next: ListingStatus) => void} props.onChange - Called with the new status when the user picks one.
* @param {boolean} [props.compact=false] - When true, renders smaller for table/grid rows; full size otherwise.
* @param {(e: React.MouseEvent) => void} [props.onTriggerClick] - Optional click handler to stop propagation on the trigger.
*/
export default function StatusControl({ status = null, onChange, compact = false, onTriggerClick }) {
const [open, setOpen] = useState(false);
const [tooltipOpen, setTooltipOpen] = useState(false);
const current = optionFor(status);
const handlePick = (next) => {
setOpen(false);
if (next === status) return;
onChange?.(next);
};
const menu = (
<Dropdown.Menu>
{STATUS_OPTIONS.map((opt) => (
<Dropdown.Item
key={opt.value ?? '__none__'}
active={opt.value === status}
onClick={() => handlePick(opt.value)}
>
{opt.label}
</Dropdown.Item>
))}
</Dropdown.Menu>
);
const className = ['status-btn', compact ? 'status-btn--compact' : null, status ? `status-btn--${status}` : null]
.filter(Boolean)
.join(' ');
const trigger = (
<Tooltip
content={STATUS_TOOLTIP}
position="top"
trigger="custom"
visible={tooltipOpen && !open}
onVisibleChange={setTooltipOpen}
>
<Button
size={compact ? 'small' : 'default'}
theme="borderless"
icon={<IconChevronDown />}
iconPosition="right"
onMouseEnter={() => setTooltipOpen(true)}
onMouseLeave={() => setTooltipOpen(false)}
onClick={(e) => {
onTriggerClick?.(e);
setTooltipOpen(false);
setOpen((o) => !o);
}}
className={className}
>
{status ? current.label : 'Status'}
</Button>
</Tooltip>
);
return (
<Dropdown
trigger="custom"
visible={open}
onVisibleChange={setOpen}
onClickOutSide={() => setOpen(false)}
position="bottom"
render={menu}
stopPropagation
>
<span className="status-btn__anchor">{trigger}</span>
</Dropdown>
);
}

View File

@@ -0,0 +1,64 @@
@import '../../tokens.less';
// Wrapper span used as the Dropdown's positioning anchor so the menu opens
// directly below the visible button rather than the implicit wrapper of the
// hover tooltip (which can have a different bounding box).
.status-btn__anchor {
display: inline-block;
line-height: 0;
}
// StatusControl shared base. Matches dimensions and border treatment
// of the surrounding Watched / Open listing / Delete buttons in the
// detail view, and shrinks via the --compact modifier for table rows
// and grid cards.
.status-btn {
color: @color-muted !important;
border: 1px solid @color-border-bright !important;
border-radius: @radius-btn !important;
background: transparent !important;
transition: color @transition-fast, border-color @transition-fast, background @transition-fast;
&:hover {
color: @color-text !important;
background: rgba(255, 255, 255, 0.06) !important;
}
&--compact {
height: 24px !important;
padding: 0 8px !important;
font-size: @text-sm !important;
border-radius: @radius-chip !important;
.semi-icon {
font-size: 12px !important;
}
}
&--applied {
color: @color-info !important;
border-color: rgba(96, 165, 250, 0.4) !important;
background: rgba(96, 165, 250, 0.08) !important;
&:hover {
background: rgba(96, 165, 250, 0.14) !important;
}
}
&--rejected {
color: @color-error !important;
border-color: rgba(251, 113, 133, 0.4) !important;
background: rgba(251, 113, 133, 0.08) !important;
&:hover {
background: rgba(251, 113, 133, 0.14) !important;
}
}
&--accepted {
color: @color-success !important;
border-color: rgba(52, 211, 153, 0.4) !important;
background: rgba(52, 211, 153, 0.08) !important;
&:hover {
background: rgba(52, 211, 153, 0.14) !important;
}
}
}

View File

@@ -8,6 +8,7 @@ import maplibregl from 'maplibre-gl';
import 'maplibre-gl/dist/maplibre-gl.css';
import '@mapbox/mapbox-gl-draw/dist/mapbox-gl-draw.css';
import { fixMapboxDrawCompatibility, addDrawingControl, setupAreaFilterEventListeners } from './MapDrawingExtension.js';
import { getBoundsFromCoords } from '../../views/listings/mapUtils.js';
import './Map.less';
export const GERMANY_BOUNDS = [
@@ -66,6 +67,7 @@ export default function Map({
const mapContainerRef = useRef(null);
const mapRef = useRef(null);
const drawRef = useRef(null);
const hasFittedToInitialAreaRef = useRef(false);
// Initialize map - ONLY when container changes, never reinitialize
useEffect(() => {
@@ -128,6 +130,17 @@ export default function Map({
} catch (error) {
console.error('Error loading spatial filter:', error);
}
if (!hasFittedToInitialAreaRef.current) {
const coords = initialSpatialFilter.features.flatMap((feature) =>
feature.geometry?.type === 'Polygon' ? feature.geometry.coordinates.flat() : [],
);
const bounds = getBoundsFromCoords(coords);
if (bounds) {
mapRef.current.fitBounds(bounds, { padding: 50, maxZoom: 15, duration: 0 });
hasFittedToInitialAreaRef.current = true;
}
}
}
// Setup drawing event listeners

View File

@@ -112,9 +112,9 @@
.semi-navigation-item {
border-radius: @radius-btn !important;
border: 1px solid transparent !important;
color: @color-muted !important;
transition: background @transition-fast, color @transition-fast !important;
margin: 2px 8px !important;
transition: background @transition-fast, color @transition-fast, border-color @transition-fast !important;
&:hover {
color: @color-text !important;
@@ -123,7 +123,7 @@
&.semi-navigation-item-selected,
&[aria-selected="true"] {
background: rgba(224,74,56,0.12) !important;
border: 1px solid rgba(224,74,56,0.25) !important;
border-color: rgba(224,74,56,0.25) !important;
color: @color-text !important;
.semi-navigation-item-icon {

View File

@@ -37,6 +37,7 @@ export default function Navigation({ isAdmin }) {
items: [
{ itemKey: '/listings', text: 'Overview' },
{ itemKey: '/map', text: 'Map View' },
{ itemKey: '/listings/watchlist', text: 'Watchlist' },
],
},
];
@@ -61,6 +62,22 @@ export default function Navigation({ isAdmin }) {
}
function parsePathName(name) {
// Collect every leaf itemKey that looks like a route (starts with '/').
// Prefer the longest exact-prefix match so nested routes like
// '/listings/watchlist' resolve to themselves instead of being collapsed
// to '/listings'.
const allKeys = [];
const collect = (nodes) => {
for (const n of nodes) {
if (typeof n.itemKey === 'string' && n.itemKey.startsWith('/')) allKeys.push(n.itemKey);
if (Array.isArray(n.items)) collect(n.items);
}
};
collect(items);
const longestMatch = allKeys
.filter((k) => name === k || name.startsWith(k + '/'))
.sort((a, b) => b.length - a.length)[0];
if (longestMatch) return longestMatch;
const split = name.split('/').filter((s) => s.length !== 0);
return '/' + split[0];
}

View File

@@ -63,7 +63,7 @@ const JobsTable = ({ jobs, onRun, onEdit, onClone, onDeleteListings, onDeleteJob
</Tag>
)}
{job.isOnlyShared && (
<Tooltip content="Shared with you read only">
<Tooltip content="Shared with you - read only">
<span style={{ display: 'flex', alignItems: 'center' }}>
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
</span>

View File

@@ -6,7 +6,6 @@
import { Button, Tooltip } from '@douyinfe/semi-ui-19';
import {
IconBriefcase,
IconCart,
IconDelete,
IconLink,
IconMapPin,
@@ -15,14 +14,16 @@ import {
IconEyeOpened,
} from '@douyinfe/semi-icons';
import no_image from '../../assets/no_image.png';
import { formatEuroPrice } from '../../services/price/priceService.js';
import * as timeService from '../../services/time/timeService.js';
import StatusControl from '../listings/StatusControl.jsx';
import './ListingsTable.less';
/**
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function }} props
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function, onStatusChange: Function }} props
*/
const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
const ListingsTable = ({ listings, onWatch, onNavigate, onDelete, onStatusChange }) => (
<div className="listingsTable">
{listings.map((item) => (
<div
@@ -51,12 +52,9 @@ const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
<div className="listingsTable__row__price">
{item.price ? (
<>
<IconCart size="small" />
{item.price}
</>
formatEuroPrice(item.price)
) : (
<span className="listingsTable__row__empty"></span>
<span className="listingsTable__row__empty">---</span>
)}
</div>
@@ -67,7 +65,7 @@ const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
{item.address}
</>
) : (
<span className="listingsTable__row__empty"></span>
<span className="listingsTable__row__empty">---</span>
)}
</div>
@@ -79,14 +77,22 @@ const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
<div className="listingsTable__row__date">{timeService.format(item.created_at, false)}</div>
<div className="listingsTable__row__actions" onClick={(e) => e.stopPropagation()}>
<button
type="button"
className="listingsTable__row__star"
onClick={(e) => onWatch(e, item)}
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
>
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
</button>
<StatusControl
status={item.status?.status ?? null}
compact
onChange={(next) => onStatusChange?.(item, next)}
onTriggerClick={(e) => e.stopPropagation()}
/>
<Tooltip content={item.isWatched === 1 ? 'Remove from Watchlist' : 'Add to Watchlist'}>
<button
type="button"
className="listingsTable__row__star"
onClick={(e) => onWatch(e, item)}
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
>
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
</button>
</Tooltip>
<Tooltip content="Original Listing">
<Button
size="small"

View File

@@ -55,8 +55,11 @@
color: @color-success;
display: flex;
align-items: center;
justify-content: flex-end;
gap: 4px;
white-space: nowrap;
text-align: right;
font-variant-numeric: tabular-nums;
}
&__address {
@@ -94,26 +97,33 @@
}
&__star {
width: 28px;
height: 28px;
background: transparent;
border: none;
width: 30px;
height: 30px;
background: rgba(0,0,0,0.28);
border: 1px solid rgba(255,255,255,0.12);
border-radius: 50%;
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
padding: 0;
transition: background @transition-fast;
transition: background @transition-fast, border-color @transition-fast, transform @transition-fast;
flex-shrink: 0;
&:hover {
background: rgba(0, 0, 0, 0.1);
background: rgba(0,0,0,0.48);
border-color: rgba(255,255,255,0.22);
transform: translateY(-1px);
}
&:focus-visible {
outline: none;
box-shadow: 0 0 0 2px rgba(224,74,56,0.35);
}
svg {
color: @color-accent;
font-size: 14px;
font-size: 15px;
}
}

View File

@@ -1,5 +1,5 @@
.versionBanner {
margin-bottom: 0 !important;
margin-bottom: 16px;
.semi-banner-body {
padding: 6px 16px;

View File

@@ -0,0 +1,22 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
const euroPriceFormatter = new Intl.NumberFormat('de-DE', {
minimumFractionDigits: 2,
maximumFractionDigits: 2,
});
/**
* @param {number|string} price
* @returns {string}
*/
export const formatEuroPrice = (price) => {
const parsedPrice = Number(price);
if (!Number.isFinite(parsedPrice)) {
return `${price}`;
}
return `${euroPriceFormatter.format(parsedPrice)}`;
};

View File

@@ -260,6 +260,22 @@ export const useFredyState = create(
console.error('Error while trying to get resource for api/listings/map. Error:', Exception);
}
},
async setListingStatus(listingId, status) {
try {
await xhrPost(`/api/listings/${listingId}/status`, { status });
} catch (Exception) {
console.error(`Error while trying to set status for listing ${listingId}. Error:`, Exception);
throw Exception;
}
},
async setListingNotes(listingId, notes) {
try {
await xhrPost(`/api/listings/${listingId}/notes`, { notes });
} catch (Exception) {
console.error(`Error while trying to set notes for listing ${listingId}. Error:`, Exception);
throw Exception;
}
},
},
userSettings: {
async getUserSettings() {
@@ -349,6 +365,20 @@ export const useFredyState = create(
throw Exception;
}
},
async setListingDeletionPreference(listing_deletion_preference) {
try {
await xhrPost('/api/user/settings/listing-deletion-preference', { listing_deletion_preference });
set((state) => ({
userSettings: {
...state.userSettings,
settings: { ...state.userSettings.settings, listing_deletion_preference },
},
}));
} catch (Exception) {
console.error('Error while trying to update listing deletion preference. Error:', Exception);
throw Exception;
}
},
},
};

View File

@@ -18,6 +18,9 @@ import {
AutoComplete,
Select,
Banner,
Radio,
RadioGroup,
Typography,
} from '@douyinfe/semi-ui-19';
import { InputNumber } from '@douyinfe/semi-ui-19';
import { xhrPost, xhrGet } from '../../services/xhr';
@@ -33,6 +36,8 @@ import { debounce } from '../../utils';
import Headline from '../../components/headline/Headline.jsx';
import './GeneralSettings.less';
const { Text } = Typography;
function formatFromTimestamp(ts) {
const date = new Date(ts);
return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
@@ -54,8 +59,10 @@ const GeneralSettings = function GeneralSettings() {
const [loading, setLoading] = React.useState(true);
const settings = useSelector((state) => state.generalSettings.settings);
const currentUser = useSelector((state) => state.user.currentUser);
const [interval, setInterval] = React.useState('');
const [proxyUrl, setProxyUrl] = React.useState('');
const [port, setPort] = React.useState('');
const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
const [workingHourTo, setWorkingHourTo] = React.useState(null);
@@ -72,9 +79,12 @@ const GeneralSettings = function GeneralSettings() {
// User settings state
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
const providerDetails = useSelector((state) => state.userSettings.settings.provider_details);
const listingDeletionPreference = useSelector((state) => state.userSettings.settings.listing_deletion_preference);
const allProviders = useSelector((state) => state.provider);
const [address, setAddress] = useState(homeAddress?.address || '');
const [coords, setCoords] = useState(homeAddress?.coords || null);
const [listingDeleteHard, setListingDeleteHard] = useState(false);
const [listingDeleteSkipPrompt, setListingDeleteSkipPrompt] = useState(false);
const saving = useIsLoading(actions.userSettings.setHomeAddress);
const [dataSource, setDataSource] = useState([]);
@@ -90,6 +100,7 @@ const GeneralSettings = function GeneralSettings() {
React.useEffect(() => {
async function init() {
setInterval(settings?.interval);
setProxyUrl(settings?.proxyUrl ?? '');
setPort(settings?.port);
setWorkingHourFrom(settings?.workingHours?.from);
setWorkingHourTo(settings?.workingHours?.to);
@@ -107,6 +118,11 @@ const GeneralSettings = function GeneralSettings() {
setCoords(homeAddress?.coords || null);
}, [homeAddress]);
useEffect(() => {
setListingDeleteHard(listingDeletionPreference?.hardDelete ?? false);
setListingDeleteSkipPrompt(listingDeletionPreference?.skipPrompt ?? false);
}, [listingDeletionPreference]);
const nullOrEmpty = (val) => val == null || val.length === 0;
const handleStore = async () => {
@@ -132,6 +148,7 @@ const GeneralSettings = function GeneralSettings() {
try {
await xhrPost('/api/admin/generalSettings', {
interval,
proxyUrl: proxyUrl?.trim() ?? '',
port,
workingHours: {
from: workingHourFrom,
@@ -214,6 +231,10 @@ const GeneralSettings = function GeneralSettings() {
try {
const responseJson = await actions.userSettings.setHomeAddress(address);
setCoords(responseJson.coords);
await actions.userSettings.setListingDeletionPreference({
skipPrompt: listingDeleteSkipPrompt,
hardDelete: listingDeleteHard,
});
await actions.userSettings.getUserSettings();
Toast.success('Settings saved. Distance calculations are running in the background.');
} catch (error) {
@@ -375,6 +396,18 @@ const GeneralSettings = function GeneralSettings() {
</div>
</SegmentPart>
<SegmentPart
name="Proxy URL"
helpText="Optional. Routes the scraping browser through a proxy. Server/datacenter IPs are frequently blocked by providers (e.g. immowelt) regardless of browser fingerprint, a German residential proxy makes requests look like a normal household and is the most effective fix. Format: http://user:pass@host:port or socks5://user:pass@host:port. Leave empty to disable."
>
<Input
type="text"
placeholder="http://user:pass@host:port"
value={proxyUrl}
onChange={(value) => setProxyUrl(value)}
/>
</SegmentPart>
<div className="generalSettings__save-row">
<Button type="primary" theme="solid" onClick={handleStore} icon={<IconSave />}>
Save
@@ -443,6 +476,48 @@ const GeneralSettings = function GeneralSettings() {
/>
</SegmentPart>
<SegmentPart
name="Listing deletion"
helpText="Choose the default deletion mode. Soft delete hides them without re-scraping; hard delete removes them from the database."
>
<RadioGroup
value={listingDeleteHard ? 'hard' : 'soft'}
onChange={(e) => setListingDeleteHard(e.target.value === 'hard')}
>
<Radio value="soft">
<div>
<Text strong>Mark as deleted (Soft Delete)</Text>
<br />
<Text type="secondary">
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during
the next scraping session.
</Text>
</div>
</Radio>
<Radio value="hard">
<div>
<Text strong>Remove from database (Hard Delete)</Text>
<br />
<Text type="secondary">
Listings are completely removed from the database.
<br />
<Text type="warning">
Consequence: They might re-appear when scraping the next time because Fredy won't know they
were previously found.
</Text>
</Text>
</div>
</Radio>
</RadioGroup>
<Checkbox
checked={listingDeleteSkipPrompt}
onChange={(e) => setListingDeleteSkipPrompt(e.target.checked)}
style={{ marginTop: 12 }}
>
Skip confirmation dialog
</Checkbox>
</SegmentPart>
<div className="generalSettings__save-row">
<Button
icon={<IconSave />}
@@ -467,12 +542,26 @@ const GeneralSettings = function GeneralSettings() {
itemKey="backup"
>
<div className="generalSettings__tab-content">
{demoMode && !currentUser?.isAdmin && (
<Banner
fullMode={false}
type="warning"
closeIcon={null}
style={{ marginBottom: '12px' }}
description="Backup and restore are not available in demo mode."
/>
)}
<SegmentPart
name="Backup & Restore"
helpText="Download a zipped backup of your database or restore from a backup zip."
>
<div style={{ display: 'flex', gap: '0.5rem', alignItems: 'center', flexWrap: 'wrap' }}>
<Button theme="solid" icon={<IconSave />} onClick={handleDownloadBackup}>
<Button
theme="solid"
icon={<IconSave />}
onClick={handleDownloadBackup}
disabled={demoMode && !currentUser?.isAdmin}
>
Download Backup
</Button>
<input
@@ -482,7 +571,12 @@ const GeneralSettings = function GeneralSettings() {
style={{ display: 'none' }}
onChange={handleSelectRestoreFile}
/>
<Button onClick={handleOpenFilePicker} theme="light" icon={<IconFolder />}>
<Button
onClick={handleOpenFilePicker}
theme="light"
icon={<IconFolder />}
disabled={demoMode && !currentUser?.isAdmin}
>
Restore from Zip
</Button>
</div>

View File

@@ -73,7 +73,9 @@ export default function NotificationAdapterMutator({
const adapter = useSelector((state) => state.notificationAdapter);
const preFilledSelectedAdapter =
editNotificationAdapter == null ? null : adapter.find((a) => a.id === editNotificationAdapter.id);
editNotificationAdapter == null
? null
: adapter.filter((a) => a != null).find((a) => a.id === editNotificationAdapter.id);
spreadPrefilledAdapterWithValues(preFilledSelectedAdapter, editNotificationAdapter?.fields);
@@ -227,9 +229,9 @@ export default function NotificationAdapterMutator({
className="providerMutator__fields"
value={selectedAdapter == null ? '' : selectedAdapter.id}
optionList={adapter
.filter((a) => a != null)
.map((a) => {
return {
otherKey: a.id,
value: a.id,
label: a.name,
};
@@ -238,7 +240,7 @@ export default function NotificationAdapterMutator({
.filter((option) =>
editNotificationAdapter != null
? true
: selected.find((selectedOption) => selectedOption.id === option.key) == null,
: selected.find((selectedOption) => selectedOption.id === option.value) == null,
)
.sort(sortAdapter)}
onChange={(value) => {

View File

@@ -20,6 +20,8 @@ import {
Banner,
Spin,
Toast,
TextArea,
Tooltip,
} from '@douyinfe/semi-ui-19';
import {
IconArrowLeft,
@@ -39,11 +41,13 @@ import maplibregl from 'maplibre-gl';
import 'maplibre-gl/dist/maplibre-gl.css';
import no_image from '../../assets/no_image.png';
import * as timeService from '../../services/time/timeService.js';
import { formatEuroPrice } from '../../services/price/priceService.js';
import { distanceMeters, getBoundsFromCoords } from './mapUtils.js';
import { xhrPost, xhrDelete } from '../../services/xhr.js';
import ListingDeletionModal from '../../components/ListingDeletionModal.jsx';
import Headline from '../../components/headline/Headline.jsx';
import StatusControl from '../../components/listings/StatusControl.jsx';
import './ListingDetail.less';
const { Title, Text } = Typography;
@@ -57,11 +61,16 @@ export default function ListingDetail() {
const navigate = useNavigate();
const actions = useActions();
const listing = useSelector((state) => state.listingsData.currentListing);
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
const userSettings = useSelector((state) => state.userSettings.settings);
const homeAddress = userSettings?.home_address;
const listingDeletionPref = userSettings?.listing_deletion_preference;
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
const mapContainer = useRef(null);
const map = useRef(null);
const [loading, setLoading] = useState(true);
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
const [notesDraft, setNotesDraft] = useState('');
const [notesSaving, setNotesSaving] = useState(false);
useEffect(() => {
async function fetchListing() {
@@ -79,6 +88,10 @@ export default function ListingDetail() {
fetchListing();
}, [listingId]);
useEffect(() => {
setNotesDraft(listing?.notes ?? '');
}, [listing?.id, listing?.notes]);
const hasGeo =
listing?.latitude != null && listing?.longitude != null && listing?.latitude !== -1 && listing?.longitude !== -1;
@@ -242,8 +255,11 @@ export default function ListingDetail() {
};
}, [listing, loading, homeAddress]);
const confirmDeletion = async (hardDelete) => {
const confirmDeletion = async (hardDelete, remember) => {
try {
if (remember) {
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
}
await xhrDelete('/api/listings/', { ids: [listing.id], hardDelete });
Toast.success('Listing successfully removed');
navigate('/listings');
@@ -265,6 +281,32 @@ export default function ListingDetail() {
}
};
const handleStatusChange = async (next) => {
try {
await actions.listingsData.setListingStatus(listing.id, next);
await actions.listingsData.getListing(listingId);
Toast.success(next ? `Marked as ${next}` : 'Status cleared');
} catch (e) {
console.error('Failed to update status:', e);
Toast.error('Failed to update status');
}
};
const handleSaveNotes = async () => {
if (!listing) return;
setNotesSaving(true);
try {
await actions.listingsData.setListingNotes(listing.id, notesDraft);
await actions.listingsData.getListing(listingId);
Toast.success('Notes saved');
} catch (e) {
console.error('Failed to save notes:', e);
Toast.error('Failed to save notes');
} finally {
setNotesSaving(false);
}
};
if (loading) {
return (
<div style={{ display: 'flex', justifyContent: 'center', alignItems: 'center', height: '100%' }}>
@@ -275,35 +317,62 @@ export default function ListingDetail() {
if (!listing) return null;
const statusLabel = listing.status?.status
? listing.status.status.charAt(0).toUpperCase() + listing.status.status.slice(1)
: null;
const data = [
{ key: 'Price', value: `${listing.price}`, Icon: <IconCart /> },
{
key: 'Price',
value: listing.price ? (
<span className="listing-detail__price">{formatEuroPrice(listing.price)}</span>
) : (
'N/A'
),
Icon: <IconCart />,
helpText: 'The asking price of this listing, as reported by the provider.',
},
{
key: 'Size',
value: listing.size ? `${listing.size}` : 'N/A',
Icon: <IconExpand />,
helpText: 'Living space of the listing in square meters.',
},
{
key: 'Rooms',
value: listing.rooms ? `${listing.rooms} Rooms` : 'N/A',
Icon: <IconGridView />,
helpText: 'Number of rooms in the listing.',
},
{
key: 'Job',
value: listing.job_name,
Icon: <IconBriefcase />,
helpText: 'The Fredy job that found this listing.',
},
{
key: 'Provider',
value: listing.provider ? listing.provider.charAt(0).toUpperCase() + listing.provider.slice(1) : 'Unknown',
Icon: <IconBriefcase />,
helpText: 'The real estate portal where this listing was scraped from.',
},
{
key: 'Added',
value: timeService.format(listing.created_at),
Icon: <IconClock />,
helpText: 'When Fredy first added this listing to your database.',
},
];
if (statusLabel) {
data.push({
key: 'Status',
value: listing.status?.setAt ? `${statusLabel} (set ${timeService.format(listing.status.setAt)})` : statusLabel,
Icon: <IconActivity />,
helpText: 'The status you marked for this listing and when you set it.',
});
}
return (
<div className="listing-detail">
<Headline
@@ -341,13 +410,20 @@ export default function ListingDetail() {
>
{listing.isWatched === 1 ? 'Watched' : 'Watch'}
</Button>
<StatusControl status={listing.status?.status ?? null} onChange={handleStatusChange} />
<a href={listing.link} target="_blank" rel="noopener noreferrer" className="listing-detail__open-btn">
<IconLink style={{ marginRight: 6 }} />
Open listing
</a>
<Button
icon={<IconDelete />}
onClick={() => setDeleteModalVisible(true)}
onClick={() => {
if (listingDeletionPref?.skipPrompt) {
confirmDeletion(listingDeletionPref.hardDelete);
return;
}
setDeleteModalVisible(true);
}}
theme="light"
type="danger"
>
@@ -368,6 +444,32 @@ export default function ListingDetail() {
preview={!!listing.image_url}
/>
</div>
<div className="listing-detail__notes">
<Title heading={4} className="listing-detail__notes-title">
Notes
</Title>
<TextArea
value={notesDraft}
onChange={(val) => setNotesDraft(val)}
placeholder="Your private notes about this listing…"
rows={5}
autosize={{ minRows: 4, maxRows: 12 }}
className="listing-detail__notes-textarea"
showClear
/>
<Space className="listing-detail__notes-actions">
<Button
theme="solid"
type="primary"
loading={notesSaving}
disabled={notesSaving || (notesDraft ?? '') === (listing.notes ?? '')}
onClick={handleSaveNotes}
>
Store notes
</Button>
</Space>
</div>
</Col>
<Col span={24} lg={12}>
<div className="listing-detail__info-section">
@@ -377,10 +479,12 @@ export default function ListingDetail() {
<Descriptions column={1}>
{data.map((item, index) => (
<Descriptions.Item key={index}>
<Space>
{item.Icon}
{item.value}
</Space>
<Tooltip content={item.helpText} position="left">
<span className="listing-detail__details-item">
{item.Icon}
{item.value}
</span>
</Tooltip>
</Descriptions.Item>
))}
</Descriptions>
@@ -423,6 +527,7 @@ export default function ListingDetail() {
<ListingDeletionModal
visible={deleteModalVisible}
defaultDeleteType={defaultDeleteType}
onConfirm={confirmDeletion}
onCancel={() => setDeleteModalVisible(false)}
/>

View File

@@ -89,6 +89,49 @@
}
}
&__notes {
padding: 1.5rem;
border-top: 1px solid var(--semi-color-border);
}
&__notes-title {
margin-bottom: 1rem !important;
}
&__notes-actions {
margin-top: 0.75rem;
width: 100%;
justify-content: flex-end;
}
&__notes-textarea {
background: #2a2a2a !important;
border: 1px solid @color-border-bright !important;
border-radius: @radius-input !important;
box-shadow: none !important;
outline: none !important;
transition: border-color @transition-fast, background @transition-fast !important;
textarea {
background: transparent !important;
border: 0 !important;
box-shadow: none !important;
outline: none !important;
color: @color-text !important;
font-family: @font-ui !important;
font-size: @text-base !important;
}
&.semi-input-textarea-focus,
&:focus,
&:focus-within {
border-color: @color-accent !important;
box-shadow: none !important;
outline: none !important;
background: #2f2f2f !important;
}
}
&__watch-btn {
color: @color-muted !important;
border: 1px solid @color-border-bright !important;
@@ -128,6 +171,13 @@
padding: 1.5rem;
}
&__details-item {
cursor: help;
display: inline-flex;
align-items: center;
gap: 8px;
}
&__map-container {
height: 400px;
width: 100%;
@@ -149,6 +199,10 @@
font-size: 0.9rem;
padding: 0.2rem 0.6rem;
}
&__price {
font-variant-numeric: tabular-nums;
}
}
.listing-detail-popup {

View File

@@ -6,11 +6,15 @@
import ListingsOverview from '../../components/listings/ListingsOverview.jsx';
import Headline from '../../components/headline/Headline.jsx';
export default function Listings() {
/**
* @param {{ mode?: 'all' | 'watchlist' }} props
*/
export default function Listings({ mode = 'all' }) {
const title = mode === 'watchlist' ? 'Watchlist' : 'Listings';
return (
<>
<Headline text="Listings" />
<ListingsOverview />
<Headline text={title} />
<ListingsOverview mode={mode} />
</>
);
}

View File

@@ -37,7 +37,10 @@ export default function MapView() {
const sp = useSearchParams();
const [searchParams, setSearchParams] = sp;
const listings = useSelector((state) => state.listingsData.mapListings);
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
const userSettings = useSelector((state) => state.userSettings.settings);
const homeAddress = userSettings?.home_address;
const listingDeletionPref = userSettings?.listing_deletion_preference;
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
const jobs = useSelector((state) => state.jobsData.jobs);
const [jobId, setJobId] = useSearchParamState(sp, 'job', null, parseString);
@@ -52,10 +55,14 @@ export default function MapView() {
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
const [listingToDelete, setListingToDelete] = useState(null);
const deleteListingRef = useRef(null);
const confirmListingDeletion = async (hardDelete) => {
const confirmListingDeletion = async (hardDelete, remember, id = listingToDelete) => {
try {
await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
if (remember) {
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
}
await xhrDelete('/api/listings/', { ids: [id], hardDelete });
Toast.success('Listing successfully removed');
fetchListings();
} catch (error) {
@@ -66,6 +73,15 @@ export default function MapView() {
}
};
deleteListingRef.current = (id) => {
if (listingDeletionPref?.skipPrompt) {
confirmListingDeletion(listingDeletionPref.hardDelete, false, id);
return;
}
setListingToDelete(id);
setDeleteModalVisible(true);
};
useEffect(() => {
// Only reset to full range when no URL override is set
if (urlPriceMax === null) {
@@ -88,10 +104,7 @@ export default function MapView() {
};
useEffect(() => {
window.deleteListing = (id) => {
setListingToDelete(id);
setDeleteModalVisible(true);
};
window.deleteListing = (id) => deleteListingRef.current(id);
window.viewDetails = (id) => {
navigate(`/listings/listing/${id}`);
@@ -472,6 +485,7 @@ export default function MapView() {
<ListingDeletionModal
visible={deleteModalVisible}
defaultDeleteType={defaultDeleteType}
onConfirm={confirmListingDeletion}
onCancel={() => {
setDeleteModalVisible(false);

View File

@@ -37,7 +37,7 @@ const Users = function Users() {
await actions.jobsData.getJobs();
await actions.user.getUsers();
} catch (error) {
Toast.error(error);
Toast.error(error.error);
setUserIdToBeRemoved(null);
}
};

2306
yarn.lock

File diff suppressed because it is too large Load Diff