mirror of
https://github.com/orangecoding/fredy.git
synced 2026-06-16 12:31:07 +00:00
Compare commits
38 Commits
21.1.1
...
listing-ma
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9296bcdc86 | ||
|
|
bbebc2a1a2 | ||
|
|
d2978c14db | ||
|
|
5ceac25aa6 | ||
|
|
34b68e1f52 | ||
|
|
696ae451d3 | ||
|
|
317ef79336 | ||
|
|
6428e7ad78 | ||
|
|
2bcec04d55 | ||
|
|
ee2112a24d | ||
|
|
5a54448288 | ||
|
|
f1b8709ab7 | ||
|
|
b56e13aa16 | ||
|
|
a834abc31c | ||
|
|
573868eccb | ||
|
|
1a210d7c1c | ||
|
|
996b841cfb | ||
|
|
b2e294e38c | ||
|
|
8afeaa05d9 | ||
|
|
ec47137b89 | ||
|
|
33161de087 | ||
|
|
acab23207e | ||
|
|
2896d531e4 | ||
|
|
0cbfa25062 | ||
|
|
bcd3042026 | ||
|
|
0ce93acaf6 | ||
|
|
cabef973a2 | ||
|
|
3d0fa87d19 | ||
|
|
8b012ef2f1 | ||
|
|
6816b0aded | ||
|
|
ac02817d4e | ||
|
|
fe0a09fe1c | ||
|
|
2f00966f27 | ||
|
|
921057252d | ||
|
|
703c602527 | ||
|
|
0e29c9b9c6 | ||
|
|
f60c5859f9 | ||
|
|
ee54cc495b |
13
Dockerfile
13
Dockerfile
@@ -1,16 +1,15 @@
|
||||
FROM node:22-slim
|
||||
|
||||
ARG TARGETARCH
|
||||
|
||||
# System deps for Chrome for Testing + build tools for native modules (better-sqlite3)
|
||||
# On ARM64 we also install system Chromium (Chrome for Testing has no ARM64 binary)
|
||||
# System deps for CloakBrowser + build tools for native modules (better-sqlite3)
|
||||
# fonts-noto-color-emoji and fonts-freefont-ttf are required so canvas fingerprint
|
||||
# hashes match real browsers; missing emoji fonts cause bot detection on Kasada/Akamai.
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl ca-certificates fonts-liberation libasound2 \
|
||||
libatk-bridge2.0-0 libatk1.0-0 libcups2 libdbus-1-3 \
|
||||
libdrm2 libgbm1 libgtk-3-0 libnspr4 libnss3 \
|
||||
libx11-xcb1 libxcomposite1 libxdamage1 libxrandr2 xdg-utils \
|
||||
fonts-noto-color-emoji fonts-freefont-ttf \
|
||||
python3 make g++ \
|
||||
&& if [ "$TARGETARCH" = "arm64" ]; then apt-get install -y --no-install-recommends chromium; fi \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& mkdir -p /db /conf /fredy
|
||||
|
||||
@@ -26,8 +25,8 @@ RUN yarn config set network-timeout 600000 \
|
||||
&& yarn --frozen-lockfile \
|
||||
&& yarn cache clean
|
||||
|
||||
# on arm64 use the system Chromium installed above
|
||||
RUN if [ "$TARGETARCH" != "arm64" ]; then npx puppeteer browsers install chrome; fi
|
||||
# Pre-download the CloakBrowser stealth Chromium binary (supports x86_64 and arm64)
|
||||
RUN node -e "import('cloakbrowser').then(({ensureBinary}) => ensureBinary())"
|
||||
|
||||
# Purge build tools now that native modules are compiled
|
||||
RUN apt-get purge -y python3 make g++ \
|
||||
|
||||
36
README.md
36
README.md
@@ -23,7 +23,7 @@
|
||||
|
||||
|
||||
|
||||
# Fredy 🏡 – Your Self-Hosted Real Estate Finder for Germany
|
||||
# Fredy 🏡 - Your Self-Hosted Real Estate Finder for Germany
|
||||
|
||||
Finding an apartment or house in Germany can be stressful and
|
||||
time-consuming.\
|
||||
@@ -167,6 +167,40 @@ For more information on how to set it up and use it, please refer to the [MCP Re
|
||||
|
||||
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
|
||||
|
||||
## 🛡️ Bot Detection & Proxies
|
||||
|
||||
Most browser-based providers (immowelt, immonet, kleinanzeigen, ...) are scraped through a hardened headless browser ([CloakBrowser](https://www.npmjs.com/package/cloakbrowser)). It makes the **browser fingerprint** indistinguishable from a real Chrome, which is enough when you run Fredy on a normal home connection.
|
||||
|
||||
On a **server / VPS the requests usually originate from a datacenter IP**, and providers behind anti-bot systems (e.g. AWS CloudFront/WAF) block those based on **IP reputation alone**, no matter how perfect the fingerprint is. The typical symptom: it works locally but you get `We have been detected as a bot :-/` on the server.
|
||||
|
||||
### The fix: a residential proxy
|
||||
|
||||
A **residential proxy** routes Fredy's browser through the internet connection of a real household, so the provider sees a "normal user" IP instead of a datacenter. For German portals, use a **German (DE) residential** (or mobile/4G) proxy. Plain VPNs and **datacenter proxies do not help** here, they share the same bad reputation as your server.
|
||||
|
||||
**Configure it** under **Settings → Execution → Proxy URL**. Supported formats:
|
||||
|
||||
```
|
||||
http://user:pass@host:port
|
||||
socks5://user:pass@host:port
|
||||
```
|
||||
|
||||
Leave the field empty to disable. The proxy applies to all headless-browser providers and takes effect on the next job run (no restart needed). Immoscout uses a separate mobile API and is not affected.
|
||||
|
||||
### Where to get a residential proxy
|
||||
|
||||
Residential proxies are a paid service (usually billed per GB, Fredy's traffic is small). Well-known providers offering German residential IPs include:
|
||||
|
||||
| Provider | Notes |
|
||||
|---|---|
|
||||
| [IPRoyal](https://iproyal.com) | Pay-as-you-go, no monthly minimum, good for low volume |
|
||||
| [Webshare](https://www.webshare.io) | Cheap entry tier, has a small free plan to test with |
|
||||
| [Decodo (formerly Smartproxy)](https://decodo.com) | Easy setup, country/city targeting |
|
||||
| [SOAX](https://soax.com) | Residential + mobile, fine-grained geo-targeting |
|
||||
| [Bright Data](https://brightdata.com) | Largest pool, most features, higher complexity/price |
|
||||
| [Oxylabs](https://oxylabs.io) | Enterprise-grade, larger plans |
|
||||
|
||||
This is not an endorsement, pick whatever fits your budget. For low-volume use like Fredy, a pay-as-you-go plan (e.g. IPRoyal) or a cheap entry tier (e.g. Webshare) is usually plenty. Make sure to select **Germany** as the proxy location and keep the search interval reasonable (the higher the interval, the less you look like a bot).
|
||||
|
||||
## Analytics
|
||||
|
||||
Fredy is completely free (and will always remain free). However, it would be a huge help if you’d allow me to collect some analytical data.
|
||||
|
||||
@@ -11,6 +11,8 @@
|
||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||
|
||||
<title>Fredy || Real Estate Finder</title>
|
||||
<link rel="icon" type="image/png" href="/ui/src/assets/heart.png" />
|
||||
<link rel="apple-touch-icon" href="/ui/src/assets/heart.png" />
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link href="https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
|
||||
|
||||
9
index.js
9
index.js
@@ -15,6 +15,15 @@ import { initGeocodingCron } from './lib/services/crons/geocoding-cron.js';
|
||||
import { getSettings } from './lib/services/storage/settingsStorage.js';
|
||||
import SqliteConnection, { computeDbPath } from './lib/services/storage/SqliteConnection.js';
|
||||
import { initJobExecutionService } from './lib/services/jobs/jobExecutionService.js';
|
||||
import { ensureValidBinary } from './lib/services/ensureValidBinary.js';
|
||||
|
||||
// Ensure the CloakBrowser stealth Chromium binary is present and complete before
|
||||
// jobs run. ensureValidBinary() also detects and auto-heals partial extractions
|
||||
// (e.g. a newer version that was downloaded but only the chrome executable was
|
||||
// written) so Chrome never crashes with "Invalid file descriptor to ICU data".
|
||||
logger.info('Checking CloakBrowser binary...');
|
||||
await ensureValidBinary();
|
||||
logger.info('CloakBrowser binary ready.');
|
||||
|
||||
//in the config, we store the path of the sqlite file, thus we must check if it is available
|
||||
const isConfigAccessible = await checkIfConfigIsAccessible();
|
||||
|
||||
@@ -5,9 +5,10 @@
|
||||
|
||||
import { NoNewListingsWarning } from './errors.js';
|
||||
import {
|
||||
storeListings,
|
||||
getKnownListingHashesForJobAndProvider,
|
||||
deleteListingsById,
|
||||
getKnownListingHashesForJobAndProvider,
|
||||
storeListings,
|
||||
updateListingDistance,
|
||||
} from './services/storage/listingsStorage.js';
|
||||
import { getJob } from './services/storage/jobStorage.js';
|
||||
import * as notify from './notification/notify.js';
|
||||
@@ -16,8 +17,7 @@ import urlModifier from './services/queryStringMutator.js';
|
||||
import logger from './services/logger.js';
|
||||
import { geocodeAddress } from './services/geocoding/geoCodingService.js';
|
||||
import { distanceMeters } from './services/listings/distanceCalculator.js';
|
||||
import { getUserSettings, getSettings } from './services/storage/settingsStorage.js';
|
||||
import { updateListingDistance } from './services/storage/listingsStorage.js';
|
||||
import { getSettings, getUserSettings } from './services/storage/settingsStorage.js';
|
||||
import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
|
||||
import { formatListing } from './utils/formatListing.js';
|
||||
|
||||
@@ -97,9 +97,9 @@ class FredyPipelineExecutioner {
|
||||
}
|
||||
|
||||
/**
|
||||
* Optionally enrich new listings with data from their detail pages.
|
||||
* Optionally, enrich new listings with data from their detail pages.
|
||||
* Only called when the provider config defines a `fetchDetails` function.
|
||||
* Runs all fetches in parallel. Each individual fetch must handle its own errors
|
||||
* Runs all fetches in parallel. Each fetch must handle its own errors
|
||||
* and always resolve (never reject) to avoid aborting other listings.
|
||||
*
|
||||
* @param {Listing[]} newListings New listings to enrich.
|
||||
@@ -132,7 +132,7 @@ class FredyPipelineExecutioner {
|
||||
for (const listing of newListings) {
|
||||
if (listing.address) {
|
||||
const coords = await geocodeAddress(listing.address);
|
||||
if (coords) {
|
||||
if (coords && coords.lat !== -1 && coords.lng !== -1) {
|
||||
listing.latitude = coords.lat;
|
||||
listing.longitude = coords.lng;
|
||||
}
|
||||
@@ -227,7 +227,7 @@ class FredyPipelineExecutioner {
|
||||
const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
|
||||
return new Promise((resolve, reject) => {
|
||||
extractor
|
||||
.execute(url, this._providerConfig.waitForSelector)
|
||||
.execute(url, this._providerConfig.waitForSelector, this._providerId)
|
||||
.then(() => {
|
||||
const listings = extractor.parseResponseText(
|
||||
this._providerConfig.crawlContainer,
|
||||
@@ -264,15 +264,15 @@ class FredyPipelineExecutioner {
|
||||
const requiredKeys = this._providerConfig.requiredFieldNames;
|
||||
const requireValues = ['id', 'link', 'title'];
|
||||
|
||||
const filteredListings = listings
|
||||
// this should never filter some listings out, because the normalize function should always extract all fields.
|
||||
.filter((item) => requiredKeys.every((key) => key in item))
|
||||
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
|
||||
.filter(this._providerConfig.filter)
|
||||
// filter out listings that are missing required fields
|
||||
.filter((item) => requireValues.every((key) => item[key] != null));
|
||||
|
||||
return filteredListings;
|
||||
return (
|
||||
listings
|
||||
// this should never filter some listings out, because the normalize function should always extract all fields.
|
||||
.filter((item) => requiredKeys.every((key) => key in item))
|
||||
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
|
||||
.filter(this._providerConfig.filter)
|
||||
// filter out listings that are missing required fields
|
||||
.filter((item) => requireValues.every((key) => item[key] != null))
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -7,4 +7,11 @@ export const TRACKING_POIS = {
|
||||
DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
|
||||
WELCOME_FINISHED: 'WELCOME_FINISHED',
|
||||
WELCOME_SKIPPED: 'WELCOME_SKIPPED',
|
||||
JOBS_TABLE_VIEW: 'JOBS_TABLE_VIEW',
|
||||
LISTING_TABLE_VIEW: 'LISTING_TABLE_VIEW',
|
||||
BASE_URL_SETTING: 'BASE_URL_SETTING',
|
||||
SET_PROXY_SETTING: 'SET_PROXY_SETTING',
|
||||
DETECTED_AS_BOT: 'DETECTED_AS_BOT',
|
||||
NOTES_CREATE: 'NOTES_CREATE',
|
||||
USING_LISTING_STATUS: 'USING_LISTING_STATUS',
|
||||
};
|
||||
|
||||
@@ -76,13 +76,13 @@ fastify.register(async (app) => {
|
||||
app.register(dashboardPlugin, { prefix: '/api/dashboard' });
|
||||
app.register(userSettingsPlugin, { prefix: '/api/user/settings' });
|
||||
app.register(trackingPlugin, { prefix: '/api/tracking' });
|
||||
app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
|
||||
});
|
||||
|
||||
// Admin-only routes
|
||||
fastify.register(async (app) => {
|
||||
app.addHook('preHandler', authHook);
|
||||
app.addHook('preHandler', adminHook);
|
||||
app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
|
||||
app.register(backupPlugin, { prefix: '/api/admin/backup' });
|
||||
app.register(userPlugin, { prefix: '/api/admin/users' });
|
||||
});
|
||||
|
||||
@@ -9,6 +9,10 @@ import {
|
||||
precheckRestore,
|
||||
restoreFromZip,
|
||||
} from '../../services/storage/backupRestoreService.js';
|
||||
import { getSettings } from '../../services/storage/settingsStorage.js';
|
||||
import { isAdmin } from '../security.js';
|
||||
|
||||
const DEMO_MODE_ERROR = 'Backup and restore are not available in demo mode.';
|
||||
|
||||
/**
|
||||
* @param {import('fastify').FastifyInstance} fastify
|
||||
@@ -21,7 +25,11 @@ export default async function backupPlugin(fastify) {
|
||||
(req, body, done) => done(null, body),
|
||||
);
|
||||
|
||||
fastify.get('/', async (_request, reply) => {
|
||||
fastify.get('/', async (request, reply) => {
|
||||
const settings = await getSettings();
|
||||
if (settings.demoMode && !isAdmin(request)) {
|
||||
return reply.code(403).send({ error: DEMO_MODE_ERROR });
|
||||
}
|
||||
const zipBuffer = await createBackupZip();
|
||||
const fileName = await buildBackupFileName();
|
||||
reply.header('Content-Type', 'application/zip');
|
||||
@@ -30,6 +38,10 @@ export default async function backupPlugin(fastify) {
|
||||
});
|
||||
|
||||
fastify.post('/restore', async (request, reply) => {
|
||||
const settings = await getSettings();
|
||||
if (settings.demoMode && !isAdmin(request)) {
|
||||
return reply.code(403).send({ error: DEMO_MODE_ERROR });
|
||||
}
|
||||
const { dryRun = 'false', force = 'false' } = request.query || {};
|
||||
const doDryRun = String(dryRun) === 'true';
|
||||
const doForce = String(force) === 'true';
|
||||
|
||||
@@ -9,6 +9,8 @@ import { ensureDemoUserExists } from '../../services/storage/userStorage.js';
|
||||
import logger from '../../services/logger.js';
|
||||
import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
|
||||
import { isAdmin } from '../security.js';
|
||||
import { trackPoi } from '../../services/tracking/Tracker.js';
|
||||
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
|
||||
|
||||
/**
|
||||
* @param {import('fastify').FastifyInstance} fastify
|
||||
@@ -25,16 +27,26 @@ export default async function generalSettingsPlugin(fastify) {
|
||||
}
|
||||
const localSettings = await getSettings();
|
||||
|
||||
if (localSettings.demoMode && !isAdmin(request)) {
|
||||
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change these settings.' });
|
||||
if (!isAdmin(request)) {
|
||||
const reason = localSettings.demoMode
|
||||
? 'In demo mode, it is not allowed to change these settings.'
|
||||
: 'Only admins can change these settings.';
|
||||
return reply.code(403).send({ error: reason });
|
||||
}
|
||||
|
||||
try {
|
||||
if (typeof sqlitepath !== 'undefined') {
|
||||
fs.writeFileSync(`${getDirName()}/../conf/config.json`, JSON.stringify({ sqlitepath }));
|
||||
}
|
||||
|
||||
upsertSettings(appSettings);
|
||||
ensureDemoUserExists();
|
||||
if (appSettings.baseUrl != null) {
|
||||
await trackPoi(TRACKING_POIS.BASE_URL_SETTING);
|
||||
}
|
||||
if (appSettings.proxyUrl != null) {
|
||||
await trackPoi(TRACKING_POIS.SET_PROXY_SETTING);
|
||||
}
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return reply.code(500).send({ error: 'Error while trying to write settings.' });
|
||||
|
||||
@@ -10,6 +10,8 @@ import logger from '../../services/logger.js';
|
||||
import { nullOrEmpty } from '../../utils.js';
|
||||
import { getJobs } from '../../services/storage/jobStorage.js';
|
||||
import { getSettings } from '../../services/storage/settingsStorage.js';
|
||||
import { trackPoi } from '../../services/tracking/Tracker.js';
|
||||
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
|
||||
|
||||
/**
|
||||
* @param {import('fastify').FastifyInstance} fastify
|
||||
@@ -23,6 +25,7 @@ export default async function listingsPlugin(fastify) {
|
||||
jobNameFilter,
|
||||
providerFilter,
|
||||
watchListFilter,
|
||||
statusFilter,
|
||||
sortfield = null,
|
||||
sortdir = 'asc',
|
||||
freeTextFilter,
|
||||
@@ -35,6 +38,11 @@ export default async function listingsPlugin(fastify) {
|
||||
};
|
||||
const normalizedActivity = toBool(activityFilter);
|
||||
const normalizedWatch = toBool(watchListFilter);
|
||||
const allowedStatuses = ['applied', 'rejected', 'accepted', 'none'];
|
||||
const normalizedStatus =
|
||||
typeof statusFilter === 'string' && allowedStatuses.includes(statusFilter.toLowerCase())
|
||||
? statusFilter.toLowerCase()
|
||||
: undefined;
|
||||
|
||||
let jobFilter = null;
|
||||
let jobIdFilter = null;
|
||||
@@ -54,6 +62,7 @@ export default async function listingsPlugin(fastify) {
|
||||
jobIdFilter: jobIdFilter,
|
||||
providerFilter,
|
||||
watchListFilter: normalizedWatch,
|
||||
statusFilter: normalizedStatus,
|
||||
sortField: sortfield || null,
|
||||
sortDir: sortdir === 'desc' ? 'desc' : 'asc',
|
||||
userId: request.session.currentUser,
|
||||
@@ -94,6 +103,55 @@ export default async function listingsPlugin(fastify) {
|
||||
return reply.send();
|
||||
});
|
||||
|
||||
fastify.post('/:listingId/notes', async (request, reply) => {
|
||||
const { listingId } = request.params || {};
|
||||
const { notes } = request.body || {};
|
||||
const userId = request.session?.currentUser;
|
||||
if (!listingId || !userId) {
|
||||
return reply.code(400).send({ message: 'listingId or user not provided' });
|
||||
}
|
||||
try {
|
||||
const changes = listingStorage.setListingNotes(listingId, typeof notes === 'string' ? notes : null);
|
||||
if (changes === 0) {
|
||||
return reply.code(404).send({ message: 'Listing not found' });
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(error);
|
||||
return reply.code(500).send({ message: 'Failed to update listing notes' });
|
||||
}
|
||||
|
||||
await trackPoi(TRACKING_POIS.NOTES_CREATE);
|
||||
return reply.send();
|
||||
});
|
||||
|
||||
fastify.post('/:listingId/status', async (request, reply) => {
|
||||
const { listingId } = request.params || {};
|
||||
const { status } = request.body || {};
|
||||
const userId = request.session?.currentUser;
|
||||
if (!listingId || !userId) {
|
||||
return reply.code(400).send({ message: 'listingId or user not provided' });
|
||||
}
|
||||
const allowed = ['applied', 'rejected', 'accepted'];
|
||||
const normalized = status == null ? null : String(status).toLowerCase();
|
||||
if (normalized != null && !allowed.includes(normalized)) {
|
||||
return reply.code(400).send({ message: `Invalid status: ${status}` });
|
||||
}
|
||||
try {
|
||||
const changes = listingStorage.setListingStatus(listingId, normalized);
|
||||
await trackPoi(TRACKING_POIS.USING_LISTING_STATUS);
|
||||
if (changes === 0) {
|
||||
return reply.code(404).send({ message: 'Listing not found' });
|
||||
}
|
||||
if (normalized != null) {
|
||||
watchListStorage.ensureWatch(listingId, userId);
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(error);
|
||||
return reply.code(500).send({ message: 'Failed to update listing status' });
|
||||
}
|
||||
return reply.send();
|
||||
});
|
||||
|
||||
fastify.delete('/job', async (request, reply) => {
|
||||
const { jobId, hardDelete = false } = request.body;
|
||||
const settings = await getSettings();
|
||||
|
||||
@@ -18,7 +18,7 @@ const notificationAdapter = await Promise.all(
|
||||
*/
|
||||
export default async function notificationAdapterPlugin(fastify) {
|
||||
fastify.get('/', async () => {
|
||||
return notificationAdapter.map((adapter) => adapter.config);
|
||||
return notificationAdapter.map((adapter) => adapter.config).filter(Boolean);
|
||||
});
|
||||
|
||||
fastify.post('/try', async (request, reply) => {
|
||||
|
||||
@@ -109,4 +109,70 @@ export default async function userSettingsPlugin(fastify) {
|
||||
return reply.code(500).send({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
fastify.post('/listings-view-mode', async (request, reply) => {
|
||||
const userId = request.session.currentUser;
|
||||
const { listings_view_mode } = request.body;
|
||||
|
||||
if (listings_view_mode !== 'grid' && listings_view_mode !== 'table') {
|
||||
return reply.code(400).send({ error: 'listings_view_mode must be "grid" or "table".' });
|
||||
}
|
||||
|
||||
if (listings_view_mode === 'table') {
|
||||
await trackPoi(TRACKING_POIS.LISTING_TABLE_VIEW);
|
||||
}
|
||||
|
||||
try {
|
||||
upsertSettings({ listings_view_mode }, userId);
|
||||
return { success: true };
|
||||
} catch (error) {
|
||||
logger.error('Error updating listings view mode setting', error);
|
||||
return reply.code(500).send({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
fastify.post('/jobs-view-mode', async (request, reply) => {
|
||||
const userId = request.session.currentUser;
|
||||
const { jobs_view_mode } = request.body;
|
||||
|
||||
if (jobs_view_mode !== 'grid' && jobs_view_mode !== 'table') {
|
||||
return reply.code(400).send({ error: 'jobs_view_mode must be "grid" or "table".' });
|
||||
}
|
||||
|
||||
if (jobs_view_mode === 'table') {
|
||||
await trackPoi(TRACKING_POIS.JOBS_TABLE_VIEW);
|
||||
}
|
||||
|
||||
try {
|
||||
upsertSettings({ jobs_view_mode }, userId);
|
||||
return { success: true };
|
||||
} catch (error) {
|
||||
logger.error('Error updating jobs view mode setting', error);
|
||||
return reply.code(500).send({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
fastify.post('/listing-deletion-preference', async (request, reply) => {
|
||||
const userId = request.session.currentUser;
|
||||
const { listing_deletion_preference } = request.body;
|
||||
|
||||
const globalSettings = await getSettings();
|
||||
if (globalSettings.demoMode && !isAdmin(request)) {
|
||||
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
|
||||
}
|
||||
|
||||
if (listing_deletion_preference == null) {
|
||||
return reply.code(400).send({ error: 'listing_deletion_preference is required.' });
|
||||
}
|
||||
|
||||
const { skipPrompt, hardDelete } = listing_deletion_preference;
|
||||
|
||||
try {
|
||||
upsertSettings({ listing_deletion_preference: { skipPrompt, hardDelete } }, userId);
|
||||
return { success: true };
|
||||
} catch (error) {
|
||||
logger.error('Error updating listing deletion preference', error);
|
||||
return reply.code(500).send({ error: error.message });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -155,6 +155,12 @@ export function createMcpServer() {
|
||||
),
|
||||
sortField: z.string().optional().describe('Sort by: created_at, price, size, provider, title, is_active'),
|
||||
sortDir: z.string().optional().describe('Sort direction: asc or desc'),
|
||||
status: z
|
||||
.enum(['applied', 'rejected', 'accepted', 'none'])
|
||||
.optional()
|
||||
.describe(
|
||||
'Filter by user-set status. "applied", "rejected", or "accepted" return only listings with that status; "none" returns only listings without a status set.',
|
||||
),
|
||||
},
|
||||
async (
|
||||
{
|
||||
@@ -170,6 +176,7 @@ export function createMcpServer() {
|
||||
maxPrice,
|
||||
sortField,
|
||||
sortDir,
|
||||
status,
|
||||
},
|
||||
extra,
|
||||
) => {
|
||||
@@ -192,6 +199,7 @@ export function createMcpServer() {
|
||||
maxPrice: maxPrice ?? null,
|
||||
sortField: sortField ?? null,
|
||||
sortDir: sortDir ?? 'desc',
|
||||
statusFilter: status,
|
||||
userId: user.id,
|
||||
isAdmin: user.isAdmin,
|
||||
});
|
||||
|
||||
@@ -124,10 +124,10 @@ export function normalizeListListings(queryResult, { page, pageSize }) {
|
||||
md += '\n\n';
|
||||
|
||||
if (listings.length > 0) {
|
||||
md += `| ID | Title | Address | Price | Size | Provider | Active | Created | Job |\n`;
|
||||
md += `|----|-------|---------|-------|------|----------|--------|---------|-----|\n`;
|
||||
md += `| ID | Title | Address | Price | Size | Provider | Active | Status | Created | Job |\n`;
|
||||
md += `|----|-------|---------|-------|------|----------|--------|--------|---------|-----|\n`;
|
||||
for (const l of listings) {
|
||||
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
|
||||
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${cell(l.status?.status)} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
|
||||
}
|
||||
md += `\nUse **get_listing** with an ID for full details (description, link, image).\n`;
|
||||
} else {
|
||||
@@ -156,6 +156,10 @@ export function normalizeGetListing(listing) {
|
||||
md += `- **Link:** ${listing.link || '–'}\n`;
|
||||
md += `- **Image:** ${listing.image_url || '–'}\n`;
|
||||
md += `- **Active:** ${listing.is_active ? 'yes' : 'no'}\n`;
|
||||
md += `- **Status:** ${listing.status?.status || '–'}\n`;
|
||||
if (listing.status?.setAt) {
|
||||
md += `- **Status set at:** ${formatDate(listing.status.setAt)}\n`;
|
||||
}
|
||||
md += `- **Created:** ${formatDate(listing.created_at)}\n`;
|
||||
md += `- **Job:** ${listing.job_name || '–'}\n`;
|
||||
if (listing.latitude != null && listing.longitude != null) {
|
||||
|
||||
@@ -13,7 +13,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
|
||||
const jobName = job == null ? jobKey : job.name;
|
||||
const promises = newListings.map((newListing) => {
|
||||
const title = `${jobName} at ${serviceName}: ${newListing.title}`;
|
||||
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
|
||||
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
|
||||
const message = `Address: ${newListing.address}\nSize: ${newListing.size}\nPrice: ${newListing.price}\nLink: ${newListing.link}${fredyLine}`;
|
||||
return fetch(server, {
|
||||
method: 'POST',
|
||||
|
||||
@@ -7,7 +7,7 @@ import { markdown2Html } from '../../services/markdown.js';
|
||||
|
||||
export const send = ({ serviceName, newListings, jobKey, baseUrl }) => {
|
||||
/* eslint-disable no-console */
|
||||
const fredyLinks = baseUrl ? newListings.map((l) => `${baseUrl}/listings/listing/${l.id}`).join(', ') : null;
|
||||
const fredyLinks = baseUrl ? newListings.map((l) => `${baseUrl}/#/listings/listing/${l.id}`).join(', ') : null;
|
||||
return [
|
||||
Promise.resolve(
|
||||
console.info(
|
||||
|
||||
@@ -83,7 +83,7 @@ const buildEmbed = (jobKey, listing, baseUrl) => {
|
||||
if (baseUrl && listing.id) {
|
||||
fields.push({
|
||||
name: 'Open in Fredy',
|
||||
value: `[Open in Fredy](${baseUrl}/listings/listing/${listing.id})`,
|
||||
value: `[Open in Fredy](${baseUrl}/#/listings/listing/${listing.id})`,
|
||||
inline: false,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -13,7 +13,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
|
||||
let message = `### *${jobName}* (${serviceName}) found **${newListings.length}** new listings:\n\n`;
|
||||
message += `| Title | Address | Size | Price |${baseUrl ? ' Open in Fredy |' : ''}\n|:----|:----|:----|:----|${baseUrl ? ':----|\n' : '\n'}`;
|
||||
message += newListings.map((o) => {
|
||||
const fredyCell = baseUrl && o.id ? ` [Open in Fredy](${baseUrl}/listings/listing/${o.id}) |` : '';
|
||||
const fredyCell = baseUrl && o.id ? ` [Open in Fredy](${baseUrl}/#/listings/listing/${o.id}) |` : '';
|
||||
return (
|
||||
`| [${o.title}](${o.link}) | ` +
|
||||
[o.address, o.size.replace(/2m/g, '$m^2$'), o.price].join(' | ') +
|
||||
|
||||
@@ -14,7 +14,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
|
||||
const jobName = job == null ? jobKey : job.name;
|
||||
|
||||
const promises = newListings.map((newListing) => {
|
||||
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
|
||||
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
|
||||
const message = `
|
||||
Address: ${newListing.address}
|
||||
Size: ${newListing.size == null ? 'N/A' : newListing.size.replace(/2m/g, '$m^2$')}
|
||||
|
||||
@@ -15,7 +15,8 @@ export const send = async ({ serviceName, newListings, notificationConfig, jobKe
|
||||
const results = await Promise.all(
|
||||
newListings.map(async (newListing) => {
|
||||
const title = `${jobName} at ${serviceName}: ${newListing.title}`;
|
||||
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
|
||||
const fredyLine =
|
||||
baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
|
||||
const message = `Address: ${newListing.address}\nSize: ${newListing.size}\nPrice: ${newListing.price}\nLink: ${newListing.link}${fredyLine}`;
|
||||
|
||||
const form = new FormData();
|
||||
|
||||
@@ -39,7 +39,7 @@ const buildBlocks = (serviceName, jobKey, p, baseUrl) => {
|
||||
if (baseUrl && p.id) {
|
||||
blocks.push({
|
||||
type: 'section',
|
||||
text: { type: 'mrkdwn', text: `<${baseUrl}/listings/listing/${p.id}|Open in Fredy>` },
|
||||
text: { type: 'mrkdwn', text: `<${baseUrl}/#/listings/listing/${p.id}|Open in Fredy>` },
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ const buildBlocks = (serviceName, jobKey, p, baseUrl) => {
|
||||
if (baseUrl && p.id) {
|
||||
blocks.push({
|
||||
type: 'section',
|
||||
text: { type: 'mrkdwn', text: `<${baseUrl}/listings/listing/${p.id}|Open in Fredy>` },
|
||||
text: { type: 'mrkdwn', text: `<${baseUrl}/#/listings/listing/${p.id}|Open in Fredy>` },
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ import fetch from 'node-fetch';
|
||||
import pThrottle from 'p-throttle';
|
||||
import { normalizeImageUrl } from '../../utils.js';
|
||||
import logger from '../../services/logger.js';
|
||||
import { shouldUseMultipart, buildPhotoFormData } from './telegramPhotoUploader.js';
|
||||
|
||||
const RATE_LIMIT_INTERVAL = 1000;
|
||||
const chatThrottleMap = new Map();
|
||||
@@ -84,7 +85,7 @@ function buildCaption(jobName, serviceName, o, baseUrl) {
|
||||
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
|
||||
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
|
||||
const fredyLink =
|
||||
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
|
||||
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/#/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
|
||||
return `<i>${escapeHtml(jobName)}</i> (${escapeHtml(serviceName)})\n<a href='${escapeHtml(
|
||||
o.link || '',
|
||||
)}'><b>${escapeHtml(title)}</b></a>\n${escapeHtml(meta)}${fredyLink}`.slice(0, 1024);
|
||||
@@ -101,7 +102,7 @@ function buildText(jobName, serviceName, o, baseUrl) {
|
||||
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
|
||||
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
|
||||
const fredyLink =
|
||||
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
|
||||
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/#/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
|
||||
return (
|
||||
`<i>${escapeHtml(jobName)}</i> (${escapeHtml(serviceName)})\n` +
|
||||
`<a href='${escapeHtml(o.link || '')}'><b>${escapeHtml(title)}</b></a>\n` +
|
||||
@@ -120,7 +121,7 @@ function buildText(jobName, serviceName, o, baseUrl) {
|
||||
function buildCaptionPlain(jobName, serviceName, o, baseUrl) {
|
||||
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
|
||||
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
|
||||
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${o.id}` : '';
|
||||
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${o.id}` : '';
|
||||
return `${jobName} (${serviceName})\n${title}\n${meta}\n\n${o.link || ''}${fredyLine}`.slice(0, 4096);
|
||||
}
|
||||
|
||||
@@ -134,7 +135,7 @@ function buildCaptionPlain(jobName, serviceName, o, baseUrl) {
|
||||
function buildTextPlain(jobName, serviceName, o, baseUrl) {
|
||||
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
|
||||
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
|
||||
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${o.id}` : '';
|
||||
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${o.id}` : '';
|
||||
return `${jobName} (${serviceName})\n${title}\n${o.link || ''}\n${meta}${fredyLine}`;
|
||||
}
|
||||
|
||||
@@ -177,11 +178,13 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
|
||||
const jobName = job == null ? jobKey : job.name;
|
||||
|
||||
const throttledCall = getThrottled(chatId, async function (endpoint, body) {
|
||||
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, {
|
||||
method: 'post',
|
||||
body: JSON.stringify(body),
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
// FormData (multipart) vs JSON. node-fetch sets its own multipart boundary
|
||||
// header, so we must NOT supply Content-Type ourselves in that case.
|
||||
const isFormData = body instanceof FormData;
|
||||
const opts = isFormData
|
||||
? { method: 'post', body }
|
||||
: { method: 'post', body: JSON.stringify(body), headers: { 'Content-Type': 'application/json' } };
|
||||
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, opts);
|
||||
|
||||
if (!res.ok) {
|
||||
const errorBody = await res.text();
|
||||
@@ -208,16 +211,28 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
|
||||
});
|
||||
}
|
||||
|
||||
return await throttledCall('sendPhoto', {
|
||||
chat_id: chatId,
|
||||
photo: img,
|
||||
caption: plainText
|
||||
? buildCaptionPlain(jobName, serviceName, o, baseUrl)
|
||||
: buildCaption(jobName, serviceName, o, baseUrl),
|
||||
...(plainText ? {} : { parse_mode: 'HTML' }),
|
||||
...(message_thread_id ? { message_thread_id } : {}),
|
||||
}).catch(async (e) => {
|
||||
logger.error(`Error sending photo to Telegram and use a fallback: ${e.message}`);
|
||||
const caption = plainText
|
||||
? buildCaptionPlain(jobName, serviceName, o, baseUrl)
|
||||
: buildCaption(jobName, serviceName, o, baseUrl);
|
||||
const parseMode = plainText ? undefined : 'HTML';
|
||||
|
||||
// .webp URLs (Immowelt/Cloudimage) fail Telegram's URL-based sendPhoto with
|
||||
// "failed to get HTTP URL content". Upload the bytes via multipart instead;
|
||||
// the rendered chat message is identical.
|
||||
const photoCall = shouldUseMultipart(img)
|
||||
? buildPhotoFormData({ chatId, imageUrl: img, caption, parseMode, messageThreadId: message_thread_id }).then(
|
||||
(fd) => throttledCall('sendPhoto', fd),
|
||||
)
|
||||
: throttledCall('sendPhoto', {
|
||||
chat_id: chatId,
|
||||
photo: img,
|
||||
caption,
|
||||
...(parseMode ? { parse_mode: parseMode } : {}),
|
||||
...(message_thread_id ? { message_thread_id } : {}),
|
||||
});
|
||||
|
||||
return await photoCall.catch(async (e) => {
|
||||
logger.warn(`Error sending photo to Telegram and use a fallback: ${e.message}`);
|
||||
return await throttledCall('sendMessage', textPayload).catch((e) => {
|
||||
logger.error(`Error sending message to Telegram: ${e.message}`);
|
||||
throw e;
|
||||
|
||||
106
lib/notification/adapter/telegramPhotoUploader.js
Normal file
106
lib/notification/adapter/telegramPhotoUploader.js
Normal file
@@ -0,0 +1,106 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
/**
|
||||
* Helpers for sending photos to Telegram via `multipart/form-data` instead of
|
||||
* the HTTP-URL path. Used when the URL is one that Telegram's URL-fetcher will
|
||||
* reject - notably `.webp` images from Cloudimage (mms.immowelt.de), which
|
||||
* Telegram refuses with "Bad Request: failed to get HTTP URL content".
|
||||
*
|
||||
* The HTTP-URL path is faster and is still the default in telegram.js; this
|
||||
* module is the fallback for URLs whose extension makes Telegram fail.
|
||||
*/
|
||||
|
||||
/** Telegram's sendPhoto limit when uploading bytes via multipart/form-data. */
|
||||
const TELEGRAM_MULTIPART_MAX_BYTES = 10 * 1024 * 1024;
|
||||
|
||||
/** Accept header used when re-fetching the image ourselves.
|
||||
* Deliberately excludes `image/webp` so CDNs that content-negotiate
|
||||
* (like Cloudimage on mms.immowelt.de) transcode WEBP to JPEG. */
|
||||
const NON_WEBP_ACCEPT = 'image/jpeg,image/png,image/*;q=0.8';
|
||||
|
||||
/**
|
||||
* Returns true if the URL's path ends in a `.webp` extension. Such URLs need
|
||||
* multipart upload because Telegram identifies media types from the URL path
|
||||
* and rejects `.webp` in sendPhoto via HTTP URL.
|
||||
*
|
||||
* Conservative: returns false for null/empty/non-string input, malformed URLs,
|
||||
* and non-https schemes.
|
||||
*
|
||||
* @param {string|null|undefined} url
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function shouldUseMultipart(url) {
|
||||
if (typeof url !== 'string' || url.length === 0) return false;
|
||||
let parsed;
|
||||
try {
|
||||
parsed = new URL(url);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (parsed.protocol !== 'https:') return false;
|
||||
return /\.webp$/i.test(parsed.pathname);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch an image from `imageUrl` and build a `FormData` body suitable for
|
||||
* POSTing to `https://api.telegram.org/bot<token>/sendPhoto`.
|
||||
*
|
||||
* - Sends an `Accept` header that excludes `image/webp` so origin/CDN servers
|
||||
* that content-negotiate return JPEG bytes.
|
||||
* - Rejects images larger than Telegram's 10 MB multipart limit, both
|
||||
* advertised via `Content-Length` and (defensively) after download.
|
||||
* - The `photo` field is named with a `.jpg` extension because Telegram
|
||||
* identifies file type from the filename.
|
||||
*
|
||||
* Throws if the image fetch fails, the size limit is exceeded, or the URL is
|
||||
* unreachable. The caller is responsible for catching and falling back.
|
||||
*
|
||||
* @param {Object} args
|
||||
* @param {string|number} args.chatId
|
||||
* @param {string} args.imageUrl
|
||||
* @param {string} args.caption
|
||||
* @param {string} [args.parseMode] - Telegram parse_mode, e.g. 'HTML'.
|
||||
* @param {number} [args.messageThreadId] - Telegram supergroup topic id.
|
||||
* @returns {Promise<FormData>}
|
||||
*/
|
||||
export async function buildPhotoFormData({ chatId, imageUrl, caption, parseMode, messageThreadId }) {
|
||||
const res = await fetch(imageUrl, {
|
||||
method: 'GET',
|
||||
headers: { Accept: NON_WEBP_ACCEPT },
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`Failed to fetch image for multipart upload (${res.status}): ${imageUrl}`);
|
||||
}
|
||||
|
||||
const advertised = Number(res.headers.get('content-length'));
|
||||
if (Number.isFinite(advertised) && advertised > TELEGRAM_MULTIPART_MAX_BYTES) {
|
||||
throw new Error(
|
||||
`Image exceeds Telegram multipart size limit (advertised ${advertised} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
|
||||
);
|
||||
}
|
||||
|
||||
const buf = await res.arrayBuffer();
|
||||
if (buf.byteLength > TELEGRAM_MULTIPART_MAX_BYTES) {
|
||||
throw new Error(
|
||||
`Image exceeds Telegram multipart size limit (downloaded ${buf.byteLength} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Telegram identifies the media type from the filename extension. We always
|
||||
// upload as .jpg because the Accept header forces JPEG bytes from CDNs that
|
||||
// honor it; for the rare CDN that ignores Accept and still returns WEBP, the
|
||||
// .jpg filename is a small lie but Telegram's image pipeline accepts it.
|
||||
const blob = new Blob([buf], { type: 'image/jpeg' });
|
||||
|
||||
const fd = new FormData();
|
||||
fd.append('chat_id', String(chatId));
|
||||
fd.append('caption', caption);
|
||||
if (parseMode) fd.append('parse_mode', parseMode);
|
||||
if (messageThreadId != null) fd.append('message_thread_id', String(messageThreadId));
|
||||
fd.append('photo', blob, 'photo.jpg');
|
||||
return fd;
|
||||
}
|
||||
@@ -26,7 +26,7 @@ function parseId(shortenedLink) {
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser });
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immobilienDe_details' });
|
||||
if (!html) return listing;
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
|
||||
@@ -16,7 +16,7 @@ let appliedBlackList = [];
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser });
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immowelt_details' });
|
||||
if (!html) return listing;
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
@@ -87,7 +87,19 @@ const config = {
|
||||
crawlContainer:
|
||||
'div[data-testid="serp-core-scrollablelistview-testid"]:not(div[data-testid="serp-enlargementlist-testid"] div[data-testid="serp-card-testid"]) div[data-testid="serp-core-classified-card-testid"]',
|
||||
sortByDateParam: 'order=DateDesc',
|
||||
waitForSelector: 'div[data-testid="serp-gridcontainer-testid"]',
|
||||
// waitForSelector is null: extract the full page via page.content() so the
|
||||
// Cheerio crawler can search anywhere in the rendered document.
|
||||
// preNavigateUrl visits the homepage first to establish a trusted session
|
||||
// before hitting the search URL; this prevents CDN-level bot challenges that
|
||||
// fire on cold sessions. waitForNetworkIdle (phase 2) then catches React's
|
||||
// listing API round-trip that fires well after domcontentloaded.
|
||||
waitForSelector: null,
|
||||
puppeteerOptions: {
|
||||
puppeteerTimeout: 60_000,
|
||||
preNavigateUrl: 'https://www.immowelt.de/',
|
||||
waitForNetworkIdle: true,
|
||||
waitForNetworkIdleTimeout: 60_000,
|
||||
},
|
||||
crawlFields: {
|
||||
id: 'a@href',
|
||||
price: 'div[data-testid="cardmfe-price-testid"] | removeNewline | trim',
|
||||
|
||||
@@ -128,7 +128,7 @@ async function enrichListingFromDetails(listing, browser) {
|
||||
if (!absoluteLink) return listing;
|
||||
|
||||
try {
|
||||
const html = await puppeteerExtractor(absoluteLink, null, { browser });
|
||||
const html = await puppeteerExtractor(absoluteLink, null, { browser, name: 'kleinanzeigen_details' });
|
||||
if (!html) return { ...listing, link: absoluteLink };
|
||||
|
||||
const { detailAddress, detailDescription } = extractDetailFromHtml(html);
|
||||
@@ -196,8 +196,8 @@ const config = {
|
||||
id: '.aditem@data-adid',
|
||||
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
|
||||
tags: '.aditem-main--middle--tags | removeNewline | trim',
|
||||
title: '.aditem-main .text-module-begin a | removeNewline | trim',
|
||||
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
|
||||
title: '.aditem-main .text-module-begin | removeNewline | trim',
|
||||
link: '.aditem@data-href',
|
||||
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
|
||||
address: '.aditem-main--top--left | trim | removeNewline',
|
||||
image: 'img@src',
|
||||
|
||||
@@ -16,7 +16,7 @@ let appliedBlackList = [];
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, 'body', { browser });
|
||||
const html = await puppeteerExtractor(listing.link, 'body', { browser, name: 'sparkasse_details' });
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
const nextDataRaw = $('#__NEXT_DATA__').text;
|
||||
|
||||
@@ -16,7 +16,7 @@ let appliedBlackList = [];
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser });
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'wgGesucht_details' });
|
||||
if (!html) return listing;
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
|
||||
147
lib/services/ensureValidBinary.js
Normal file
147
lib/services/ensureValidBinary.js
Normal file
@@ -0,0 +1,147 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { ensureBinary } from 'cloakbrowser';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
|
||||
/**
|
||||
* Resource files required on Linux/Windows — they must live next to the chrome binary.
|
||||
* macOS packages these inside the .app bundle's Frameworks directory so a different
|
||||
* check is used there (see isBinaryComplete).
|
||||
*/
|
||||
const LINUX_WIN_REQUIRED_FILES = ['icudtl.dat', 'resources.pak'];
|
||||
|
||||
/**
|
||||
* Return the top-level versioned installation directory for any platform.
|
||||
*
|
||||
* - Linux/Windows: binaryPath is ~/.cloakbrowser/chromium-X.Y.Z/chrome
|
||||
* → dirname ~/.cloakbrowser/chromium-X.Y.Z/
|
||||
* - macOS: binaryPath is ~/.cloakbrowser/chromium-X.Y.Z/Chromium.app/Contents/MacOS/Chromium
|
||||
* → 4 levels up ~/.cloakbrowser/chromium-X.Y.Z/
|
||||
*
|
||||
* @param {string} binaryPath
|
||||
* @returns {string}
|
||||
*/
|
||||
function getVersionedDir(binaryPath) {
|
||||
if (process.platform === 'darwin') {
|
||||
return path.resolve(path.dirname(binaryPath), '../../..');
|
||||
}
|
||||
return path.dirname(binaryPath);
|
||||
}
|
||||
|
||||
/**
|
||||
* Return true when the binary at binaryPath belongs to a complete installation.
|
||||
*
|
||||
* On macOS the binary lives inside an .app bundle:
|
||||
* Chromium.app/Contents/MacOS/Chromium
|
||||
* Resource files (icudtl.dat etc.) are deep inside
|
||||
* Chromium.app/Contents/Frameworks/…
|
||||
* so checking for them next to the binary is wrong. Instead we verify the two
|
||||
* structural markers that are only present after a full extraction: Info.plist
|
||||
* and the Frameworks directory inside Contents/.
|
||||
*
|
||||
* On Linux/Windows the binary and all resource files are siblings in the same
|
||||
* directory.
|
||||
*
|
||||
* @param {string} binaryPath
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function isBinaryComplete(binaryPath) {
|
||||
if (process.platform === 'darwin') {
|
||||
const contentsDir = path.resolve(path.dirname(binaryPath), '..');
|
||||
return fs.existsSync(path.join(contentsDir, 'Info.plist')) && fs.existsSync(path.join(contentsDir, 'Frameworks'));
|
||||
}
|
||||
const dir = path.dirname(binaryPath);
|
||||
return LINUX_WIN_REQUIRED_FILES.every((f) => fs.existsSync(path.join(dir, f)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Return a human-readable description of which required files/dirs are missing.
|
||||
*
|
||||
* @param {string} binaryPath
|
||||
* @returns {string}
|
||||
*/
|
||||
function missingDescription(binaryPath) {
|
||||
if (process.platform === 'darwin') {
|
||||
const contentsDir = path.resolve(path.dirname(binaryPath), '..');
|
||||
return ['Info.plist', 'Frameworks'].filter((f) => !fs.existsSync(path.join(contentsDir, f))).join(', ');
|
||||
}
|
||||
const dir = path.dirname(binaryPath);
|
||||
return LINUX_WIN_REQUIRED_FILES.filter((f) => !fs.existsSync(path.join(dir, f))).join(', ');
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a corrupt binary installation and all `latest_version*` markers from
|
||||
* the CloakBrowser cache so the next `ensureBinary()` call falls back to the
|
||||
* package-bundled version.
|
||||
*
|
||||
* Removes the full versioned directory (e.g. chromium-X.Y.Z/) on all platforms,
|
||||
* not just the subdirectory that contains the binary.
|
||||
*
|
||||
* @param {string} binaryPath - Path to the (corrupt) chrome/Chromium binary.
|
||||
*/
|
||||
function removeCorruptInstallation(binaryPath) {
|
||||
const versionedDir = getVersionedDir(binaryPath);
|
||||
const cacheDir = process.env.CLOAKBROWSER_CACHE_DIR || path.join(os.homedir(), '.cloakbrowser');
|
||||
|
||||
fs.rmSync(versionedDir, { recursive: true, force: true });
|
||||
|
||||
try {
|
||||
for (const entry of fs.readdirSync(cacheDir)) {
|
||||
if (entry.startsWith('latest_version')) {
|
||||
fs.rmSync(path.join(cacheDir, entry), { force: true });
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Cache dir may not exist if versionedDir was the only entry — ignore.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure the CloakBrowser stealth Chromium binary is present **and** complete.
|
||||
*
|
||||
* `cloakbrowser`'s own `ensureBinary()` only checks that the chrome/Chromium
|
||||
* file exists. An incomplete extraction (e.g. interrupted download, disk full)
|
||||
* can leave a directory that contains the executable but is missing essential
|
||||
* resource files. Chrome then crashes immediately on launch.
|
||||
*
|
||||
* This wrapper validates the path returned by `ensureBinary()`. If the
|
||||
* installation is incomplete it removes the corrupt directory, clears the
|
||||
* version marker files, and calls `ensureBinary()` again so it falls back to
|
||||
* (or re-downloads) a complete build.
|
||||
*
|
||||
* The validated path is also pinned via `CLOAKBROWSER_BINARY_PATH` so that
|
||||
* CloakBrowser's own internal `ensureBinary()` call inside `launch()` always
|
||||
* picks up the same, verified binary.
|
||||
*
|
||||
* @returns {Promise<string>} Absolute path to the validated binary.
|
||||
* @throws {Error} When even the fallback binary is incomplete.
|
||||
*/
|
||||
export async function ensureValidBinary() {
|
||||
const binaryPath = await ensureBinary();
|
||||
|
||||
if (isBinaryComplete(binaryPath)) {
|
||||
process.env.CLOAKBROWSER_BINARY_PATH = binaryPath;
|
||||
return binaryPath;
|
||||
}
|
||||
|
||||
console.warn(
|
||||
`[fredy] CloakBrowser installation at ${getVersionedDir(binaryPath)} is missing: ${missingDescription(binaryPath)}. Removing and retrying.`,
|
||||
);
|
||||
|
||||
removeCorruptInstallation(binaryPath);
|
||||
|
||||
const fallbackPath = await ensureBinary();
|
||||
if (!isBinaryComplete(fallbackPath)) {
|
||||
throw new Error(
|
||||
`CloakBrowser binary at ${getVersionedDir(fallbackPath)} is still missing required files after re-download: ${missingDescription(fallbackPath)}`,
|
||||
);
|
||||
}
|
||||
|
||||
process.env.CLOAKBROWSER_BINARY_PATH = fallbackPath;
|
||||
return fallbackPath;
|
||||
}
|
||||
@@ -29,11 +29,12 @@ export default class Extractor {
|
||||
* your response will never contain what you are really looking for
|
||||
* @param url
|
||||
* @param waitForSelector
|
||||
* @param jobKey
|
||||
*/
|
||||
execute = async (url, waitForSelector = null) => {
|
||||
execute = async (url, waitForSelector = null, jobKey = null) => {
|
||||
this.responseText = null;
|
||||
try {
|
||||
this.responseText = await puppeteerExtractor(url, waitForSelector, this.options);
|
||||
this.responseText = await puppeteerExtractor(url, waitForSelector, { ...this.options, name: jobKey });
|
||||
if (this.responseText != null) {
|
||||
loadParser(this.responseText);
|
||||
}
|
||||
|
||||
@@ -3,121 +3,133 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import puppeteer from 'puppeteer-extra';
|
||||
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
|
||||
import { debug, botDetected } from './utils.js';
|
||||
import {
|
||||
getPreLaunchConfig,
|
||||
applyBotPreventionToPage,
|
||||
applyLanguagePersistence,
|
||||
applyPostNavigationHumanSignals,
|
||||
} from './botPrevention.js';
|
||||
import { launch } from 'cloakbrowser/puppeteer';
|
||||
import { botDetected, debug } from './utils.js';
|
||||
import { getPreLaunchConfig } from './botPrevention.js';
|
||||
import logger from '../logger.js';
|
||||
import fs from 'fs';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
|
||||
puppeteer.use(StealthPlugin());
|
||||
import { trackPoi } from '../tracking/Tracker.js';
|
||||
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
|
||||
|
||||
/**
|
||||
* Launch a CloakBrowser/Puppeteer browser instance with stealth and humanizer enabled.
|
||||
*
|
||||
* CloakBrowser applies 49 C++ source-level patches (canvas, WebGL, audio, WebRTC,
|
||||
* navigator.*, automation signals) that are indistinguishable from a real browser.
|
||||
* All fingerprinting and human-behaviour simulation is handled natively; no CDP
|
||||
* overrides (setUserAgent, setExtraHTTPHeaders, evaluateOnNewDocument) are applied
|
||||
* here because they would create detectable inconsistencies on top of the C++ patches.
|
||||
*
|
||||
* @param {string} url - Initial URL (used to derive locale/timezone hints).
|
||||
* @param {object} [options]
|
||||
* @param {boolean} [options.puppeteerHeadless]
|
||||
* @param {number} [options.puppeteerTimeout]
|
||||
* @param {string} [options.proxyUrl]
|
||||
* @param {string} [options.timezone]
|
||||
* @param {string} [options.acceptLanguage]
|
||||
* @param {object} [options.viewport]
|
||||
* @returns {Promise<import('puppeteer-core').Browser>}
|
||||
*/
|
||||
export async function launchBrowser(url, options) {
|
||||
const preCfg = getPreLaunchConfig(url, options || {});
|
||||
const launchArgs = [
|
||||
|
||||
// Docker requires --no-sandbox; CloakBrowser handles all stealth args internally.
|
||||
// --ignore-certificate-errors is needed because CloakBrowser ships its own Chromium
|
||||
// binary with an independent CA bundle that may not trust proxies or interceptors
|
||||
// present in the host environment.
|
||||
const args = [
|
||||
'--no-sandbox',
|
||||
'--disable-gpu',
|
||||
'--disable-setuid-sandbox',
|
||||
'--disable-dev-shm-usage',
|
||||
'--disable-crash-reporter',
|
||||
'--no-first-run',
|
||||
'--no-default-browser-check',
|
||||
preCfg.langArg,
|
||||
'--ignore-certificate-errors',
|
||||
// Disables the zygote process model. Required in some container environments
|
||||
// (e.g. limited kernel namespaces) where the zygote cannot acquire the
|
||||
// locks it needs and exits with "Invalid file descriptor to ICU data received".
|
||||
'--no-zygote',
|
||||
preCfg.windowSizeArg,
|
||||
...preCfg.extraArgs,
|
||||
];
|
||||
if (options?.proxyUrl) {
|
||||
launchArgs.push(`--proxy-server=${options.proxyUrl}`);
|
||||
}
|
||||
|
||||
let userDataDir;
|
||||
let removeUserDataDir = false;
|
||||
if (options && options.userDataDir) {
|
||||
userDataDir = options.userDataDir;
|
||||
} else {
|
||||
const prefix = path.join(os.tmpdir(), 'puppeteer-fredy-');
|
||||
userDataDir = fs.mkdtempSync(prefix);
|
||||
removeUserDataDir = true;
|
||||
}
|
||||
|
||||
// On ARM64 Docker, Chrome for Testing has no native binary - use system Chromium instead.
|
||||
const executablePath =
|
||||
options?.executablePath ||
|
||||
(process.arch === 'arm64' && process.env.IS_DOCKER === 'true' ? '/usr/bin/chromium' : undefined);
|
||||
|
||||
const browser = await puppeteer.launch({
|
||||
return await launch({
|
||||
headless: options?.puppeteerHeadless ?? true,
|
||||
args: launchArgs,
|
||||
timeout: options?.puppeteerTimeout || 45_000,
|
||||
userDataDir,
|
||||
executablePath,
|
||||
humanize: true,
|
||||
args,
|
||||
// locale sets Accept-Language headers and JS navigator.language consistently
|
||||
locale: preCfg.langForFlag,
|
||||
...(options?.proxyUrl ? { proxy: options.proxyUrl } : {}),
|
||||
...(preCfg.timezone ? { timezone: preCfg.timezone } : {}),
|
||||
});
|
||||
|
||||
browser.__fredy_userDataDir = userDataDir;
|
||||
browser.__fredy_removeUserDataDir = removeUserDataDir;
|
||||
|
||||
return browser;
|
||||
}
|
||||
|
||||
/**
|
||||
* Close a browser instance returned by {@link launchBrowser}.
|
||||
*
|
||||
* @param {import('puppeteer-core').Browser | null} browser
|
||||
*/
|
||||
export async function closeBrowser(browser) {
|
||||
if (!browser) return;
|
||||
const userDataDir = browser.__fredy_userDataDir;
|
||||
const removeUserDataDir = browser.__fredy_removeUserDataDir;
|
||||
try {
|
||||
await browser.close();
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
if (removeUserDataDir && userDataDir) {
|
||||
try {
|
||||
await fs.promises.rm(userDataDir, { recursive: true, force: true });
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Open a page in a (possibly reused) browser, navigate to `url`, and return the HTML source.
|
||||
* Returns `null` when a bot-detection page is encountered or on timeout.
|
||||
*
|
||||
* @param {string} url
|
||||
* @param {string | null} waitForSelector
|
||||
* @param {object} [options]
|
||||
* @returns {Promise<string | null>}
|
||||
*/
|
||||
export default async function execute(url, waitForSelector, options) {
|
||||
let browser = options?.browser;
|
||||
let isExternalBrowser = !!browser;
|
||||
let page;
|
||||
let result;
|
||||
try {
|
||||
debug(`Sending request to ${url} using Puppeteer.`);
|
||||
debug(`Sending request to ${url} using CloakBrowser.`);
|
||||
|
||||
if (!isExternalBrowser) {
|
||||
browser = await launchBrowser(url, options);
|
||||
}
|
||||
|
||||
page = await browser.newPage();
|
||||
const preCfg = getPreLaunchConfig(url, options || {});
|
||||
await applyBotPreventionToPage(page, preCfg);
|
||||
// Provide languages value before navigation
|
||||
await applyLanguagePersistence(page, preCfg);
|
||||
|
||||
// Optional cookies
|
||||
if (Array.isArray(options?.cookies) && options.cookies.length > 0) {
|
||||
await page.setCookie(...options.cookies);
|
||||
}
|
||||
|
||||
// Navigation
|
||||
// Warm-up navigation: visit a trusted page first so the site sees an
|
||||
// established session before the actual target URL. Silently ignored on
|
||||
// failure so it never blocks the main request.
|
||||
if (options?.preNavigateUrl) {
|
||||
try {
|
||||
await page.goto(options.preNavigateUrl, { waitUntil: 'domcontentloaded', timeout: 30_000 });
|
||||
await new Promise((r) => setTimeout(r, 1500 + Math.random() * 2000));
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
|
||||
const response = await page.goto(url, {
|
||||
waitUntil: options?.waitUntil || 'domcontentloaded',
|
||||
timeout: options?.puppeteerTimeout || 60000,
|
||||
});
|
||||
|
||||
// Optionally wait and add subtle human-like interactions
|
||||
await applyPostNavigationHumanSignals(page, preCfg);
|
||||
// Optional second idle wait: useful for React SPAs that trigger API calls
|
||||
// after domcontentloaded. Times out silently so we use whatever is rendered.
|
||||
if (options?.waitForNetworkIdle) {
|
||||
try {
|
||||
await page.waitForNetworkIdle({ timeout: options?.waitForNetworkIdleTimeout ?? 60_000 });
|
||||
} catch {
|
||||
// ignore — we proceed with whatever the DOM contains at this point
|
||||
}
|
||||
}
|
||||
|
||||
let pageSource;
|
||||
// if we're extracting data from a SPA, we must wait for the selector
|
||||
if (waitForSelector != null) {
|
||||
const selectorTimeout = options?.puppeteerSelectorTimeout ?? options?.puppeteerTimeout ?? 30_000;
|
||||
await page.waitForSelector(waitForSelector, { timeout: selectorTimeout });
|
||||
@@ -133,15 +145,22 @@ export default async function execute(url, waitForSelector, options) {
|
||||
|
||||
if (botDetected(pageSource, statusCode)) {
|
||||
logger.warn('We have been detected as a bot :-/ Tried url: => ', url);
|
||||
|
||||
if (options != null && options.name != null) {
|
||||
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT + '_' + options.name);
|
||||
} else {
|
||||
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
|
||||
}
|
||||
|
||||
result = null;
|
||||
} else {
|
||||
result = pageSource || (await page.content());
|
||||
}
|
||||
} catch (error) {
|
||||
if (error?.name?.includes('Timeout')) {
|
||||
logger.debug('Error executing with puppeteer executor', error);
|
||||
logger.debug('Error executing with CloakBrowser executor', error);
|
||||
} else {
|
||||
logger.warn('Error executing with puppeteer executor', error);
|
||||
logger.warn('Error executing with CloakBrowser executor', error);
|
||||
}
|
||||
result = null;
|
||||
} finally {
|
||||
|
||||
@@ -141,6 +141,43 @@ const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
|
||||
'barrierefreie-wohnung-mieten': { equipment: ['handicappedaccessible'] },
|
||||
};
|
||||
|
||||
// SEO-optimized rental paths used by the ImmoScout web UI when the user
|
||||
// configures a maximum warmrent. Example: "wohnung-bis-800-euro-warm" means
|
||||
// "apartment for rent up to 800 EUR warmrent". The web UI generates these
|
||||
// paths instead of explicit `price` / `pricetype` query parameters.
|
||||
// Note: only the warmrent variant uses an SEO slug; max coldrent searches
|
||||
// use the regular "wohnung-mieten" path with explicit `price` and
|
||||
// `pricetype=rentpermonth` query params, which the existing translator
|
||||
// already handles.
|
||||
const SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE = {
|
||||
wohnung: 'apartmentrent',
|
||||
haus: 'houserent',
|
||||
};
|
||||
const SEO_MAX_WARMRENT_PATH_PATTERN = /^(?<type>wohnung|haus)-bis-(?<price>\d+)-euro-warm$/;
|
||||
|
||||
/**
|
||||
* Parses SEO-optimized ImmoScout web paths that encode a maximum warmrent, such
|
||||
* as "wohnung-bis-800-euro-warm". Returns the corresponding mobile API real
|
||||
* estate type and the implicit price/pricetype parameters, or null if the path
|
||||
* does not match the known SEO max-warmrent pattern.
|
||||
*
|
||||
* @param {string} realTypeKey The last segment of the URL path.
|
||||
* @returns {{ realType: string, additionalParams: Record<string, string> } | null}
|
||||
*/
|
||||
function parseSeoMaxWarmrentPath(realTypeKey) {
|
||||
const match = realTypeKey.match(SEO_MAX_WARMRENT_PATH_PATTERN);
|
||||
if (!match) return null;
|
||||
|
||||
const { type, price } = match.groups;
|
||||
return {
|
||||
realType: SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE[type],
|
||||
additionalParams: {
|
||||
price: `-${price}`,
|
||||
pricetype: 'calculatedtotalrent',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function convertWebToMobile(webUrl) {
|
||||
let url;
|
||||
try {
|
||||
@@ -164,14 +201,17 @@ export function convertWebToMobile(webUrl) {
|
||||
additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey];
|
||||
realType = REAL_ESTATE_TYPE['wohnung-mieten'];
|
||||
} else {
|
||||
throw new Error(`Real estate type not found: ${realTypeKey}`);
|
||||
// Test for SEO max-warmrent path, e.g. "wohnung-bis-800-euro-warm"
|
||||
const seoMaxWarmrent = parseSeoMaxWarmrentPath(realTypeKey);
|
||||
if (seoMaxWarmrent) {
|
||||
realType = seoMaxWarmrent.realType;
|
||||
additionalParamsFromWebPath = seoMaxWarmrent.additionalParams;
|
||||
} else {
|
||||
throw new Error(`Real estate type not found: ${realTypeKey}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (segments.includes('shape')) {
|
||||
throw new Error('Shape is currently not supported using Immoscout');
|
||||
}
|
||||
|
||||
const { query: rawParams } = queryString.parseUrl(webUrl, { arrayFormat: 'comma' });
|
||||
const webParams = Object.fromEntries(
|
||||
Object.entries(rawParams).filter(([key]) => key !== 'enteredFrom' && PARAM_NAME_MAP[key]),
|
||||
@@ -179,18 +219,31 @@ export function convertWebToMobile(webUrl) {
|
||||
|
||||
const geocodes = `/${segments.slice(2, segments.length - 1).join('/')}`;
|
||||
const isRadius = segments.includes('radius');
|
||||
const isShape = segments.includes('shape');
|
||||
const mobileParams = {
|
||||
searchType: isRadius ? 'radius' : 'region',
|
||||
searchType: isRadius ? 'radius' : isShape ? 'shape' : 'region',
|
||||
realestatetype: realType,
|
||||
...(isRadius ? {} : { geocodes }),
|
||||
...(isRadius || isShape ? {} : { geocodes }),
|
||||
...additionalParamsFromWebPath,
|
||||
};
|
||||
|
||||
if (isShape && !webParams.shape) {
|
||||
throw new Error('Shape search URL is missing the required "shape" query parameter');
|
||||
}
|
||||
|
||||
if (isShape && webParams.shape) {
|
||||
const browserShape = webParams.shape;
|
||||
const normalized = browserShape.replace(/\.\./g, '==').replace(/\./g, '=');
|
||||
const polyline = Buffer.from(normalized, 'base64').toString('utf-8');
|
||||
mobileParams.shape = polyline;
|
||||
}
|
||||
|
||||
if (webParams.geocoordinates) {
|
||||
mobileParams.geocoordinates = webParams.geocoordinates;
|
||||
}
|
||||
|
||||
for (const [key, val] of Object.entries(webParams)) {
|
||||
if (key === 'shape') continue;
|
||||
if (key === 'equipment') {
|
||||
const items = [].concat(val).flatMap((v) => `${v}`.split(','));
|
||||
const currentEquipmentParams = mobileParams[PARAM_NAME_MAP[key]];
|
||||
|
||||
@@ -14,6 +14,7 @@ import * as similarityCache from '../similarity-check/similarityCache.js';
|
||||
import { isRunning, markFinished, markRunning } from './run-state.js';
|
||||
import { sendToUsers } from '../sse/sse-broker.js';
|
||||
import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
|
||||
import { getSettings } from '../storage/settingsStorage.js';
|
||||
|
||||
/**
|
||||
* Initializes the job execution service.
|
||||
@@ -160,6 +161,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
|
||||
}
|
||||
let browser;
|
||||
try {
|
||||
// Read the proxy live (not from the startup snapshot) so changing it in the
|
||||
// UI takes effect on the next run without a backend restart. An empty value
|
||||
// disables the proxy. Routing the headless browser through a (German
|
||||
// residential) proxy avoids datacenter-IP based bot detection on the
|
||||
// Puppeteer-based providers (immowelt, immonet, kleinanzeigen, ...).
|
||||
const liveSettings = await getSettings();
|
||||
const proxyUrl = typeof liveSettings?.proxyUrl === 'string' ? liveSettings.proxyUrl.trim() : '';
|
||||
|
||||
const jobProviders = job.provider.filter(
|
||||
(p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
|
||||
);
|
||||
@@ -168,14 +177,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
|
||||
const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
|
||||
matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
|
||||
|
||||
if (browser && !browser.isConnected()) {
|
||||
if (browser && !browser.connected) {
|
||||
logger.debug('Browser is disconnected, nullifying to launch a new one.');
|
||||
await puppeteerExtractor.closeBrowser(browser);
|
||||
browser = null;
|
||||
}
|
||||
|
||||
if (!browser && matchedProvider.config.getListings == null) {
|
||||
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
|
||||
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, proxyUrl ? { proxyUrl } : {});
|
||||
}
|
||||
|
||||
await new FredyPipelineExecutioner(matchedProvider.config, job, prov.id, similarityCache, browser).execute();
|
||||
|
||||
@@ -3,10 +3,27 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { nullOrEmpty } from '../../utils.js';
|
||||
import { nullOrEmpty, fromJson } from '../../utils.js';
|
||||
import SqliteConnection from './SqliteConnection.js';
|
||||
import { nanoid } from 'nanoid';
|
||||
|
||||
/**
|
||||
* Parse the JSON `status` column of a listing row in place.
|
||||
*
|
||||
* The DB stores status as a JSON payload `{ status, setAt }` (or NULL).
|
||||
* Consumers expect an object/null, so we normalize before returning.
|
||||
*
|
||||
* @param {Object|null|undefined} row - A raw row from the listings table.
|
||||
* @returns {Object|null|undefined} The same row with `status` parsed.
|
||||
*/
|
||||
const parseListingStatus = (row) => {
|
||||
if (row == null) return row;
|
||||
if (typeof row.status === 'string') {
|
||||
row.status = fromJson(row.status, null);
|
||||
}
|
||||
return row;
|
||||
};
|
||||
|
||||
/**
|
||||
* Return a list of known listing hashes for a given job and provider.
|
||||
* Useful to de-duplicate before inserting new listings.
|
||||
@@ -214,6 +231,8 @@ export const storeListings = (jobId, providerId, listings) => {
|
||||
longitude: item.longitude || null,
|
||||
};
|
||||
stmt.run(params);
|
||||
// Propagate the DB primary key back so downstream pipeline steps use the correct id
|
||||
item.id = params.id;
|
||||
}
|
||||
});
|
||||
|
||||
@@ -242,6 +261,7 @@ export const storeListings = (jobId, providerId, listings) => {
|
||||
* @param {object} [params.jobNameFilter]
|
||||
* @param {object} [params.providerFilter]
|
||||
* @param {object} [params.watchListFilter]
|
||||
* @param {('applied'|'rejected'|'accepted'|'none')} [params.statusFilter] - Filter by listing status. 'none' matches NULL.
|
||||
* @param {string|null} [params.sortField=null] - One of: 'created_at','price','size','provider','title'.
|
||||
* @param {('asc'|'desc')} [params.sortDir='asc']
|
||||
* @param {number} [params.createdAfter] - Only include listings created at or after this unix timestamp (ms).
|
||||
@@ -258,6 +278,7 @@ export const queryListings = ({
|
||||
jobIdFilter,
|
||||
providerFilter,
|
||||
watchListFilter,
|
||||
statusFilter,
|
||||
freeTextFilter,
|
||||
sortField = null,
|
||||
sortDir = 'asc',
|
||||
@@ -315,6 +336,18 @@ export const queryListings = ({
|
||||
} else if (watchListFilter === false) {
|
||||
whereParts.push('(wl.id IS NULL)');
|
||||
}
|
||||
// statusFilter: 'applied'|'rejected'|'accepted' -> equality on JSON status field; 'none' -> NULL.
|
||||
// The status column is a JSON payload `{ status, setAt }`, so we extract the inner
|
||||
// status string for comparison instead of matching the raw text.
|
||||
if (statusFilter === 'none') {
|
||||
whereParts.push('(l.status IS NULL)');
|
||||
} else if (
|
||||
typeof statusFilter === 'string' &&
|
||||
['applied', 'rejected', 'accepted'].includes(statusFilter.toLowerCase())
|
||||
) {
|
||||
params.statusValue = statusFilter.toLowerCase();
|
||||
whereParts.push(`(json_extract(l.status, '$.status') = @statusValue)`);
|
||||
}
|
||||
// Time range filters (unix timestamps in milliseconds)
|
||||
if (Number.isFinite(createdAfter) && createdAfter > 0) {
|
||||
params.createdAfter = createdAfter;
|
||||
@@ -389,7 +422,7 @@ export const queryListings = ({
|
||||
params,
|
||||
);
|
||||
|
||||
return { totalNumber, page: safePage, result: rows };
|
||||
return { totalNumber, page: safePage, result: rows.map(parseListingStatus) };
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -417,9 +450,10 @@ export const deleteListingsByJobId = (jobId, hardDelete = false) => {
|
||||
};
|
||||
|
||||
/**
|
||||
* Delete listings by a list of listing IDs.
|
||||
* Delete listings by a list of listing IDs (the nanoid primary key stored in the `id` column).
|
||||
* Used by API routes that receive row IDs from the client.
|
||||
*
|
||||
* @param {string[]} ids - Array of listing IDs to delete.
|
||||
* @param {string[]} ids - Array of DB row IDs to delete.
|
||||
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
|
||||
* @returns {any} The result from SqliteConnection.execute.
|
||||
*/
|
||||
@@ -623,7 +657,7 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
|
||||
if (!isAdmin) {
|
||||
whereScoping = `AND (j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`;
|
||||
}
|
||||
return (
|
||||
return parseListingStatus(
|
||||
SqliteConnection.query(
|
||||
`SELECT l.*, j.name AS job_name, CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END AS isWatched
|
||||
FROM listings l
|
||||
@@ -631,10 +665,57 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
|
||||
LEFT JOIN watch_list wl ON wl.listing_id = l.id AND wl.user_id = @userId
|
||||
WHERE l.id = @id AND l.manually_deleted = 0 ${whereScoping}`,
|
||||
params,
|
||||
)[0] || null
|
||||
)[0] || null,
|
||||
);
|
||||
};
|
||||
|
||||
/**
|
||||
* Set or clear the notes attached to a single listing.
|
||||
*
|
||||
* Empty strings are normalized to NULL so the DB doesn't keep meaningless
|
||||
* whitespace and queries can filter "has notes" with a simple IS NOT NULL.
|
||||
*
|
||||
* @param {string} id - The listing ID.
|
||||
* @param {string|null} notes - The note text to store, or null/empty to clear.
|
||||
* @returns {number} Number of rows affected (0 if listing not found).
|
||||
*/
|
||||
export const setListingNotes = (id, notes) => {
|
||||
if (!id) return 0;
|
||||
const trimmed = typeof notes === 'string' ? notes.trim() : null;
|
||||
const value = trimmed && trimmed.length > 0 ? trimmed : null;
|
||||
const res = SqliteConnection.execute(`UPDATE listings SET notes = @notes WHERE id = @id`, {
|
||||
id,
|
||||
notes: value,
|
||||
});
|
||||
return res?.changes ?? 0;
|
||||
};
|
||||
|
||||
/**
|
||||
* Set or clear the status of a single listing.
|
||||
*
|
||||
* The status column stores a JSON payload `{ status, setAt }` so consumers
|
||||
* can show both the user's decision and when it was made. Passing `null`
|
||||
* clears the column.
|
||||
*
|
||||
* @param {string} id - The listing ID.
|
||||
* @param {('applied'|'rejected'|'accepted'|null)} status - New status, or null to clear.
|
||||
* @returns {number} Number of rows affected (0 if listing not found).
|
||||
*/
|
||||
export const setListingStatus = (id, status) => {
|
||||
if (!id) return 0;
|
||||
const allowed = ['applied', 'rejected', 'accepted'];
|
||||
const normalized = status == null ? null : String(status).toLowerCase();
|
||||
if (normalized != null && !allowed.includes(normalized)) {
|
||||
throw new Error(`Invalid listing status: ${status}`);
|
||||
}
|
||||
const payload = normalized == null ? null : JSON.stringify({ status: normalized, setAt: Date.now() });
|
||||
const res = SqliteConnection.execute(`UPDATE listings SET status = @status WHERE id = @id`, {
|
||||
id,
|
||||
status: payload,
|
||||
});
|
||||
return res?.changes ?? 0;
|
||||
};
|
||||
|
||||
/**
|
||||
* Resets geocoordinates and distance for all listings related to a user.
|
||||
*
|
||||
|
||||
11
lib/services/storage/migrations/sql/18.add-listing-status.js
Normal file
11
lib/services/storage/migrations/sql/18.add-listing-status.js
Normal file
@@ -0,0 +1,11 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
export function up(db) {
|
||||
db.exec(`
|
||||
ALTER TABLE listings ADD COLUMN status JSON;
|
||||
CREATE INDEX IF NOT EXISTS idx_listings_status ON listings (json_extract(status, '$.status'));
|
||||
`);
|
||||
}
|
||||
10
lib/services/storage/migrations/sql/19.add-listing-notes.js
Normal file
10
lib/services/storage/migrations/sql/19.add-listing-notes.js
Normal file
@@ -0,0 +1,10 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
export function up(db) {
|
||||
db.exec(`
|
||||
ALTER TABLE listings ADD COLUMN notes TEXT;
|
||||
`);
|
||||
}
|
||||
@@ -47,6 +47,25 @@ export const deleteWatch = (listingId, userId) => {
|
||||
return { deleted: Boolean(res?.changes) };
|
||||
};
|
||||
|
||||
/**
|
||||
* Ensure a watch entry exists. Does not toggle; safe to call when row may already exist.
|
||||
* Used by the status endpoint to auto-watch a listing when a status is set.
|
||||
* @param {string} listingId
|
||||
* @param {string} userId
|
||||
* @returns {{watched:boolean}}
|
||||
*/
|
||||
export const ensureWatch = (listingId, userId) => {
|
||||
if (!listingId || !userId) return { watched: false };
|
||||
const { created } = createWatch(listingId, userId);
|
||||
if (created) return { watched: true };
|
||||
const exists =
|
||||
SqliteConnection.query(
|
||||
`SELECT 1 AS ok FROM watch_list WHERE listing_id = @listing_id AND user_id = @user_id LIMIT 1`,
|
||||
{ listing_id: listingId, user_id: userId },
|
||||
).length > 0;
|
||||
return { watched: exists };
|
||||
};
|
||||
|
||||
/**
|
||||
* Toggle a watch entry. If exists -> delete, otherwise create.
|
||||
* @param {string} listingId
|
||||
|
||||
51
package.json
51
package.json
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "fredy",
|
||||
"version": "21.1.1",
|
||||
"version": "22.3.1",
|
||||
"description": "[F]ind [R]eal [E]states [d]amn eas[y].",
|
||||
"scripts": {
|
||||
"prepare": "husky",
|
||||
@@ -62,9 +62,9 @@
|
||||
"Firefox ESR"
|
||||
],
|
||||
"dependencies": {
|
||||
"@douyinfe/semi-icons": "^2.96.0",
|
||||
"@douyinfe/semi-ui": "2.96.0",
|
||||
"@douyinfe/semi-ui-19": "^2.96.0",
|
||||
"@douyinfe/semi-icons": "^2.99.3",
|
||||
"@douyinfe/semi-ui": "2.99.3",
|
||||
"@douyinfe/semi-ui-19": "^2.99.3",
|
||||
"@fastify/cookie": "^11.0.2",
|
||||
"@fastify/helmet": "^13.0.2",
|
||||
"@fastify/session": "^11.1.1",
|
||||
@@ -73,11 +73,12 @@
|
||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||
"@sendgrid/mail": "8.1.6",
|
||||
"@turf/boolean-point-in-polygon": "^7.3.5",
|
||||
"@vitejs/plugin-react": "6.0.1",
|
||||
"@vitejs/plugin-react": "6.0.2",
|
||||
"adm-zip": "^0.5.17",
|
||||
"better-sqlite3": "^12.9.0",
|
||||
"better-sqlite3": "^12.10.0",
|
||||
"chart.js": "^4.5.1",
|
||||
"cheerio": "^1.2.0",
|
||||
"cloakbrowser": "^0.3.31",
|
||||
"fastify": "^5.8.5",
|
||||
"handlebars": "4.7.9",
|
||||
"maplibre-gl": "^5.24.0",
|
||||
@@ -85,43 +86,41 @@
|
||||
"node-cron": "^4.2.1",
|
||||
"node-fetch": "3.3.2",
|
||||
"node-mailjet": "6.0.11",
|
||||
"nodemailer": "^8.0.7",
|
||||
"nodemailer": "^8.0.10",
|
||||
"p-throttle": "^8.1.0",
|
||||
"package-up": "^5.0.0",
|
||||
"puppeteer": "^24.42.0",
|
||||
"puppeteer-extra": "^3.3.6",
|
||||
"puppeteer-extra-plugin-stealth": "^2.11.2",
|
||||
"query-string": "9.3.1",
|
||||
"react": "19.2.5",
|
||||
"puppeteer-core": "^25.1.0",
|
||||
"query-string": "9.4.0",
|
||||
"react": "19.2.7",
|
||||
"react-chartjs-2": "^5.3.1",
|
||||
"react-dom": "19.2.5",
|
||||
"react-dom": "19.2.7",
|
||||
"react-range-slider-input": "^3.3.5",
|
||||
"react-router": "7.14.2",
|
||||
"react-router-dom": "7.14.2",
|
||||
"resend": "^6.12.2",
|
||||
"semver": "^7.7.4",
|
||||
"react-router": "7.16.0",
|
||||
"react-router-dom": "7.16.0",
|
||||
"resend": "^6.12.4",
|
||||
"semver": "^7.8.1",
|
||||
"slack": "11.0.2",
|
||||
"vite": "8.0.10",
|
||||
"vite": "8.0.16",
|
||||
"x-var": "^3.0.1",
|
||||
"zustand": "^5.0.12"
|
||||
"zustand": "^5.0.14"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@babel/core": "7.29.0",
|
||||
"@babel/eslint-parser": "7.28.6",
|
||||
"@babel/preset-env": "7.29.3",
|
||||
"@babel/preset-react": "7.28.5",
|
||||
"@babel/core": "7.29.7",
|
||||
"@babel/eslint-parser": "7.29.7",
|
||||
"@babel/preset-env": "7.29.7",
|
||||
"@babel/preset-react": "7.29.7",
|
||||
"@eslint/js": "^10.0.1",
|
||||
"chalk": "^5.6.2",
|
||||
"eslint": "10.3.0",
|
||||
"eslint": "10.4.1",
|
||||
"eslint-config-prettier": "10.1.8",
|
||||
"eslint-plugin-react": "7.37.5",
|
||||
"globals": "^17.6.0",
|
||||
"history": "5.3.0",
|
||||
"husky": "9.1.7",
|
||||
"less": "4.6.4",
|
||||
"lint-staged": "16.4.0",
|
||||
"lint-staged": "17.0.7",
|
||||
"nodemon": "^3.1.14",
|
||||
"prettier": "3.8.3",
|
||||
"vitest": "^4.1.5"
|
||||
"vitest": "^4.1.8"
|
||||
}
|
||||
}
|
||||
|
||||
18
test/globalSetup.js
Normal file
18
test/globalSetup.js
Normal file
@@ -0,0 +1,18 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { ensureValidBinary } from '../lib/services/ensureValidBinary.js';
|
||||
|
||||
/**
|
||||
* Vitest global setup — runs once in the main process before any workers start.
|
||||
* Downloads and validates the CloakBrowser stealth Chromium binary.
|
||||
* ensureValidBinary() also removes and re-downloads partial/corrupt installations
|
||||
* so tests never fail with "Invalid file descriptor to ICU data received".
|
||||
* Skipped in offline mode because the browser is fully mocked there.
|
||||
*/
|
||||
export async function setup() {
|
||||
if (process.env.TEST_MODE === 'offline') return;
|
||||
await ensureValidBinary();
|
||||
}
|
||||
@@ -32,4 +32,7 @@ export const deletedIds = [];
|
||||
export const deleteListingsById = (ids) => {
|
||||
deletedIds.push(...ids);
|
||||
};
|
||||
export const deleteListingsByHash = (hashes) => {
|
||||
deletedIds.push(...hashes);
|
||||
};
|
||||
/* eslint-enable no-unused-vars */
|
||||
|
||||
360
test/notification/telegram.test.js
Normal file
360
test/notification/telegram.test.js
Normal file
@@ -0,0 +1,360 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
// Mock external deps BEFORE importing the module under test.
|
||||
vi.mock('node-fetch', () => ({ default: vi.fn() }));
|
||||
vi.mock('../../lib/services/storage/jobStorage.js', () => ({
|
||||
getJob: (jobKey) => ({ id: jobKey, name: jobKey }),
|
||||
}));
|
||||
vi.mock('../../lib/services/markdown.js', () => ({
|
||||
markdown2Html: () => '',
|
||||
}));
|
||||
|
||||
// Helpers to build mock fetch responses.
|
||||
function jsonOk(body = { ok: true }) {
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
text: async () => JSON.stringify(body),
|
||||
};
|
||||
}
|
||||
|
||||
function jsonErr(status, body) {
|
||||
return {
|
||||
ok: false,
|
||||
status,
|
||||
text: async () => JSON.stringify(body),
|
||||
};
|
||||
}
|
||||
|
||||
function imageOk(bytes = new Uint8Array([0xff, 0xd8, 0xff])) {
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: {
|
||||
get: (h) => {
|
||||
const k = h.toLowerCase();
|
||||
if (k === 'content-type') return 'image/jpeg';
|
||||
if (k === 'content-length') return String(bytes.byteLength);
|
||||
return null;
|
||||
},
|
||||
},
|
||||
arrayBuffer: async () => bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength),
|
||||
};
|
||||
}
|
||||
|
||||
// Globals are mocked too so buildPhotoFormData (which uses global fetch) can be
|
||||
// intercepted by the same single mock.
|
||||
let mockNodeFetch;
|
||||
let mockGlobalFetch;
|
||||
let send;
|
||||
|
||||
beforeEach(async () => {
|
||||
// Reset modules to get a fresh import with our mocks applied.
|
||||
vi.resetModules();
|
||||
const nodeFetchMod = await import('node-fetch');
|
||||
mockNodeFetch = nodeFetchMod.default;
|
||||
mockNodeFetch.mockReset();
|
||||
|
||||
mockGlobalFetch = vi.fn();
|
||||
vi.stubGlobal('fetch', mockGlobalFetch);
|
||||
|
||||
({ send } = await import('../../lib/notification/adapter/telegram.js'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
const baseConfig = {
|
||||
id: 'telegram',
|
||||
fields: { token: 'TKN', chatId: '999' },
|
||||
};
|
||||
|
||||
describe('telegram send() - HTTP URL path (default for .jpg / .png)', () => {
|
||||
it('POSTs JSON to sendPhoto for a .jpg image URL', async () => {
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 'Listing',
|
||||
link: 'https://example.com/a',
|
||||
address: 'Addr',
|
||||
price: '500€',
|
||||
size: '50m²',
|
||||
image: 'https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
const [url, opts] = mockNodeFetch.mock.calls[0];
|
||||
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
|
||||
expect(opts.method).toBe('post');
|
||||
expect(opts.headers?.['Content-Type']).toBe('application/json');
|
||||
const body = JSON.parse(opts.body);
|
||||
expect(body.chat_id).toBe('999');
|
||||
expect(body.photo).toBe('https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394');
|
||||
expect(body.parse_mode).toBe('HTML');
|
||||
});
|
||||
|
||||
it('does NOT pre-fetch the image when using HTTP URL path', async () => {
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/x.jpg',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
// global fetch (used by buildPhotoFormData) must not be called
|
||||
expect(mockGlobalFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('falls back to sendMessage when sendPhoto fails', async () => {
|
||||
mockNodeFetch
|
||||
.mockResolvedValueOnce(jsonErr(400, { ok: false, description: 'boom' }))
|
||||
.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/x.jpg',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
|
||||
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendPhoto');
|
||||
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
});
|
||||
});
|
||||
|
||||
describe('telegram send() - multipart path (.webp URLs)', () => {
|
||||
it('pre-fetches the image then POSTs FormData to sendPhoto for a .webp URL', async () => {
|
||||
// 1st: GET image via global fetch
|
||||
mockGlobalFetch.mockResolvedValueOnce(imageOk());
|
||||
// 2nd: POST sendPhoto via node-fetch
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 'Listing',
|
||||
link: 'https://example.com/a',
|
||||
address: 'Addr',
|
||||
price: '500€',
|
||||
size: '50m²',
|
||||
image: 'https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
// image was fetched
|
||||
expect(mockGlobalFetch).toHaveBeenCalledTimes(1);
|
||||
expect(mockGlobalFetch.mock.calls[0][0]).toBe('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394');
|
||||
|
||||
// sendPhoto called via node-fetch with FormData
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
const [url, opts] = mockNodeFetch.mock.calls[0];
|
||||
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
|
||||
expect(opts.method).toBe('post');
|
||||
expect(opts.body).toBeInstanceOf(FormData);
|
||||
// No explicit Content-Type header - fetch sets multipart boundary itself
|
||||
expect(opts.headers).toBeUndefined();
|
||||
expect(opts.body.get('chat_id')).toBe('999');
|
||||
expect(opts.body.get('parse_mode')).toBe('HTML');
|
||||
const photo = opts.body.get('photo');
|
||||
expect(photo).toBeTruthy();
|
||||
expect(photo.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('falls back to sendMessage when the image pre-fetch fails for a .webp URL', async () => {
|
||||
// image fetch fails (404 from CDN)
|
||||
mockGlobalFetch.mockResolvedValueOnce({
|
||||
ok: false,
|
||||
status: 404,
|
||||
headers: { get: () => null },
|
||||
arrayBuffer: async () => new ArrayBuffer(0),
|
||||
});
|
||||
// then sendMessage succeeds via node-fetch
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/gone.webp',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
});
|
||||
|
||||
it('falls back to sendMessage when multipart sendPhoto returns a Telegram error', async () => {
|
||||
mockGlobalFetch.mockResolvedValueOnce(imageOk());
|
||||
mockNodeFetch
|
||||
.mockResolvedValueOnce(jsonErr(400, { description: 'broke' })) // multipart sendPhoto
|
||||
.mockResolvedValueOnce(jsonOk()); // sendMessage fallback
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/x.webp',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
|
||||
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
});
|
||||
});
|
||||
|
||||
describe('telegram send() - mixed batch (regression-safety)', () => {
|
||||
it('handles a batch with both .jpg and .webp - jpg uses URL, webp uses multipart', async () => {
|
||||
// .webp image fetch
|
||||
mockGlobalFetch.mockResolvedValueOnce(imageOk());
|
||||
// both sendPhoto calls succeed
|
||||
mockNodeFetch
|
||||
.mockResolvedValueOnce(jsonOk()) // could be either listing first
|
||||
.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'jpg-listing',
|
||||
title: 'a',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/a.jpg',
|
||||
},
|
||||
{
|
||||
id: 'webp-listing',
|
||||
title: 'b',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/b.webp',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockGlobalFetch).toHaveBeenCalledTimes(1); // only webp pre-fetches
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
|
||||
|
||||
// Verify one call had FormData and one had JSON body
|
||||
const bodies = mockNodeFetch.mock.calls.map((c) => c[1].body);
|
||||
const hasFormData = bodies.some((b) => b instanceof FormData);
|
||||
const hasJson = bodies.some((b) => typeof b === 'string' && b.startsWith('{'));
|
||||
expect(hasFormData).toBe(true);
|
||||
expect(hasJson).toBe(true);
|
||||
});
|
||||
|
||||
it('uses sendMessage (not sendPhoto) when image is null', async () => {
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: null,
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
expect(mockGlobalFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('telegram send() - config validation', () => {
|
||||
it('throws when telegram adapter config is missing', () => {
|
||||
expect(() =>
|
||||
send({
|
||||
serviceName: 's',
|
||||
newListings: [],
|
||||
notificationConfig: [],
|
||||
jobKey: 'k',
|
||||
}),
|
||||
).toThrow(/configuration missing/);
|
||||
});
|
||||
|
||||
it('throws when token or chatId is missing', () => {
|
||||
expect(() =>
|
||||
send({
|
||||
serviceName: 's',
|
||||
newListings: [],
|
||||
notificationConfig: [{ id: 'telegram', fields: { token: '' } }],
|
||||
jobKey: 'k',
|
||||
}),
|
||||
).toThrow(/token.*chatId/);
|
||||
});
|
||||
});
|
||||
287
test/notification/telegramPhotoUploader.test.js
Normal file
287
test/notification/telegramPhotoUploader.test.js
Normal file
@@ -0,0 +1,287 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
import { shouldUseMultipart, buildPhotoFormData } from '../../lib/notification/adapter/telegramPhotoUploader.js';
|
||||
|
||||
describe('shouldUseMultipart', () => {
|
||||
it('returns true for .webp URL with query string', () => {
|
||||
expect(shouldUseMultipart('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true for .webp URL without query string', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.webp')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true for uppercase .WEBP extension', () => {
|
||||
expect(shouldUseMultipart('https://example.com/IMG.WEBP?x=1')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false for .jpg URL with query string', () => {
|
||||
expect(shouldUseMultipart('https://mms.immowelt.de/a/b/c/d/xyz.jpg?ci_seal=hash&w=525&h=394')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for .jpeg URL', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.jpeg')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for .png URL with query string', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.png?w=100')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for .gif URL', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.gif')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for null', () => {
|
||||
expect(shouldUseMultipart(null)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for undefined', () => {
|
||||
expect(shouldUseMultipart(undefined)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for empty string', () => {
|
||||
expect(shouldUseMultipart('')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for malformed URL', () => {
|
||||
expect(shouldUseMultipart('not a url')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for URL where webp is in the query but not the path', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.jpg?format=webp')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for URL with no extension at all', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for non-https schemes', () => {
|
||||
// file/data/ftp URLs should not be relevant; safer to skip multipart
|
||||
expect(shouldUseMultipart('http://example.com/photo.webp')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildPhotoFormData', () => {
|
||||
let mockFetch;
|
||||
|
||||
beforeEach(() => {
|
||||
mockFetch = vi.fn();
|
||||
vi.stubGlobal('fetch', mockFetch);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
function makeImageResponse({ contentType = 'image/jpeg', bytes = new Uint8Array([0xff, 0xd8, 0xff]) } = {}) {
|
||||
const buf = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: {
|
||||
get: (h) =>
|
||||
h.toLowerCase() === 'content-type'
|
||||
? contentType
|
||||
: h.toLowerCase() === 'content-length'
|
||||
? String(bytes.byteLength)
|
||||
: null,
|
||||
},
|
||||
arrayBuffer: async () => buf,
|
||||
};
|
||||
}
|
||||
|
||||
it('fetches image with Accept header that excludes webp so the CDN transcodes to JPEG', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
await buildPhotoFormData({
|
||||
chatId: '123',
|
||||
imageUrl: 'https://example.com/photo.webp',
|
||||
caption: 'hi',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(mockFetch).toHaveBeenCalledTimes(1);
|
||||
const [url, opts] = mockFetch.mock.calls[0];
|
||||
expect(url).toBe('https://example.com/photo.webp');
|
||||
expect(opts?.headers?.Accept || opts?.headers?.accept).toMatch(/image\/jpeg/);
|
||||
expect(opts?.headers?.Accept || opts?.headers?.accept).not.toMatch(/image\/webp/);
|
||||
});
|
||||
|
||||
it('returns FormData containing chat_id, caption, parse_mode, and photo fields', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '12345',
|
||||
imageUrl: 'https://example.com/abc.webp',
|
||||
caption: 'My caption',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd).toBeInstanceOf(FormData);
|
||||
expect(fd.get('chat_id')).toBe('12345');
|
||||
expect(fd.get('caption')).toBe('My caption');
|
||||
expect(fd.get('parse_mode')).toBe('HTML');
|
||||
const photo = fd.get('photo');
|
||||
expect(photo).toBeTruthy();
|
||||
// File-like (Blob); has a name and a size
|
||||
expect(typeof photo.name).toBe('string');
|
||||
expect(photo.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('uses a .jpg filename (Telegram uses URL/filename extension for type detection)', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
const photo = fd.get('photo');
|
||||
expect(photo.name).toMatch(/\.jpg$/i);
|
||||
});
|
||||
|
||||
it('includes message_thread_id when provided', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
messageThreadId: 42,
|
||||
});
|
||||
|
||||
expect(fd.get('message_thread_id')).toBe('42');
|
||||
});
|
||||
|
||||
it('omits message_thread_id when not provided', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd.get('message_thread_id')).toBeNull();
|
||||
});
|
||||
|
||||
it('omits parse_mode when not provided (plain text mode)', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
});
|
||||
|
||||
expect(fd.get('parse_mode')).toBeNull();
|
||||
});
|
||||
|
||||
it('throws when the image fetch returns non-200', async () => {
|
||||
mockFetch.mockResolvedValueOnce({
|
||||
ok: false,
|
||||
status: 404,
|
||||
headers: { get: () => null },
|
||||
arrayBuffer: async () => new ArrayBuffer(0),
|
||||
});
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/gone.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/404/);
|
||||
});
|
||||
|
||||
it('throws when the image fetch throws (network error)', async () => {
|
||||
mockFetch.mockRejectedValueOnce(new Error('ECONNREFUSED'));
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/x.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/ECONNREFUSED/);
|
||||
});
|
||||
|
||||
it('throws when the image exceeds 10 MB (Telegram multipart limit)', async () => {
|
||||
// 11 MB
|
||||
const big = new Uint8Array(11 * 1024 * 1024);
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes: big }));
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/huge.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/size|large|10/i);
|
||||
});
|
||||
|
||||
it('rejects early when content-length header advertises > 10 MB (avoids download)', async () => {
|
||||
mockFetch.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: {
|
||||
get: (h) => {
|
||||
const k = h.toLowerCase();
|
||||
if (k === 'content-type') return 'image/jpeg';
|
||||
if (k === 'content-length') return String(50 * 1024 * 1024);
|
||||
return null;
|
||||
},
|
||||
},
|
||||
arrayBuffer: async () => {
|
||||
throw new Error('should not be called - size check should reject first');
|
||||
},
|
||||
});
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/huge.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/size|large|10/i);
|
||||
});
|
||||
|
||||
it('accepts exactly 10 MB images (boundary)', async () => {
|
||||
const bytes = new Uint8Array(10 * 1024 * 1024);
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes }));
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/exact.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd.get('photo').size).toBe(10 * 1024 * 1024);
|
||||
});
|
||||
|
||||
it('coerces non-string chatId (number) to string in form data', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: 999,
|
||||
imageUrl: 'https://example.com/x.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd.get('chat_id')).toBe('999');
|
||||
});
|
||||
});
|
||||
@@ -38,6 +38,20 @@ async function tryReadFile(filepath) {
|
||||
}
|
||||
}
|
||||
|
||||
function withRealEstateType(data, realEstateType) {
|
||||
if (!realEstateType?.length || !Array.isArray(data?.resultListItems)) {
|
||||
return data;
|
||||
}
|
||||
|
||||
const cloned = typeof structuredClone === 'function' ? structuredClone(data) : JSON.parse(JSON.stringify(data));
|
||||
for (const item of cloned.resultListItems) {
|
||||
if (item?.type === 'EXPOSE_RESULT' && item?.item) {
|
||||
item.item.realEstateType = realEstateType;
|
||||
}
|
||||
}
|
||||
return cloned;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns fixture HTML for the given URL by mapping hostname → provider name,
|
||||
* then distinguishing list vs detail pages by comparing the URL path against
|
||||
@@ -83,7 +97,10 @@ export function buildFetchMock() {
|
||||
const raw = await tryReadFile(path.join(FIXTURES_DIR, 'immoscout_list.json'));
|
||||
listData = raw ? JSON.parse(raw) : { resultListItems: [] };
|
||||
}
|
||||
return { ok: true, status: 200, json: () => Promise.resolve(listData) };
|
||||
|
||||
const requestedType = new URL(urlStr).searchParams.get('realestatetype');
|
||||
const responseData = withRealEstateType(listData, requestedType);
|
||||
return { ok: true, status: 200, json: () => Promise.resolve(responseData) };
|
||||
}
|
||||
|
||||
if (urlStr.includes('api.mobile.immobilienscout24.de/expose/')) {
|
||||
|
||||
@@ -6,83 +6,89 @@
|
||||
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
|
||||
import { get } from '../mocks/mockNotification.js';
|
||||
import { providerConfig, mockFredy } from '../utils.js';
|
||||
import { expect, vi } from 'vitest';
|
||||
import { expect } from 'vitest';
|
||||
import * as provider from '../../lib/provider/immobilienDe.js';
|
||||
import * as mockStore from '../mocks/mockStore.js';
|
||||
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
|
||||
|
||||
// One browser shared across the whole suite so both requests (search + detail)
|
||||
// come from the same warm session, avoiding double cold-start bot detection.
|
||||
const TEST_TIMEOUT = 120_000;
|
||||
|
||||
describe('#immobilien.de testsuite()', () => {
|
||||
provider.init(providerConfig.immobilienDe, [], []);
|
||||
it('should test immobilien.de provider', async () => {
|
||||
const mockedJob = {
|
||||
id: 'test1',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
|
||||
const Fredy = await mockFredy();
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
|
||||
const listing = await fredy.execute();
|
||||
let browser;
|
||||
let liveListings;
|
||||
|
||||
if (listing == null || listing.length === 0) {
|
||||
throw new Error('Listings is empty!');
|
||||
}
|
||||
beforeAll(async () => {
|
||||
browser = await launchBrowser(providerConfig.immobilienDe.url);
|
||||
}, TEST_TIMEOUT);
|
||||
|
||||
expect(listing).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('immobilienDe');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.price).toContain('€');
|
||||
expect(notify.size).toContain('m²');
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.link).toContain('https://www.immobilien.de');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
afterAll(async () => {
|
||||
await closeBrowser(browser);
|
||||
});
|
||||
|
||||
it(
|
||||
'should test immobilien.de provider',
|
||||
async () => {
|
||||
const mockedJob = {
|
||||
id: 'test1',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
|
||||
const Fredy = await mockFredy();
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
|
||||
liveListings = await fredy.execute();
|
||||
|
||||
if (liveListings == null || liveListings.length === 0) {
|
||||
throw new Error('Listings is empty!');
|
||||
}
|
||||
|
||||
expect(liveListings).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('immobilienDe');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.price).toContain('€');
|
||||
expect(notify.size).toContain('m²');
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.link).toContain('https://www.immobilien.de');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
|
||||
describe('with provider_details enabled', () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
|
||||
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
|
||||
});
|
||||
it(
|
||||
'should enrich listings with details',
|
||||
async () => {
|
||||
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
// Call fetchDetails directly on the first live listing — no need to
|
||||
// re-scrape the search page. The shared browser keeps the session warm.
|
||||
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
|
||||
|
||||
it('should enrich listings with details', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
provider.init(providerConfig.immobilienDe, [], []);
|
||||
const mockedJob = { id: 'test1', notificationAdapter: null, specFilter: null, spatialFilter: null };
|
||||
|
||||
const fredy = new Fredy(
|
||||
provider.config,
|
||||
mockedJob,
|
||||
provider.metaInformation.id,
|
||||
{ checkAndAddEntry: () => false },
|
||||
undefined,
|
||||
);
|
||||
const listings = await fredy.execute();
|
||||
if (listings == null) return;
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
listings.forEach((listing) => {
|
||||
expect(listing.link).toContain('https://www.immobilien.de');
|
||||
expect(listing.address).toBeTypeOf('string');
|
||||
expect(listing.address).not.toBe('');
|
||||
// description may be null if selectors don't match yet - falls back gracefully
|
||||
if (listing.description != null) {
|
||||
expect(listing.description).toBeTypeOf('string');
|
||||
if (enriched == null) return;
|
||||
expect(enriched.link).toContain('https://www.immobilien.de');
|
||||
expect(enriched.address).toBeTypeOf('string');
|
||||
expect(enriched.address).not.toBe('');
|
||||
// description may be null if selectors don't match yet — falls back gracefully
|
||||
if (enriched.description != null) {
|
||||
expect(enriched.description).toBeTypeOf('string');
|
||||
}
|
||||
});
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -3,85 +3,85 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { expect, vi } from 'vitest';
|
||||
import { expect } from 'vitest';
|
||||
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
|
||||
import { mockFredy, providerConfig } from '../utils.js';
|
||||
import { get } from '../mocks/mockNotification.js';
|
||||
import * as provider from '../../lib/provider/immoscout.js';
|
||||
import * as mockStore from '../mocks/mockStore.js';
|
||||
|
||||
// immoscout uses the mobile REST API (fetch-based, no browser). Both tests share
|
||||
// the same module-level listings so the API is only queried once, avoiding
|
||||
// duplicate requests that could trigger rate-limiting.
|
||||
const TEST_TIMEOUT = 120_000;
|
||||
|
||||
describe('#immoscout provider testsuite()', () => {
|
||||
provider.init(providerConfig.immoscout, [], []);
|
||||
it('should test immoscout provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: '',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
|
||||
return await new Promise((resolve, reject) => {
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
|
||||
fredy.execute().then((listings) => {
|
||||
if (listings == null || listings.length === 0) {
|
||||
reject('Listings is empty!');
|
||||
return;
|
||||
}
|
||||
let liveListings;
|
||||
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
it(
|
||||
'should test immoscout provider',
|
||||
async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: '',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
|
||||
// check if there is at least one valid notification
|
||||
const hasValidNotification = notificationObj.payload.some((notify) => {
|
||||
return (
|
||||
typeof notify.id === 'string' &&
|
||||
typeof notify.price === 'string' &&
|
||||
notify.price.includes('€') &&
|
||||
typeof notify.size === 'string' &&
|
||||
notify.size.includes('m²') &&
|
||||
typeof notify.title === 'string' &&
|
||||
notify.title !== '' &&
|
||||
typeof notify.link === 'string' &&
|
||||
notify.link.includes('https://www.immobilienscout24.de/') &&
|
||||
typeof notify.address === 'string'
|
||||
);
|
||||
return await new Promise((resolve, reject) => {
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
|
||||
fredy.execute().then((listings) => {
|
||||
if (listings == null || listings.length === 0) {
|
||||
reject('Listings is empty!');
|
||||
return;
|
||||
}
|
||||
|
||||
liveListings = listings;
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
|
||||
// check if there is at least one valid notification
|
||||
const hasValidNotification = notificationObj.payload.some((notify) => {
|
||||
return (
|
||||
typeof notify.id === 'string' &&
|
||||
typeof notify.price === 'string' &&
|
||||
notify.price.includes('€') &&
|
||||
typeof notify.size === 'string' &&
|
||||
notify.size.includes('m²') &&
|
||||
typeof notify.title === 'string' &&
|
||||
notify.title !== '' &&
|
||||
typeof notify.link === 'string' &&
|
||||
notify.link.includes('https://www.immobilienscout24.de/') &&
|
||||
typeof notify.address === 'string'
|
||||
);
|
||||
});
|
||||
|
||||
expect(hasValidNotification).toBe(true);
|
||||
resolve();
|
||||
});
|
||||
|
||||
expect(hasValidNotification).toBe(true);
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
|
||||
describe('with provider_details enabled', () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
|
||||
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
|
||||
});
|
||||
it(
|
||||
'should enrich listings with details',
|
||||
async () => {
|
||||
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
// Call fetchDetails directly on the first live listing — no need to
|
||||
// re-query the search API. immoscout uses fetch (no browser).
|
||||
const enriched = await provider.config.fetchDetails(liveListings[0]);
|
||||
|
||||
it('should enrich listings with details', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
provider.init(providerConfig.immoscout, [], []);
|
||||
const mockedJob = { id: '', notificationAdapter: null, specFilter: null, spatialFilter: null };
|
||||
const fredy = new Fredy(
|
||||
provider.config,
|
||||
mockedJob,
|
||||
provider.metaInformation.id,
|
||||
{ checkAndAddEntry: () => false },
|
||||
undefined,
|
||||
);
|
||||
const listings = await fredy.execute();
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
listings.forEach((listing) => {
|
||||
expect(listing.description).toBeTypeOf('string');
|
||||
expect(listing.description).not.toBe('');
|
||||
});
|
||||
});
|
||||
expect(enriched).toBeTruthy();
|
||||
expect(enriched.description).toBeTypeOf('string');
|
||||
expect(enriched.description).not.toBe('');
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,87 +6,95 @@
|
||||
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
|
||||
import { get } from '../mocks/mockNotification.js';
|
||||
import { mockFredy, providerConfig } from '../utils.js';
|
||||
import { expect, vi } from 'vitest';
|
||||
import { expect } from 'vitest';
|
||||
import * as provider from '../../lib/provider/immowelt.js';
|
||||
import * as mockStore from '../mocks/mockStore.js';
|
||||
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
|
||||
|
||||
// One browser shared across the whole suite so both requests (search + detail)
|
||||
// come from the same warm session. Immowelt's CDN challenges cold sessions
|
||||
// aggressively; a shared warm browser prevents the second request from being
|
||||
// blocked as a bot hit.
|
||||
const TEST_TIMEOUT = 180_000;
|
||||
|
||||
describe('#immowelt testsuite()', () => {
|
||||
it('should test immowelt provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'immowelt',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
provider.init(providerConfig.immowelt, [], []);
|
||||
let browser;
|
||||
let liveListings;
|
||||
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
|
||||
beforeAll(async () => {
|
||||
browser = await launchBrowser(providerConfig.immowelt.url);
|
||||
}, TEST_TIMEOUT);
|
||||
|
||||
const listing = await fredy.execute();
|
||||
|
||||
if (listing == null || listing.length === 0) {
|
||||
throw new Error('Listings is empty!');
|
||||
}
|
||||
|
||||
expect(listing).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('immowelt');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
if (notify.price != null) {
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.price).toContain('€');
|
||||
}
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.size).toContain('m²');
|
||||
}
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.link).toContain('https://www.immowelt.de');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
afterAll(async () => {
|
||||
await closeBrowser(browser);
|
||||
});
|
||||
|
||||
it(
|
||||
'should test immowelt provider',
|
||||
async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'immowelt',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
provider.init(providerConfig.immowelt, [], []);
|
||||
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
|
||||
|
||||
liveListings = await fredy.execute();
|
||||
|
||||
if (liveListings == null || liveListings.length === 0) {
|
||||
throw new Error('Listings is empty!');
|
||||
}
|
||||
|
||||
expect(liveListings).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('immowelt');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
if (notify.price != null) {
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.price).toContain('€');
|
||||
}
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.size).toContain('m²');
|
||||
}
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.link).toContain('https://www.immowelt.de');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
|
||||
describe('with provider_details enabled', () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
|
||||
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
|
||||
});
|
||||
it(
|
||||
'should enrich listings with details',
|
||||
async () => {
|
||||
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
// Call fetchDetails directly on the first live listing — no need to
|
||||
// re-scrape the search page. The shared browser keeps the session warm.
|
||||
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
|
||||
|
||||
it('should enrich listings with details', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
provider.init(providerConfig.immowelt, [], []);
|
||||
const mockedJob = { id: 'immowelt', notificationAdapter: null, specFilter: null, spatialFilter: null };
|
||||
|
||||
const fredy = new Fredy(
|
||||
provider.config,
|
||||
mockedJob,
|
||||
provider.metaInformation.id,
|
||||
{ checkAndAddEntry: () => false },
|
||||
undefined,
|
||||
);
|
||||
const listings = await fredy.execute();
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
listings.forEach((listing) => {
|
||||
expect(listing.link).toContain('https://www.immowelt.de');
|
||||
expect(listing.address).toBeTypeOf('string');
|
||||
expect(listing.address).not.toBe('');
|
||||
expect(enriched).toBeTruthy();
|
||||
expect(enriched.link).toContain('https://www.immowelt.de');
|
||||
expect(enriched.address).toBeTypeOf('string');
|
||||
expect(enriched.address).not.toBe('');
|
||||
// description is enriched from the detail page; falls back gracefully if blocked
|
||||
if (listing.description != null) {
|
||||
expect(listing.description).toBeTypeOf('string');
|
||||
if (enriched.description != null) {
|
||||
expect(enriched.description).toBeTypeOf('string');
|
||||
}
|
||||
});
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,80 +6,88 @@
|
||||
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
|
||||
import { get } from '../mocks/mockNotification.js';
|
||||
import { mockFredy, providerConfig } from '../utils.js';
|
||||
import { expect, vi } from 'vitest';
|
||||
import { expect } from 'vitest';
|
||||
import * as provider from '../../lib/provider/kleinanzeigen.js';
|
||||
import * as mockStore from '../mocks/mockStore.js';
|
||||
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
|
||||
|
||||
// One browser shared across the whole suite so both requests (search + detail)
|
||||
// come from the same warm session. Kleinanzeigen rate-limits cold browser
|
||||
// sessions; a shared warm browser prevents the second request from being blocked.
|
||||
const TEST_TIMEOUT = 180_000;
|
||||
|
||||
describe('#kleinanzeigen testsuite()', () => {
|
||||
it('should test kleinanzeigen provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'kleinanzeigen',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
provider.init(providerConfig.kleinanzeigen, [], []);
|
||||
return await new Promise((resolve, reject) => {
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
|
||||
let browser;
|
||||
let liveListings;
|
||||
|
||||
fredy.execute().then((listing) => {
|
||||
if (listing == null || listing.length === 0) {
|
||||
reject('Listings is empty!');
|
||||
return;
|
||||
}
|
||||
beforeAll(async () => {
|
||||
browser = await launchBrowser(providerConfig.kleinanzeigen.url);
|
||||
}, TEST_TIMEOUT);
|
||||
|
||||
expect(listing).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('kleinanzeigen');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.link).toContain('https://www.kleinanzeigen.de');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
afterAll(async () => {
|
||||
await closeBrowser(browser);
|
||||
});
|
||||
|
||||
it(
|
||||
'should test kleinanzeigen provider',
|
||||
async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'kleinanzeigen',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
provider.init(providerConfig.kleinanzeigen, [], []);
|
||||
return await new Promise((resolve, reject) => {
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
|
||||
|
||||
fredy.execute().then((listing) => {
|
||||
if (listing == null || listing.length === 0) {
|
||||
reject('Listings is empty!');
|
||||
return;
|
||||
}
|
||||
|
||||
liveListings = listing;
|
||||
expect(listing).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('kleinanzeigen');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.link).toContain('https://www.kleinanzeigen.de');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
|
||||
describe('with provider_details enabled', () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
|
||||
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
|
||||
});
|
||||
it(
|
||||
'should enrich listings with details',
|
||||
async () => {
|
||||
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
// Call fetchDetails directly on the first live listing — no need to
|
||||
// re-scrape the search page. The shared browser keeps the session warm.
|
||||
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
|
||||
|
||||
it('should enrich listings with details', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
provider.init(providerConfig.kleinanzeigen, [], []);
|
||||
const mockedJob = { id: 'kleinanzeigen', notificationAdapter: null, specFilter: null, spatialFilter: null };
|
||||
|
||||
const fredy = new Fredy(
|
||||
provider.config,
|
||||
mockedJob,
|
||||
provider.metaInformation.id,
|
||||
{ checkAndAddEntry: () => false },
|
||||
undefined,
|
||||
);
|
||||
const listings = await fredy.execute();
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
listings.forEach((listing) => {
|
||||
expect(listing.link).toContain('https://www.kleinanzeigen.de');
|
||||
expect(listing.address).toBeTypeOf('string');
|
||||
expect(listing.address).not.toBe('');
|
||||
expect(listing.description).toBeTypeOf('string');
|
||||
expect(listing.description).not.toBe('');
|
||||
});
|
||||
});
|
||||
expect(enriched).toBeTruthy();
|
||||
expect(enriched.link).toContain('https://www.kleinanzeigen.de');
|
||||
expect(enriched.address).toBeTypeOf('string');
|
||||
expect(enriched.address).not.toBe('');
|
||||
expect(enriched.description).toBeTypeOf('string');
|
||||
expect(enriched.description).not.toBe('');
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -9,81 +9,97 @@ import { mockFredy, providerConfig } from '../utils.js';
|
||||
import { expect, vi } from 'vitest';
|
||||
import * as provider from '../../lib/provider/sparkasse.js';
|
||||
import * as mockStore from '../mocks/mockStore.js';
|
||||
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
|
||||
|
||||
// One browser shared across the whole suite so both requests (search + detail)
|
||||
// come from the same warm session. This prevents the second request from being
|
||||
// flagged as a cold-start bot hit.
|
||||
const TEST_TIMEOUT = 120_000;
|
||||
|
||||
describe('#sparkasse testsuite()', () => {
|
||||
it('should test sparkasse provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'sparkasse',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
provider.init(providerConfig.sparkasse, []);
|
||||
let browser;
|
||||
let liveListings;
|
||||
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
|
||||
beforeAll(async () => {
|
||||
browser = await launchBrowser(providerConfig.sparkasse.url);
|
||||
}, TEST_TIMEOUT);
|
||||
|
||||
const listing = await fredy.execute();
|
||||
|
||||
if (listing == null || listing.length === 0) {
|
||||
throw new Error('Listings is empty!');
|
||||
}
|
||||
|
||||
expect(listing).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('sparkasse');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.price).toContain('€');
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.size).toContain('m²');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
afterAll(async () => {
|
||||
await closeBrowser(browser);
|
||||
});
|
||||
|
||||
it(
|
||||
'should test sparkasse provider',
|
||||
async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'sparkasse',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
provider.init(providerConfig.sparkasse, []);
|
||||
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
|
||||
|
||||
liveListings = await fredy.execute();
|
||||
|
||||
if (liveListings == null || liveListings.length === 0) {
|
||||
throw new Error('Listings is empty!');
|
||||
}
|
||||
|
||||
expect(liveListings).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).toBeTypeOf('object');
|
||||
expect(notificationObj.serviceName).toBe('sparkasse');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.price).toContain('€');
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.size).toContain('m²');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
expect(notify.address).toBeTypeOf('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.size).toBeTypeOf('string');
|
||||
expect(notify.title).not.toBe('');
|
||||
expect(notify.address).not.toBe('');
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
|
||||
describe('with provider_details enabled', () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
|
||||
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('should enrich listings with details', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
provider.init(providerConfig.sparkasse, []);
|
||||
const mockedJob = { id: 'sparkasse', notificationAdapter: null, specFilter: null, spatialFilter: null };
|
||||
it(
|
||||
'should enrich listings with details',
|
||||
async () => {
|
||||
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
|
||||
|
||||
const fredy = new Fredy(
|
||||
provider.config,
|
||||
mockedJob,
|
||||
provider.metaInformation.id,
|
||||
{ checkAndAddEntry: () => false },
|
||||
undefined,
|
||||
);
|
||||
const listings = await fredy.execute();
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
listings.forEach((listing) => {
|
||||
expect(listing.link).toContain('https://immobilien.sparkasse.de');
|
||||
expect(listing.address).toBeTypeOf('string');
|
||||
expect(listing.address).not.toBe('');
|
||||
// description is enriched from the detail page; falls back gracefully if bot-detected
|
||||
if (listing.description != null) {
|
||||
expect(listing.description).toBeTypeOf('string');
|
||||
expect(listing.description).not.toBe('');
|
||||
// Call fetchDetails directly on the first live listing — no need to
|
||||
// re-scrape the search page. The shared browser keeps the session warm.
|
||||
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
|
||||
|
||||
expect(enriched).toBeTruthy();
|
||||
expect(enriched.link).toContain('https://immobilien.sparkasse.de');
|
||||
expect(enriched.address).toBeTypeOf('string');
|
||||
expect(enriched.address).not.toBe('');
|
||||
// description is enriched from the detail page; falls back gracefully if blocked
|
||||
if (enriched.description != null) {
|
||||
expect(enriched.description).toBeTypeOf('string');
|
||||
expect(enriched.description).not.toBe('');
|
||||
}
|
||||
});
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,77 +6,85 @@
|
||||
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
|
||||
import { get } from '../mocks/mockNotification.js';
|
||||
import { mockFredy, providerConfig } from '../utils.js';
|
||||
import { expect, vi } from 'vitest';
|
||||
import { expect } from 'vitest';
|
||||
import * as provider from '../../lib/provider/wgGesucht.js';
|
||||
import * as mockStore from '../mocks/mockStore.js';
|
||||
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
|
||||
|
||||
// One browser shared across the whole suite so both requests (search + detail)
|
||||
// come from the same warm session, avoiding double cold-start bot detection.
|
||||
const TEST_TIMEOUT = 120_000;
|
||||
|
||||
describe('#wgGesucht testsuite()', () => {
|
||||
provider.init(providerConfig.wgGesucht, [], []);
|
||||
it('should test wgGesucht provider', { timeout: 120000 }, async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'wgGesucht',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
|
||||
return await new Promise((resolve, reject) => {
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
|
||||
let browser;
|
||||
let liveListings;
|
||||
|
||||
fredy.execute().then((listing) => {
|
||||
if (listing == null || listing.length === 0) {
|
||||
reject('Listings is empty!');
|
||||
return;
|
||||
}
|
||||
beforeAll(async () => {
|
||||
browser = await launchBrowser(providerConfig.wgGesucht.url);
|
||||
}, TEST_TIMEOUT);
|
||||
|
||||
expect(listing).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj.serviceName).toBe('wgGesucht');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
expect(notify).toBeTypeOf('object');
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
// expect(notify.details).toBeTypeOf('string');
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.price).toContain('€');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
afterAll(async () => {
|
||||
await closeBrowser(browser);
|
||||
});
|
||||
|
||||
it(
|
||||
'should test wgGesucht provider',
|
||||
async () => {
|
||||
const Fredy = await mockFredy();
|
||||
const mockedJob = {
|
||||
id: 'wgGesucht',
|
||||
notificationAdapter: null,
|
||||
spatialFilter: null,
|
||||
specFilter: null,
|
||||
};
|
||||
|
||||
return await new Promise((resolve, reject) => {
|
||||
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
|
||||
|
||||
fredy.execute().then((listing) => {
|
||||
if (listing == null || listing.length === 0) {
|
||||
reject('Listings is empty!');
|
||||
return;
|
||||
}
|
||||
|
||||
liveListings = listing;
|
||||
expect(listing).toBeInstanceOf(Array);
|
||||
const notificationObj = get();
|
||||
expect(notificationObj.serviceName).toBe('wgGesucht');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
expect(notify).toBeTypeOf('object');
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).toBeTypeOf('string');
|
||||
expect(notify.title).toBeTypeOf('string');
|
||||
// expect(notify.details).toBeTypeOf('string');
|
||||
expect(notify.price).toBeTypeOf('string');
|
||||
expect(notify.price).toContain('€');
|
||||
expect(notify.link).toBeTypeOf('string');
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
|
||||
describe('with provider_details enabled', () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
|
||||
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
|
||||
});
|
||||
it(
|
||||
'should enrich listings with details',
|
||||
async () => {
|
||||
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
// Call fetchDetails directly on the first live listing — no need to
|
||||
// re-scrape the search page. The shared browser keeps the session warm.
|
||||
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
|
||||
|
||||
it('should enrich listings with details', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
provider.init(providerConfig.wgGesucht, [], []);
|
||||
const mockedJob = { id: 'wgGesucht', notificationAdapter: null, specFilter: null, spatialFilter: null };
|
||||
|
||||
const fredy = new Fredy(
|
||||
provider.config,
|
||||
mockedJob,
|
||||
provider.metaInformation.id,
|
||||
{ checkAndAddEntry: () => false },
|
||||
undefined,
|
||||
);
|
||||
const listings = await fredy.execute();
|
||||
expect(listings).toBeInstanceOf(Array);
|
||||
listings.forEach((listing) => {
|
||||
expect(listing.link).toContain('https://www.wg-gesucht.de');
|
||||
expect(listing.description).toBeTypeOf('string');
|
||||
expect(listing.description).not.toBe('');
|
||||
});
|
||||
});
|
||||
expect(enriched).toBeTruthy();
|
||||
expect(enriched.link).toContain('https://www.wg-gesucht.de');
|
||||
expect(enriched.description).toBeTypeOf('string');
|
||||
expect(enriched.description).not.toBe('');
|
||||
},
|
||||
TEST_TIMEOUT,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
37
test/services/extractor/puppeteerExtractor.test.js
Normal file
37
test/services/extractor/puppeteerExtractor.test.js
Normal file
@@ -0,0 +1,37 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { vi, describe, it, expect, beforeEach } from 'vitest';
|
||||
|
||||
// Mock the CloakBrowser launcher so no real Chromium binary is needed and we can
|
||||
// assert which options get forwarded to it.
|
||||
const { launchMock } = vi.hoisted(() => ({ launchMock: vi.fn() }));
|
||||
|
||||
vi.mock('cloakbrowser/puppeteer', () => ({
|
||||
launch: launchMock,
|
||||
}));
|
||||
|
||||
const { launchBrowser } = await import('../../../lib/services/extractor/puppeteerExtractor.js');
|
||||
|
||||
describe('launchBrowser proxy forwarding', () => {
|
||||
beforeEach(() => {
|
||||
launchMock.mockReset();
|
||||
launchMock.mockResolvedValue({ close: async () => {} });
|
||||
});
|
||||
|
||||
it('forwards proxyUrl to CloakBrowser as the proxy option', async () => {
|
||||
await launchBrowser('https://www.immowelt.de/', { proxyUrl: 'http://user:pass@host:8080' });
|
||||
|
||||
expect(launchMock).toHaveBeenCalledTimes(1);
|
||||
expect(launchMock.mock.calls[0][0]).toMatchObject({ proxy: 'http://user:pass@host:8080' });
|
||||
});
|
||||
|
||||
it('does not set a proxy when no proxyUrl is given', async () => {
|
||||
await launchBrowser('https://www.immowelt.de/', {});
|
||||
|
||||
expect(launchMock).toHaveBeenCalledTimes(1);
|
||||
expect(launchMock.mock.calls[0][0].proxy).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -4,12 +4,28 @@
|
||||
*/
|
||||
|
||||
import { convertWebToMobile } from '../../../lib/services/immoscout/immoscout-web-translator.js';
|
||||
import { expect } from 'vitest';
|
||||
import { expect, vi } from 'vitest';
|
||||
import { readFile } from 'fs/promises';
|
||||
import { buildFetchMock } from '../../offlineFixtures.js';
|
||||
|
||||
export const testData = JSON.parse(await readFile(new URL('./testdata.json', import.meta.url)));
|
||||
|
||||
if (process.env.TEST_MODE === 'offline') {
|
||||
vi.stubGlobal('fetch', buildFetchMock());
|
||||
}
|
||||
|
||||
describe('#immoscout-mobile URL conversion', () => {
|
||||
// Test shape URL conversion
|
||||
it('should convert a full web URL with shape to mobile URL', () => {
|
||||
const webUrl =
|
||||
'https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list';
|
||||
const expectedMobileUrl =
|
||||
'https://api.mobile.immobilienscout24.de/search/list?ground=240.0-&price=-600000.0&realestatetype=housebuy&searchType=shape&shape=ior~H_kxmAr%60Pig%60%40fzHm%7Bp%40qs%40%7D%60l%40crCidPuol%40wy~%40onVb~Ek%60KhdPcoEhkS%7CHf%7B%7CAtpDtqLjiGngQ%7CxLrlOxyVbzR';
|
||||
|
||||
const actualMobileUrl = convertWebToMobile(webUrl);
|
||||
expect(actualMobileUrl).toBe(expectedMobileUrl);
|
||||
});
|
||||
|
||||
// Test URL conversion
|
||||
it('should convert a full web URL to mobile URL', () => {
|
||||
const webUrl =
|
||||
@@ -30,6 +46,60 @@ describe('#immoscout-mobile URL conversion', () => {
|
||||
expect(queryParams.get('equipment').split(',')).toEqual(expect.arrayContaining(['garden', 'balcony']));
|
||||
});
|
||||
|
||||
// Test URL conversion of SEO web path for max warmrent. The ImmoScout web UI
|
||||
// generates this special SEO slug instead of explicit price/pricetype params
|
||||
// when the user configures a "Warmmiete" filter (real-world URL).
|
||||
it('should convert a SEO apartment max warmrent path to rent + price + pricetype', () => {
|
||||
const webUrl =
|
||||
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-bis-800-euro-warm?livingspace=-800.0&enteredFrom=result_list';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
|
||||
expect(queryParams.get('price')).toBe('-800');
|
||||
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
|
||||
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
|
||||
expect(queryParams.get('livingspace')).toBe('-800.0');
|
||||
});
|
||||
|
||||
// Same SEO pattern for houses ("haus-bis-X-euro-warm" → houserent).
|
||||
it('should convert a SEO house max warmrent path to rent + price + pricetype', () => {
|
||||
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/haus-bis-1500-euro-warm';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('houserent');
|
||||
expect(queryParams.get('price')).toBe('-1500');
|
||||
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
|
||||
});
|
||||
|
||||
// Sanity check: max coldrent ("Kaltmiete") does NOT use an SEO slug. The web
|
||||
// UI keeps the regular "wohnung-mieten" path and passes explicit
|
||||
// price + pricetype query params, which the existing translator already
|
||||
// handles (real-world URL).
|
||||
it('should convert a max coldrent search via the regular wohnung-mieten path', () => {
|
||||
const webUrl =
|
||||
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-mieten?price=-800.0&livingspace=-800.0&pricetype=rentpermonth&enteredFrom=result_list';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
|
||||
expect(queryParams.get('price')).toBe('-800.0');
|
||||
expect(queryParams.get('pricetype')).toBe('rentpermonth');
|
||||
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
|
||||
});
|
||||
|
||||
// Explicit query params win over the SEO slug's implicit defaults.
|
||||
it('should let explicit query params override SEO path price defaults', () => {
|
||||
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-bis-800-euro-warm?price=100-500';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
|
||||
expect(queryParams.get('price')).toBe('100-500');
|
||||
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
|
||||
});
|
||||
|
||||
// Test URL conversion with unsupported query parameters
|
||||
it('should remove unsupported query parameters', () => {
|
||||
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?minimuminternetspeed=100000';
|
||||
|
||||
@@ -18,5 +18,9 @@
|
||||
"rentHouse": {
|
||||
"url": "https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/haus-mieten?enteredFrom=one_step_search",
|
||||
"type": "houserent"
|
||||
},
|
||||
"buyHouseWithShape": {
|
||||
"url": "https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list",
|
||||
"type": "housebuy"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ describe('services/jobs/jobExecutionService', () => {
|
||||
const busPath = root + '/lib/services/events/event-bus.js';
|
||||
const jobStoragePath = root + '/lib/services/storage/jobStorage.js';
|
||||
const userStoragePath = root + '/lib/services/storage/userStorage.js';
|
||||
const settingsStoragePath = root + '/lib/services/storage/settingsStorage.js';
|
||||
const brokerPath = root + '/lib/services/sse/sse-broker.js';
|
||||
const utilsPath = root + '/lib/utils.js';
|
||||
const loggerPath = root + '/lib/services/logger.js';
|
||||
@@ -33,11 +34,15 @@ describe('services/jobs/jobExecutionService', () => {
|
||||
getUsers: () => state.users.slice(),
|
||||
getUser: (id) => state.users.find((u) => u.id === id) || null,
|
||||
}));
|
||||
vi.doMock(settingsStoragePath, () => ({
|
||||
getSettings: async () => ({}),
|
||||
}));
|
||||
vi.doMock(brokerPath, () => ({
|
||||
sendToUsers: (...args) => calls.sent.push(args),
|
||||
}));
|
||||
vi.doMock(utilsPath, () => ({
|
||||
duringWorkingHoursOrNotSet: () => false,
|
||||
getPackageVersion: async () => '0.0.0-test',
|
||||
}));
|
||||
vi.doMock(loggerPath, () => {
|
||||
const m = { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} };
|
||||
|
||||
193
test/storage/listingStatus.test.js
Normal file
193
test/storage/listingStatus.test.js
Normal file
@@ -0,0 +1,193 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { vi, describe, it, expect, beforeEach } from 'vitest';
|
||||
|
||||
// We mock SqliteConnection so we can assert which SQL the storage layer
|
||||
// runs and with which params, without spinning up a real SQLite DB.
|
||||
|
||||
const calls = {
|
||||
execute: [],
|
||||
query: [],
|
||||
};
|
||||
|
||||
const sqliteMock = {
|
||||
execute: (sql, params) => {
|
||||
calls.execute.push({ sql, params });
|
||||
// Default: pretend 1 row was affected (so setListingStatus reports success).
|
||||
return { changes: 1 };
|
||||
},
|
||||
query: (sql, params) => {
|
||||
calls.query.push({ sql, params });
|
||||
// Return shape varies by test — overridden via queryHandler when needed.
|
||||
if (sqliteMock.__queryHandler) return sqliteMock.__queryHandler(sql, params);
|
||||
return [];
|
||||
},
|
||||
__queryHandler: null,
|
||||
};
|
||||
|
||||
vi.mock('../../lib/services/storage/SqliteConnection.js', () => ({
|
||||
default: sqliteMock,
|
||||
}));
|
||||
|
||||
describe('listingsStorage.setListingStatus', () => {
|
||||
let listingsStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
sqliteMock.__queryHandler = null;
|
||||
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
|
||||
});
|
||||
|
||||
it('runs an UPDATE storing a JSON payload with status and setAt', () => {
|
||||
const before = Date.now();
|
||||
const changes = listingsStorage.setListingStatus('listing-1', 'Applied');
|
||||
const after = Date.now();
|
||||
expect(changes).toBe(1);
|
||||
expect(calls.execute).toHaveLength(1);
|
||||
expect(calls.execute[0].sql).toMatch(/UPDATE listings SET status = @status WHERE id = @id/);
|
||||
expect(calls.execute[0].params.id).toBe('listing-1');
|
||||
const parsed = JSON.parse(calls.execute[0].params.status);
|
||||
expect(parsed.status).toBe('applied');
|
||||
expect(parsed.setAt).toBeGreaterThanOrEqual(before);
|
||||
expect(parsed.setAt).toBeLessThanOrEqual(after);
|
||||
});
|
||||
|
||||
it('accepts null to clear the status (no JSON wrapping)', () => {
|
||||
listingsStorage.setListingStatus('listing-2', null);
|
||||
expect(calls.execute[0].params).toEqual({ id: 'listing-2', status: null });
|
||||
});
|
||||
|
||||
it('rejects invalid statuses', () => {
|
||||
expect(() => listingsStorage.setListingStatus('listing-3', 'maybe')).toThrow(/Invalid listing status/);
|
||||
expect(calls.execute).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('returns 0 when no id is supplied (no SQL is run)', () => {
|
||||
const result = listingsStorage.setListingStatus(null, 'applied');
|
||||
expect(result).toBe(0);
|
||||
expect(calls.execute).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('listingsStorage.queryListings statusFilter', () => {
|
||||
let listingsStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
// Return empty rows for both the count and the page-fetch queries.
|
||||
sqliteMock.__queryHandler = (sql) => {
|
||||
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 0 }];
|
||||
return [];
|
||||
};
|
||||
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
|
||||
});
|
||||
|
||||
it("adds 'l.status IS NULL' to WHERE when statusFilter is 'none'", () => {
|
||||
listingsStorage.queryListings({ statusFilter: 'none', userId: 'u1', isAdmin: true });
|
||||
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
|
||||
expect(pageQuery.sql).toMatch(/\(l\.status IS NULL\)/);
|
||||
});
|
||||
|
||||
it('extracts the inner status field via json_extract for a concrete status', () => {
|
||||
listingsStorage.queryListings({ statusFilter: 'applied', userId: 'u1', isAdmin: true });
|
||||
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
|
||||
expect(pageQuery.sql).toMatch(/json_extract\(l\.status, '\$\.status'\) = @statusValue/);
|
||||
expect(pageQuery.params.statusValue).toBe('applied');
|
||||
});
|
||||
|
||||
it('ignores unknown statusFilter values silently', () => {
|
||||
listingsStorage.queryListings({ statusFilter: 'bogus', userId: 'u1', isAdmin: true });
|
||||
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
|
||||
expect(pageQuery.sql).not.toMatch(/status/i);
|
||||
});
|
||||
|
||||
it('parses the JSON status payload of returned rows into an object', () => {
|
||||
sqliteMock.__queryHandler = (sql) => {
|
||||
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 2 }];
|
||||
return [
|
||||
{ id: 'a', status: JSON.stringify({ status: 'applied', setAt: 1700000000000 }) },
|
||||
{ id: 'b', status: null },
|
||||
];
|
||||
};
|
||||
const result = listingsStorage.queryListings({ userId: 'u1', isAdmin: true });
|
||||
expect(result.result[0].status).toEqual({ status: 'applied', setAt: 1700000000000 });
|
||||
expect(result.result[1].status).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('listingsStorage.getListingById', () => {
|
||||
let listingsStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
|
||||
});
|
||||
|
||||
it('parses the JSON status payload of the returned row', () => {
|
||||
sqliteMock.__queryHandler = () => [
|
||||
{ id: 'a', status: JSON.stringify({ status: 'rejected', setAt: 1700000000001 }) },
|
||||
];
|
||||
const row = listingsStorage.getListingById('a', 'u1', true);
|
||||
expect(row.status).toEqual({ status: 'rejected', setAt: 1700000000001 });
|
||||
});
|
||||
|
||||
it('returns null status untouched', () => {
|
||||
sqliteMock.__queryHandler = () => [{ id: 'a', status: null }];
|
||||
const row = listingsStorage.getListingById('a', 'u1', true);
|
||||
expect(row.status).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null when no row is found', () => {
|
||||
sqliteMock.__queryHandler = () => [];
|
||||
const row = listingsStorage.getListingById('missing', 'u1', true);
|
||||
expect(row).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('watchListStorage.ensureWatch', () => {
|
||||
let watchListStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
sqliteMock.__queryHandler = null;
|
||||
watchListStorage = await import('../../lib/services/storage/watchListStorage.js');
|
||||
});
|
||||
|
||||
it('inserts and reports watched=true on first call', () => {
|
||||
// After INSERT, createWatch queries for existence and gets a row back.
|
||||
sqliteMock.__queryHandler = () => [{ ok: 1 }];
|
||||
const result = watchListStorage.ensureWatch('listing-1', 'user-1');
|
||||
expect(result).toEqual({ watched: true });
|
||||
// INSERT should have been issued.
|
||||
expect(calls.execute.some((c) => /INSERT INTO watch_list/.test(c.sql))).toBe(true);
|
||||
});
|
||||
|
||||
it('returns watched=true when an entry already exists', () => {
|
||||
// Simulate ON CONFLICT being a no-op: execute reports no changes, then SELECT confirms row exists.
|
||||
sqliteMock.execute = (sql, params) => {
|
||||
calls.execute.push({ sql, params });
|
||||
return { changes: 0 };
|
||||
};
|
||||
sqliteMock.__queryHandler = () => [{ ok: 1 }];
|
||||
const result = watchListStorage.ensureWatch('listing-2', 'user-2');
|
||||
expect(result).toEqual({ watched: true });
|
||||
// Restore execute to default for subsequent tests.
|
||||
sqliteMock.execute = (sql, params) => {
|
||||
calls.execute.push({ sql, params });
|
||||
return { changes: 1 };
|
||||
};
|
||||
});
|
||||
|
||||
it('returns watched=false when listingId or userId is missing', () => {
|
||||
expect(watchListStorage.ensureWatch(null, 'u')).toEqual({ watched: false });
|
||||
expect(watchListStorage.ensureWatch('l', null)).toEqual({ watched: false });
|
||||
expect(calls.execute).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
@@ -29,7 +29,7 @@ vi.mock('../lib/services/extractor/puppeteerExtractor.js', async (importOriginal
|
||||
const { readFixture } = await import('./offlineFixtures.js');
|
||||
return {
|
||||
default: (url) => readFixture(url),
|
||||
launchBrowser: async () => ({ close: async () => {}, __fredy_removeUserDataDir: false }),
|
||||
launchBrowser: async () => ({ close: async () => {}, isConnected: () => true }),
|
||||
closeBrowser: async () => {},
|
||||
};
|
||||
});
|
||||
|
||||
@@ -155,6 +155,7 @@ const routes = {
|
||||
'GET /api/dashboard': dashboard,
|
||||
'GET /api/demo': { demoMode: false },
|
||||
'POST /api/user/settings/news-hash': {},
|
||||
'POST /api/user/settings/listing-deletion-preference': {},
|
||||
};
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
|
||||
@@ -95,7 +95,10 @@ async function downloadHtmlProvider(name, providerConfig, launchBrowser, closeBr
|
||||
|
||||
const browser = await launchBrowser(providerConfig.url, {});
|
||||
try {
|
||||
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, { browser });
|
||||
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, {
|
||||
browser,
|
||||
name: 'dowload_fixtures',
|
||||
});
|
||||
|
||||
if (!html) {
|
||||
console.warn(` Failed to download ${name}`);
|
||||
|
||||
@@ -97,6 +97,7 @@ export default function FredyApp() {
|
||||
<Route path="/dashboard" element={<Dashboard />} />
|
||||
<Route path="/jobs" element={<Jobs />} />
|
||||
<Route path="/listings" element={<Listings />} />
|
||||
<Route path="/listings/watchlist" element={<Listings mode="watchlist" />} />
|
||||
<Route path="/listings/listing/:listingId" element={<ListingDetail />} />
|
||||
<Route path="/map" element={<MapView />} />
|
||||
<Route path="/watchlistManagement" element={<WatchlistManagement />} />
|
||||
|
||||
BIN
ui/src/assets/news/1.png
Normal file
BIN
ui/src/assets/news/1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 835 KiB |
BIN
ui/src/assets/news/2.png
Normal file
BIN
ui/src/assets/news/2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 367 KiB |
@@ -1,10 +1,16 @@
|
||||
{
|
||||
"key": "00e6b81777a275f5a140fc9101cb943810db6a69f6eb3927319c5aee0c876542",
|
||||
"key": "00e6b81777a275f5a140fc9101cb943810db6a69f6eb3927319c5aee0c876221",
|
||||
"content":
|
||||
[
|
||||
{
|
||||
"title": "Open in...Fredy ;)",
|
||||
"text": "With the latest version of Fredy, every notification now comes with a link that opens the listing directly inside Fredy. This is also a key step toward an upcoming...milestone :).<br/>To make this work, Fredy needs to know where it lives on the network. We try to guess the public base URL, but let’s be honest, you probably know better. Take a quick look at the baseUrl in the system settings and fix it if it looks off."
|
||||
"title": "Table overview for listings",
|
||||
"text": "Thanks to https://github.com/datenwurm, we now have a table overview for listings. If you decide to use the table view, the decision will be stored.",
|
||||
"media": "1.png"
|
||||
},
|
||||
{
|
||||
"title": "Table overview for jobs",
|
||||
"text": "Based on datenwurm's, work, I created a table overview for jobs. If you decide to use the table view, the decision will be stored.",
|
||||
"media": "2.png"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { useState } from 'react';
|
||||
import { Modal, Radio, RadioGroup, Typography } from '@douyinfe/semi-ui-19';
|
||||
import { useState, useEffect } from 'react';
|
||||
import { Modal, Radio, RadioGroup, Typography, Checkbox } from '@douyinfe/semi-ui-19';
|
||||
|
||||
const { Text } = Typography;
|
||||
|
||||
@@ -15,11 +15,24 @@ const ListingDeletionModal = ({
|
||||
title = 'Delete Listings',
|
||||
showOptions = true,
|
||||
message = 'How would you like to delete the selected listing(s)?',
|
||||
defaultDeleteType = 'soft',
|
||||
}) => {
|
||||
const [deleteType, setDeleteType] = useState('soft');
|
||||
const [remember, setRemember] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
if (visible) {
|
||||
setDeleteType(defaultDeleteType);
|
||||
setRemember(false);
|
||||
}
|
||||
}, [visible, defaultDeleteType]);
|
||||
|
||||
const handleOk = () => {
|
||||
onConfirm(!showOptions || deleteType === 'hard');
|
||||
if (showOptions) {
|
||||
onConfirm(deleteType === 'hard', remember);
|
||||
} else {
|
||||
onConfirm(true);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
@@ -36,32 +49,37 @@ const ListingDeletionModal = ({
|
||||
<Text>{message}</Text>
|
||||
</div>
|
||||
{showOptions && (
|
||||
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
|
||||
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Mark as deleted (Soft Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
|
||||
scraping session.
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Remove from database (Hard Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are completely removed from the database.
|
||||
<>
|
||||
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
|
||||
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Mark as deleted (Soft Delete)</Text>
|
||||
<br />
|
||||
<Text type="warning">
|
||||
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
|
||||
previously found.
|
||||
<Text type="secondary">
|
||||
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
|
||||
scraping session.
|
||||
</Text>
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
</RadioGroup>
|
||||
</div>
|
||||
</Radio>
|
||||
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Remove from database (Hard Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are completely removed from the database.
|
||||
<br />
|
||||
<Text type="warning">
|
||||
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
|
||||
previously found.
|
||||
</Text>
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
</RadioGroup>
|
||||
<Checkbox checked={remember} onChange={(e) => setRemember(e.target.checked)} style={{ marginTop: 16 }}>
|
||||
Remember my choice and skip this dialog next time
|
||||
</Checkbox>
|
||||
</>
|
||||
)}
|
||||
</Modal>
|
||||
);
|
||||
|
||||
@@ -21,6 +21,7 @@ import {
|
||||
Empty,
|
||||
Radio,
|
||||
RadioGroup,
|
||||
Tooltip,
|
||||
} from '@douyinfe/semi-ui-19';
|
||||
import {
|
||||
IconAlertTriangle,
|
||||
@@ -35,6 +36,8 @@ import {
|
||||
IconArrowUp,
|
||||
IconArrowDown,
|
||||
IconHome,
|
||||
IconGridView,
|
||||
IconList,
|
||||
} from '@douyinfe/semi-icons';
|
||||
import { useNavigate } from 'react-router-dom';
|
||||
import ListingDeletionModal from '../../ListingDeletionModal.jsx';
|
||||
@@ -42,6 +45,7 @@ import { useActions, useSelector } from '../../../services/state/store.js';
|
||||
import { xhrDelete, xhrPut, xhrPost } from '../../../services/xhr.js';
|
||||
import { debounce } from '../../../utils';
|
||||
import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-illustrations';
|
||||
import JobsTable from '../../table/JobsTable.jsx';
|
||||
|
||||
import './JobGrid.less';
|
||||
|
||||
@@ -54,6 +58,11 @@ const JobGrid = () => {
|
||||
const actions = useActions();
|
||||
const navigate = useNavigate();
|
||||
|
||||
const userSettings = useSelector((state) => state.userSettings.settings);
|
||||
const viewMode = userSettings?.jobs_view_mode ?? 'grid';
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
|
||||
const [page, setPage] = useState(1);
|
||||
const pageSize = 12;
|
||||
|
||||
@@ -135,13 +144,21 @@ const JobGrid = () => {
|
||||
};
|
||||
|
||||
const onListingRemoval = (jobId) => {
|
||||
setPendingDeletion({ type: 'listings', jobId });
|
||||
const deletion = { type: 'listings', jobId };
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmDeletion(listingDeletionPref.hardDelete, false, deletion);
|
||||
return;
|
||||
}
|
||||
setPendingDeletion(deletion);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
|
||||
const confirmDeletion = async (hardDelete) => {
|
||||
const { type, jobId } = pendingDeletion;
|
||||
const confirmDeletion = async (hardDelete, remember, deletion = pendingDeletion) => {
|
||||
const { type, jobId } = deletion;
|
||||
try {
|
||||
if (remember && type === 'listings') {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
if (type === 'job') {
|
||||
await xhrDelete('/api/jobs', { jobId });
|
||||
Toast.success('Job and listings successfully removed');
|
||||
@@ -167,7 +184,7 @@ const JobGrid = () => {
|
||||
Toast.success('Job status successfully changed');
|
||||
loadData();
|
||||
} catch (error) {
|
||||
Toast.error(error);
|
||||
Toast.error(error.error);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -234,6 +251,27 @@ const JobGrid = () => {
|
||||
onClick={() => setSortDir((d) => (d === 'asc' ? 'desc' : 'asc'))}
|
||||
title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
|
||||
/>
|
||||
|
||||
<div className="jobGrid__topbar__view-toggle">
|
||||
<Tooltip content="Grid view">
|
||||
<Button
|
||||
icon={<IconGridView />}
|
||||
theme={viewMode === 'grid' ? 'solid' : 'borderless'}
|
||||
onClick={() => actions.userSettings.setJobsViewMode('grid')}
|
||||
aria-label="Grid view"
|
||||
aria-pressed={viewMode === 'grid'}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Table view">
|
||||
<Button
|
||||
icon={<IconList />}
|
||||
theme={viewMode === 'table' ? 'solid' : 'borderless'}
|
||||
onClick={() => actions.userSettings.setJobsViewMode('table')}
|
||||
aria-label="Table view"
|
||||
aria-pressed={viewMode === 'table'}
|
||||
/>
|
||||
</Tooltip>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{(jobsData?.result || []).length === 0 && (
|
||||
@@ -244,136 +282,144 @@ const JobGrid = () => {
|
||||
/>
|
||||
)}
|
||||
|
||||
<Row gutter={[16, 16]}>
|
||||
{(jobsData?.result || []).map((job) => (
|
||||
<Col key={job.id} xs={24} sm={12} md={12} lg={8} xl={8} xxl={6}>
|
||||
<Card className="jobGrid__card" bodyStyle={{ padding: '16px' }}>
|
||||
<div className="jobGrid__card__header">
|
||||
<div className="jobGrid__card__name">
|
||||
<span className={`jobGrid__card__dot${job.enabled ? ' jobGrid__card__dot--active' : ''}`} />
|
||||
<Title heading={5} ellipsis={{ showTooltip: true }} className="jobGrid__title">
|
||||
{job.name}
|
||||
</Title>
|
||||
{viewMode === 'grid' ? (
|
||||
<Row gutter={[16, 16]}>
|
||||
{(jobsData?.result || []).map((job) => (
|
||||
<Col key={job.id} xs={24} sm={12} md={12} lg={8} xl={8} xxl={6}>
|
||||
<Card className="jobGrid__card" bodyStyle={{ padding: '16px' }}>
|
||||
<div className="jobGrid__card__header">
|
||||
<div className="jobGrid__card__name">
|
||||
<span className={`jobGrid__card__dot${job.enabled ? ' jobGrid__card__dot--active' : ''}`} />
|
||||
<Title heading={5} ellipsis={{ showTooltip: true }} className="jobGrid__title">
|
||||
{job.name}
|
||||
</Title>
|
||||
</div>
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: 6, flexShrink: 0 }}>
|
||||
{job.isOnlyShared && (
|
||||
<Popover content={getPopoverContent('This job has been shared with you — read only.')}>
|
||||
<div>
|
||||
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
|
||||
</div>
|
||||
</Popover>
|
||||
)}
|
||||
{job.running && (
|
||||
<Tag color="green" variant="light" size="small">
|
||||
RUNNING
|
||||
</Tag>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: 6, flexShrink: 0 }}>
|
||||
{job.isOnlyShared && (
|
||||
<Popover
|
||||
content={getPopoverContent(
|
||||
'This job has been shared with you by another user, therefor it is read-only.',
|
||||
)}
|
||||
>
|
||||
|
||||
<div className="jobGrid__card__stats">
|
||||
<div className="jobGrid__card__stat jobGrid__card__stat--blue">
|
||||
<span className="jobGrid__card__stat__number">{job.numberOfFoundListings || 0}</span>
|
||||
<span className="jobGrid__card__stat__label">
|
||||
<IconHome size="small" /> Listings
|
||||
</span>
|
||||
</div>
|
||||
<div className="jobGrid__card__stat jobGrid__card__stat--orange">
|
||||
<span className="jobGrid__card__stat__number">{job.provider?.length || 0}</span>
|
||||
<span className="jobGrid__card__stat__label">
|
||||
<IconBriefcase size="small" /> Providers
|
||||
</span>
|
||||
</div>
|
||||
<div className="jobGrid__card__stat jobGrid__card__stat--purple">
|
||||
<span className="jobGrid__card__stat__number">{job.notificationAdapter?.length || 0}</span>
|
||||
<span className="jobGrid__card__stat__label">
|
||||
<IconBell size="small" /> Adapters
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<Divider margin="12px" />
|
||||
|
||||
<div className="jobGrid__card__footer">
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
|
||||
<Switch
|
||||
onChange={(checked) => onJobStatusChanged(job.id, checked)}
|
||||
checked={job.enabled}
|
||||
disabled={job.isOnlyShared}
|
||||
size="small"
|
||||
/>
|
||||
<Text type="secondary" size="small">
|
||||
Active
|
||||
</Text>
|
||||
</div>
|
||||
<div className="jobGrid__actions">
|
||||
<Popover content={getPopoverContent('Run Job')}>
|
||||
<div>
|
||||
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
|
||||
<Button
|
||||
type="primary"
|
||||
style={{ background: '#21aa21b5' }}
|
||||
size="small"
|
||||
theme="solid"
|
||||
icon={<IconPlayCircle />}
|
||||
disabled={job.isOnlyShared || job.running}
|
||||
onClick={() => onJobRun(job.id)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
)}
|
||||
{job.running && (
|
||||
<Tag color="green" variant="light" size="small">
|
||||
RUNNING
|
||||
</Tag>
|
||||
)}
|
||||
<Popover content={getPopoverContent('Edit a Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="secondary"
|
||||
size="small"
|
||||
icon={<IconEdit />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => navigate(`/jobs/edit/${job.id}`)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
<Popover content={getPopoverContent('Clone Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="tertiary"
|
||||
size="small"
|
||||
icon={<IconCopy />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => navigate('/jobs/new', { state: { cloneFrom: job.id } })}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
<Popover content={getPopoverContent('Delete all found Listings of this Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="danger"
|
||||
size="small"
|
||||
icon={<IconDescend2 />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onListingRemoval(job.id)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
<Popover content={getPopoverContent('Delete Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="danger"
|
||||
size="small"
|
||||
icon={<IconDelete />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onJobRemoval(job.id)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="jobGrid__card__stats">
|
||||
<div className="jobGrid__card__stat jobGrid__card__stat--blue">
|
||||
<span className="jobGrid__card__stat__number">{job.numberOfFoundListings || 0}</span>
|
||||
<span className="jobGrid__card__stat__label">
|
||||
<IconHome size="small" /> Listings
|
||||
</span>
|
||||
</div>
|
||||
<div className="jobGrid__card__stat jobGrid__card__stat--orange">
|
||||
<span className="jobGrid__card__stat__number">{job.provider.length || 0}</span>
|
||||
<span className="jobGrid__card__stat__label">
|
||||
<IconBriefcase size="small" /> Providers
|
||||
</span>
|
||||
</div>
|
||||
<div className="jobGrid__card__stat jobGrid__card__stat--purple">
|
||||
<span className="jobGrid__card__stat__number">{job.notificationAdapter.length || 0}</span>
|
||||
<span className="jobGrid__card__stat__label">
|
||||
<IconBell size="small" /> Adapters
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<Divider margin="12px" />
|
||||
|
||||
<div className="jobGrid__card__footer">
|
||||
<div style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
|
||||
<Switch
|
||||
onChange={(checked) => onJobStatusChanged(job.id, checked)}
|
||||
checked={job.enabled}
|
||||
disabled={job.isOnlyShared}
|
||||
size="small"
|
||||
/>
|
||||
<Text type="secondary" size="small">
|
||||
Active
|
||||
</Text>
|
||||
</div>
|
||||
<div className="jobGrid__actions">
|
||||
<Popover content={getPopoverContent('Run Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="primary"
|
||||
style={{ background: '#21aa21b5' }}
|
||||
size="small"
|
||||
theme="solid"
|
||||
icon={<IconPlayCircle />}
|
||||
disabled={job.isOnlyShared || job.running}
|
||||
onClick={() => onJobRun(job.id)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
<Popover content={getPopoverContent('Edit a Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="secondary"
|
||||
size="small"
|
||||
icon={<IconEdit />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => navigate(`/jobs/edit/${job.id}`)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
<Popover content={getPopoverContent('Clone Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="tertiary"
|
||||
size="small"
|
||||
icon={<IconCopy />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => navigate('/jobs/new', { state: { cloneFrom: job.id } })}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
<Popover content={getPopoverContent('Delete all found Listings of this Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="danger"
|
||||
size="small"
|
||||
icon={<IconDescend2 />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onListingRemoval(job.id)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
<Popover content={getPopoverContent('Delete Job')}>
|
||||
<div>
|
||||
<Button
|
||||
type="danger"
|
||||
size="small"
|
||||
icon={<IconDelete />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onJobRemoval(job.id)}
|
||||
/>
|
||||
</div>
|
||||
</Popover>
|
||||
</div>
|
||||
</div>
|
||||
</Card>
|
||||
</Col>
|
||||
))}
|
||||
</Row>
|
||||
</Card>
|
||||
</Col>
|
||||
))}
|
||||
</Row>
|
||||
) : (
|
||||
<JobsTable
|
||||
jobs={jobsData?.result || []}
|
||||
onRun={onJobRun}
|
||||
onEdit={(id) => navigate(`/jobs/edit/${id}`)}
|
||||
onClone={(id) => navigate('/jobs/new', { state: { cloneFrom: id } })}
|
||||
onDeleteListings={onListingRemoval}
|
||||
onDeleteJob={onJobRemoval}
|
||||
onStatusChange={onJobStatusChanged}
|
||||
/>
|
||||
)}
|
||||
{(jobsData?.result || []).length > 0 && jobsData?.totalNumber > 12 && (
|
||||
<div className="jobGrid__pagination">
|
||||
<Pagination
|
||||
@@ -389,6 +435,7 @@ const JobGrid = () => {
|
||||
visible={deleteModalVisible}
|
||||
title={pendingDeletion?.type === 'job' ? 'Delete Job' : 'Delete Listings'}
|
||||
showOptions={pendingDeletion?.type !== 'job'}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
message={
|
||||
pendingDeletion?.type === 'job'
|
||||
? 'Are you sure you want to delete this job? All associated listings will be removed from the database.'
|
||||
|
||||
@@ -17,6 +17,12 @@
|
||||
flex: 1;
|
||||
min-width: 160px;
|
||||
}
|
||||
|
||||
&__view-toggle {
|
||||
display: flex;
|
||||
gap: 4px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
}
|
||||
|
||||
&__card {
|
||||
|
||||
@@ -3,14 +3,7 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { useState, useEffect, useMemo } from 'react';
|
||||
import {
|
||||
useSearchParamState,
|
||||
parseNumber,
|
||||
parseString,
|
||||
parseNullableBoolean,
|
||||
} from '../../../hooks/useSearchParamState.js';
|
||||
import { Button, Pagination, Toast, Input, Select, Empty, Radio, RadioGroup, Tooltip } from '@douyinfe/semi-ui-19';
|
||||
import { Button, Tooltip } from '@douyinfe/semi-ui-19';
|
||||
import {
|
||||
IconBriefcase,
|
||||
IconCart,
|
||||
@@ -19,323 +12,126 @@ import {
|
||||
IconMapPin,
|
||||
IconStar,
|
||||
IconStarStroked,
|
||||
IconSearch,
|
||||
IconEyeOpened,
|
||||
IconArrowUp,
|
||||
IconArrowDown,
|
||||
} from '@douyinfe/semi-icons';
|
||||
import { useNavigate, useSearchParams } from 'react-router-dom';
|
||||
import ListingDeletionModal from '../../ListingDeletionModal.jsx';
|
||||
import no_image from '../../../assets/no_image.png';
|
||||
import * as timeService from '../../../services/time/timeService.js';
|
||||
import { xhrDelete, xhrPost } from '../../../services/xhr.js';
|
||||
import { useActions, useSelector } from '../../../services/state/store.js';
|
||||
import { debounce } from '../../../utils';
|
||||
import StatusControl from '../../listings/StatusControl.jsx';
|
||||
|
||||
import './ListingsGrid.less';
|
||||
import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-illustrations';
|
||||
|
||||
const ListingsGrid = () => {
|
||||
const listingsData = useSelector((state) => state.listingsData);
|
||||
const providers = useSelector((state) => state.provider);
|
||||
const jobs = useSelector((state) => state.jobsData.jobs);
|
||||
const actions = useActions();
|
||||
const navigate = useNavigate();
|
||||
const sp = useSearchParams();
|
||||
|
||||
const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
|
||||
const pageSize = 40;
|
||||
|
||||
const [sortField, setSortField] = useSearchParamState(sp, 'sort', 'created_at', parseString);
|
||||
const [sortDir, setSortDir] = useSearchParamState(sp, 'dir', 'desc', parseString);
|
||||
const [freeTextFilter, setFreeTextFilter] = useSearchParamState(sp, 'q', null, parseString);
|
||||
const [watchListFilter, setWatchListFilter] = useSearchParamState(sp, 'watch', null, parseNullableBoolean);
|
||||
const [jobNameFilter, setJobNameFilter] = useSearchParamState(sp, 'job', null, parseString);
|
||||
const [activityFilter, setActivityFilter] = useSearchParamState(sp, 'active', null, parseNullableBoolean);
|
||||
const [providerFilter, setProviderFilter] = useSearchParamState(sp, 'provider', null, parseString);
|
||||
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
|
||||
const [listingToDelete, setListingToDelete] = useState(null);
|
||||
|
||||
const loadData = () => {
|
||||
actions.listingsData.getListingsData({
|
||||
page,
|
||||
pageSize,
|
||||
sortfield: sortField,
|
||||
sortdir: sortDir,
|
||||
freeTextFilter,
|
||||
filter: { watchListFilter, jobNameFilter, activityFilter, providerFilter },
|
||||
});
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
loadData();
|
||||
}, [page, sortField, sortDir, freeTextFilter, providerFilter, activityFilter, jobNameFilter, watchListFilter]);
|
||||
|
||||
const handleFilterChange = useMemo(
|
||||
() =>
|
||||
debounce((value) => {
|
||||
setFreeTextFilter(value || null);
|
||||
setPage(1);
|
||||
}, 500),
|
||||
[],
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
// cleanup debounced handler to avoid memory leaks
|
||||
handleFilterChange.cancel && handleFilterChange.cancel();
|
||||
};
|
||||
}, [handleFilterChange]);
|
||||
|
||||
const handleWatch = async (e, item) => {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
try {
|
||||
await xhrPost('/api/listings/watch', { listingId: item.id });
|
||||
Toast.success(item.isWatched === 1 ? 'Listing removed from Watchlist' : 'Listing added to Watchlist');
|
||||
loadData();
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
Toast.error('Failed to operate Watchlist');
|
||||
}
|
||||
};
|
||||
|
||||
const handlePageChange = (_page) => {
|
||||
setPage(_page);
|
||||
};
|
||||
|
||||
const confirmDeletion = async (hardDelete) => {
|
||||
try {
|
||||
await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
|
||||
Toast.success('Listing successfully removed');
|
||||
loadData();
|
||||
} catch (error) {
|
||||
Toast.error(error.message || 'Error deleting listing');
|
||||
} finally {
|
||||
setDeleteModalVisible(false);
|
||||
setListingToDelete(null);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="listingsGrid">
|
||||
<div className="listingsGrid__topbar">
|
||||
<Input
|
||||
className="listingsGrid__topbar__search"
|
||||
prefix={<IconSearch />}
|
||||
showClear
|
||||
placeholder="Search"
|
||||
defaultValue={freeTextFilter ?? ''}
|
||||
onChange={handleFilterChange}
|
||||
/>
|
||||
|
||||
<RadioGroup
|
||||
type="button"
|
||||
buttonSize="middle"
|
||||
value={activityFilter === null ? 'all' : String(activityFilter)}
|
||||
onChange={(e) => {
|
||||
const v = e.target.value;
|
||||
setActivityFilter(v === 'all' ? null : v === 'true');
|
||||
setPage(1);
|
||||
}}
|
||||
>
|
||||
<Radio value="all">All</Radio>
|
||||
<Radio value="true">Active</Radio>
|
||||
<Radio value="false">Inactive</Radio>
|
||||
</RadioGroup>
|
||||
|
||||
<RadioGroup
|
||||
type="button"
|
||||
buttonSize="middle"
|
||||
value={watchListFilter === null ? 'all' : String(watchListFilter)}
|
||||
onChange={(e) => {
|
||||
const v = e.target.value;
|
||||
setWatchListFilter(v === 'all' ? null : v === 'true');
|
||||
setPage(1);
|
||||
}}
|
||||
>
|
||||
<Radio value="all">All</Radio>
|
||||
<Radio value="true">Watched</Radio>
|
||||
<Radio value="false">Unwatched</Radio>
|
||||
</RadioGroup>
|
||||
|
||||
<Select
|
||||
placeholder="Provider"
|
||||
showClear
|
||||
onChange={(val) => {
|
||||
setProviderFilter(val);
|
||||
setPage(1);
|
||||
}}
|
||||
value={providerFilter}
|
||||
style={{ width: 130 }}
|
||||
>
|
||||
{providers?.map((p) => (
|
||||
<Select.Option key={p.id} value={p.id}>
|
||||
{p.name}
|
||||
</Select.Option>
|
||||
))}
|
||||
</Select>
|
||||
|
||||
<Select
|
||||
placeholder="Job"
|
||||
showClear
|
||||
onChange={(val) => {
|
||||
setJobNameFilter(val);
|
||||
setPage(1);
|
||||
}}
|
||||
value={jobNameFilter}
|
||||
style={{ width: 130 }}
|
||||
>
|
||||
{jobs?.map((j) => (
|
||||
<Select.Option key={j.id} value={j.id}>
|
||||
{j.name}
|
||||
</Select.Option>
|
||||
))}
|
||||
</Select>
|
||||
|
||||
<Select prefix="Sort by" style={{ width: 185 }} value={sortField} onChange={(val) => setSortField(val)}>
|
||||
<Select.Option value="job_name">Job Name</Select.Option>
|
||||
<Select.Option value="created_at">Listing Date</Select.Option>
|
||||
<Select.Option value="price">Price</Select.Option>
|
||||
<Select.Option value="provider">Provider</Select.Option>
|
||||
</Select>
|
||||
|
||||
<Button
|
||||
icon={sortDir === 'asc' ? <IconArrowUp /> : <IconArrowDown />}
|
||||
onClick={() => setSortDir((d) => (d === 'asc' ? 'desc' : 'asc'))}
|
||||
title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{(listingsData?.result || []).length === 0 && (
|
||||
<Empty
|
||||
image={<IllustrationNoResult />}
|
||||
darkModeImage={<IllustrationNoResultDark />}
|
||||
description="No listings available yet..."
|
||||
/>
|
||||
)}
|
||||
<div className="listingsGrid__grid">
|
||||
{(listingsData?.result || []).map((item) => (
|
||||
<div
|
||||
key={item.id}
|
||||
className="listingsGrid__card"
|
||||
style={{ cursor: 'pointer' }}
|
||||
role="button"
|
||||
tabIndex={0}
|
||||
onClick={() => navigate(`/listings/listing/${item.id}`)}
|
||||
onKeyDown={(e) => {
|
||||
if (e.key === 'Enter' || e.key === ' ') navigate(`/listings/listing/${item.id}`);
|
||||
}}
|
||||
>
|
||||
<div className="listingsGrid__card__image-wrapper">
|
||||
<img
|
||||
src={item.image_url || no_image}
|
||||
alt={item.title}
|
||||
onError={(e) => {
|
||||
e.target.src = no_image;
|
||||
}}
|
||||
/>
|
||||
{!item.is_active && (
|
||||
<div className="listingsGrid__card__inactive-watermark">
|
||||
<span>Inactive</span>
|
||||
</div>
|
||||
)}
|
||||
<button
|
||||
type="button"
|
||||
className="listingsGrid__card__star"
|
||||
onClick={(e) => handleWatch(e, item)}
|
||||
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
|
||||
>
|
||||
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="listingsGrid__card__body">
|
||||
<div className="listingsGrid__card__title" title={item.title}>
|
||||
{item.title}
|
||||
</div>
|
||||
{item.price && (
|
||||
<div className="listingsGrid__card__price">
|
||||
<IconCart size="small" />
|
||||
{item.price}
|
||||
</div>
|
||||
)}
|
||||
{item.address && (
|
||||
<div className="listingsGrid__card__meta">
|
||||
<IconMapPin />
|
||||
{item.address}
|
||||
</div>
|
||||
)}
|
||||
<div className="listingsGrid__card__meta">
|
||||
<IconBriefcase />
|
||||
{item.provider}
|
||||
</div>
|
||||
<div className="listingsGrid__card__provider">{timeService.format(item.created_at, false)}</div>
|
||||
</div>
|
||||
|
||||
<div className="listingsGrid__card__actions" onClick={(e) => e.stopPropagation()}>
|
||||
<Tooltip content="Original Listing">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconLink />}
|
||||
style={{ color: '#60a5fa' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
window.open(item.link, '_blank');
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="View in Fredy">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconEyeOpened />}
|
||||
style={{ color: '#34d399' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
navigate(`/listings/listing/${item.id}`);
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Remove">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconDelete />}
|
||||
style={{ color: '#fb7185' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
setListingToDelete(item.id);
|
||||
setDeleteModalVisible(true);
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
{(listingsData?.result || []).length > 0 && (
|
||||
<div className="listingsGrid__pagination">
|
||||
<Pagination
|
||||
currentPage={page}
|
||||
pageSize={pageSize}
|
||||
total={listingsData?.totalNumber || 0}
|
||||
onPageChange={handlePageChange}
|
||||
showSizeChanger={false}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
<ListingDeletionModal
|
||||
visible={deleteModalVisible}
|
||||
onConfirm={confirmDeletion}
|
||||
onCancel={() => {
|
||||
setDeleteModalVisible(false);
|
||||
setListingToDelete(null);
|
||||
/**
|
||||
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function, onStatusChange: Function }} props
|
||||
*/
|
||||
const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete, onStatusChange }) => (
|
||||
<div className="listingsGrid__grid">
|
||||
{listings.map((item) => (
|
||||
<div
|
||||
key={item.id}
|
||||
className="listingsGrid__card"
|
||||
style={{ cursor: 'pointer' }}
|
||||
role="button"
|
||||
tabIndex={0}
|
||||
onClick={() => onNavigate(item.id)}
|
||||
onKeyDown={(e) => {
|
||||
if (e.key === 'Enter' || e.key === ' ') onNavigate(item.id);
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
>
|
||||
<div className="listingsGrid__card__image-wrapper">
|
||||
<img
|
||||
src={item.image_url || no_image}
|
||||
alt={item.title}
|
||||
onError={(e) => {
|
||||
e.target.src = no_image;
|
||||
}}
|
||||
/>
|
||||
{!item.is_active && (
|
||||
<div className="listingsGrid__card__inactive-watermark">
|
||||
<span>Inactive</span>
|
||||
</div>
|
||||
)}
|
||||
<Tooltip content={item.isWatched === 1 ? 'Remove from Watchlist' : 'Add to Watchlist'}>
|
||||
<button
|
||||
type="button"
|
||||
className="listingsGrid__card__star"
|
||||
onClick={(e) => onWatch(e, item)}
|
||||
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
|
||||
>
|
||||
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
|
||||
</button>
|
||||
</Tooltip>
|
||||
</div>
|
||||
|
||||
<div className="listingsGrid__card__body">
|
||||
<div className="listingsGrid__card__title" title={item.title}>
|
||||
{item.title}
|
||||
</div>
|
||||
{item.price && (
|
||||
<div className="listingsGrid__card__price">
|
||||
<IconCart size="small" />
|
||||
{item.price}
|
||||
</div>
|
||||
)}
|
||||
{item.address && (
|
||||
<div className="listingsGrid__card__meta">
|
||||
<IconMapPin />
|
||||
{item.address}
|
||||
</div>
|
||||
)}
|
||||
<div className="listingsGrid__card__meta">
|
||||
<IconBriefcase />
|
||||
{item.provider}
|
||||
</div>
|
||||
<div className="listingsGrid__card__provider">{timeService.format(item.created_at, false)}</div>
|
||||
</div>
|
||||
|
||||
<div className="listingsGrid__card__actions" onClick={(e) => e.stopPropagation()}>
|
||||
<StatusControl
|
||||
status={item.status?.status ?? null}
|
||||
compact
|
||||
onChange={(next) => onStatusChange?.(item, next)}
|
||||
onTriggerClick={(e) => e.stopPropagation()}
|
||||
/>
|
||||
<Tooltip content="Original Listing">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconLink />}
|
||||
style={{ color: '#60a5fa' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
window.open(item.link, '_blank');
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="View in Fredy">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconEyeOpened />}
|
||||
style={{ color: '#34d399' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
onNavigate(item.id);
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Remove">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconDelete />}
|
||||
style={{ color: '#fb7185' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
onDelete(item.id);
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
|
||||
export default ListingsGrid;
|
||||
|
||||
@@ -1,181 +1,142 @@
|
||||
@import '../../../tokens.less';
|
||||
|
||||
.listingsGrid {
|
||||
&__topbar {
|
||||
.listingsGrid__grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(240px, 1fr));
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.listingsGrid__card {
|
||||
background: @color-elevated !important;
|
||||
border: 1px solid @color-border !important;
|
||||
border-radius: @radius-card !important;
|
||||
overflow: hidden;
|
||||
transition: box-shadow @transition-card;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
|
||||
&:hover {
|
||||
box-shadow: 0 6px 24px -4px rgba(0,0,0,0.6);
|
||||
}
|
||||
|
||||
&__image-wrapper {
|
||||
position: relative;
|
||||
height: 160px;
|
||||
overflow: hidden;
|
||||
flex-shrink: 0;
|
||||
|
||||
img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
display: block;
|
||||
}
|
||||
}
|
||||
|
||||
&__inactive-watermark {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: @space-3;
|
||||
margin-bottom: @space-4;
|
||||
flex-wrap: wrap;
|
||||
justify-content: center;
|
||||
background: rgba(0,0,0,0.35);
|
||||
|
||||
&__search {
|
||||
min-width: 200px;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.listingsGrid__topbar__search {
|
||||
width: 100%;
|
||||
flex: unset;
|
||||
}
|
||||
|
||||
.semi-radio-group {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.semi-select {
|
||||
flex: 1;
|
||||
min-width: 100px;
|
||||
width: auto !important;
|
||||
}
|
||||
span {
|
||||
font-size: 18px;
|
||||
font-weight: 800;
|
||||
color: rgba(251,113,133,0.9);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.15em;
|
||||
transform: rotate(-30deg);
|
||||
border: 2px solid rgba(251,113,133,0.5);
|
||||
padding: 4px 12px;
|
||||
border-radius: @radius-chip;
|
||||
backdrop-filter: blur(2px);
|
||||
}
|
||||
}
|
||||
|
||||
&__grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(240px, 1fr));
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
&__card {
|
||||
background: @color-elevated !important;
|
||||
border: 1px solid @color-border !important;
|
||||
border-radius: @radius-card !important;
|
||||
overflow: hidden;
|
||||
transition: transform @transition-card, box-shadow @transition-card;
|
||||
&__star {
|
||||
position: absolute;
|
||||
top: 8px;
|
||||
right: 8px;
|
||||
background: rgba(0,0,0,0.5);
|
||||
border: none;
|
||||
border-radius: 50%;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
cursor: pointer;
|
||||
transition: background @transition-fast;
|
||||
padding: 0;
|
||||
|
||||
&:hover {
|
||||
transform: translateY(-2px);
|
||||
box-shadow: 0 6px 24px -4px rgba(0,0,0,0.6);
|
||||
background: rgba(0,0,0,0.75);
|
||||
}
|
||||
|
||||
&__image-wrapper {
|
||||
position: relative;
|
||||
height: 160px;
|
||||
overflow: hidden;
|
||||
flex-shrink: 0;
|
||||
|
||||
img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
display: block;
|
||||
}
|
||||
svg {
|
||||
color: @color-accent;
|
||||
font-size: 14px;
|
||||
}
|
||||
}
|
||||
|
||||
&__inactive-watermark {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
background: rgba(0,0,0,0.35);
|
||||
&__body {
|
||||
padding: 12px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
span {
|
||||
font-size: 18px;
|
||||
font-weight: 800;
|
||||
color: rgba(251,113,133,0.9);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.15em;
|
||||
transform: rotate(-30deg);
|
||||
border: 2px solid rgba(251,113,133,0.5);
|
||||
padding: 4px 12px;
|
||||
border-radius: @radius-chip;
|
||||
backdrop-filter: blur(2px);
|
||||
}
|
||||
}
|
||||
&__title {
|
||||
font-weight: 700;
|
||||
font-size: @text-sm;
|
||||
color: @color-text;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__star {
|
||||
position: absolute;
|
||||
top: 8px;
|
||||
right: 8px;
|
||||
background: rgba(0,0,0,0.5);
|
||||
border: none;
|
||||
border-radius: 50%;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
cursor: pointer;
|
||||
transition: background @transition-fast;
|
||||
padding: 0;
|
||||
&__price {
|
||||
font-size: @text-base;
|
||||
font-weight: 600;
|
||||
color: @color-success;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
&:hover {
|
||||
background: rgba(0,0,0,0.75);
|
||||
}
|
||||
&__meta {
|
||||
font-size: @text-xs;
|
||||
color: @color-muted;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
|
||||
svg {
|
||||
color: @color-accent;
|
||||
font-size: 14px;
|
||||
}
|
||||
}
|
||||
|
||||
&__body {
|
||||
padding: 12px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
&__title {
|
||||
font-weight: 700;
|
||||
font-size: @text-sm;
|
||||
color: @color-text;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__price {
|
||||
font-size: @text-base;
|
||||
font-weight: 600;
|
||||
color: @color-success;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
&__meta {
|
||||
font-size: @text-xs;
|
||||
color: @color-muted;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
|
||||
.semi-icon {
|
||||
font-size: 11px;
|
||||
color: @color-faint;
|
||||
}
|
||||
}
|
||||
|
||||
&__provider {
|
||||
font-size: @text-xs;
|
||||
.semi-icon {
|
||||
font-size: 11px;
|
||||
color: @color-faint;
|
||||
}
|
||||
}
|
||||
|
||||
&__actions {
|
||||
display: flex;
|
||||
justify-content: space-around;
|
||||
padding: 8px 12px;
|
||||
border-top: 1px solid @color-border;
|
||||
gap: 4px;
|
||||
margin-top: auto;
|
||||
&__provider {
|
||||
font-size: @text-xs;
|
||||
color: @color-faint;
|
||||
}
|
||||
|
||||
button {
|
||||
flex: 1;
|
||||
border: none !important;
|
||||
border-radius: @radius-chip !important;
|
||||
}
|
||||
&__actions {
|
||||
display: flex;
|
||||
justify-content: space-around;
|
||||
padding: 8px 12px;
|
||||
border-top: 1px solid @color-border;
|
||||
gap: 4px;
|
||||
margin-top: auto;
|
||||
|
||||
button {
|
||||
flex: 1;
|
||||
border: none !important;
|
||||
border-radius: @radius-chip !important;
|
||||
}
|
||||
}
|
||||
|
||||
&__pagination {
|
||||
margin-top: @space-4;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
}
|
||||
}
|
||||
|
||||
343
ui/src/components/listings/ListingsOverview.jsx
Normal file
343
ui/src/components/listings/ListingsOverview.jsx
Normal file
@@ -0,0 +1,343 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { useState, useEffect, useMemo } from 'react';
|
||||
import {
|
||||
useSearchParamState,
|
||||
parseNumber,
|
||||
parseString,
|
||||
parseNullableBoolean,
|
||||
} from '../../hooks/useSearchParamState.js';
|
||||
import { Button, Pagination, Toast, Input, Select, Empty, Radio, RadioGroup, Tooltip } from '@douyinfe/semi-ui-19';
|
||||
import { IconSearch, IconArrowUp, IconArrowDown, IconGridView, IconList } from '@douyinfe/semi-icons';
|
||||
import { useNavigate, useSearchParams } from 'react-router-dom';
|
||||
import ListingDeletionModal from '../ListingDeletionModal.jsx';
|
||||
import { xhrDelete, xhrPost } from '../../services/xhr.js';
|
||||
import { useActions, useSelector } from '../../services/state/store.js';
|
||||
import { debounce } from '../../utils';
|
||||
import ListingsGrid from '../grid/listings/ListingsGrid.jsx';
|
||||
import ListingsTable from '../table/ListingsTable.jsx';
|
||||
import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-illustrations';
|
||||
|
||||
import './ListingsOverview.less';
|
||||
|
||||
const ListingsOverview = ({ mode = 'all' }) => {
|
||||
const isWatchlistMode = mode === 'watchlist';
|
||||
const listingsData = useSelector((state) => state.listingsData);
|
||||
const providers = useSelector((state) => state.provider);
|
||||
const jobs = useSelector((state) => state.jobsData.jobs);
|
||||
const userSettings = useSelector((state) => state.userSettings.settings);
|
||||
const actions = useActions();
|
||||
const navigate = useNavigate();
|
||||
const sp = useSearchParams();
|
||||
|
||||
const viewMode = userSettings?.listings_view_mode ?? 'grid';
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
|
||||
const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
|
||||
const pageSize = 40;
|
||||
|
||||
const [sortField, setSortField] = useSearchParamState(sp, 'sort', 'created_at', parseString);
|
||||
const [sortDir, setSortDir] = useSearchParamState(sp, 'dir', 'desc', parseString);
|
||||
const [freeTextFilter, setFreeTextFilter] = useSearchParamState(sp, 'q', null, parseString);
|
||||
const [watchListFilter, setWatchListFilter] = useSearchParamState(sp, 'watch', null, parseNullableBoolean);
|
||||
const [jobNameFilter, setJobNameFilter] = useSearchParamState(sp, 'job', null, parseString);
|
||||
const [activityFilter, setActivityFilter] = useSearchParamState(sp, 'active', null, parseNullableBoolean);
|
||||
const [providerFilter, setProviderFilter] = useSearchParamState(sp, 'provider', null, parseString);
|
||||
const [statusFilter, setStatusFilter] = useSearchParamState(sp, 'status', null, parseString);
|
||||
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
|
||||
const [listingToDelete, setListingToDelete] = useState(null);
|
||||
|
||||
// In watchlist mode the watch filter is forced to "watched only" — regardless of the URL.
|
||||
const effectiveWatchListFilter = isWatchlistMode ? true : watchListFilter;
|
||||
|
||||
const loadData = () => {
|
||||
actions.listingsData.getListingsData({
|
||||
page,
|
||||
pageSize,
|
||||
sortfield: sortField,
|
||||
sortdir: sortDir,
|
||||
freeTextFilter,
|
||||
filter: {
|
||||
watchListFilter: effectiveWatchListFilter,
|
||||
jobNameFilter,
|
||||
activityFilter,
|
||||
providerFilter,
|
||||
statusFilter,
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
loadData();
|
||||
}, [
|
||||
page,
|
||||
sortField,
|
||||
sortDir,
|
||||
freeTextFilter,
|
||||
providerFilter,
|
||||
activityFilter,
|
||||
jobNameFilter,
|
||||
watchListFilter,
|
||||
statusFilter,
|
||||
isWatchlistMode,
|
||||
]);
|
||||
|
||||
const handleFilterChange = useMemo(
|
||||
() =>
|
||||
debounce((value) => {
|
||||
setFreeTextFilter(value || null);
|
||||
setPage(1);
|
||||
}, 500),
|
||||
[],
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
handleFilterChange.cancel && handleFilterChange.cancel();
|
||||
};
|
||||
}, [handleFilterChange]);
|
||||
|
||||
const handleWatch = async (e, item) => {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
try {
|
||||
await xhrPost('/api/listings/watch', { listingId: item.id });
|
||||
Toast.success(item.isWatched === 1 ? 'Listing removed from Watchlist' : 'Listing added to Watchlist');
|
||||
loadData();
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
Toast.error('Failed to operate Watchlist');
|
||||
}
|
||||
};
|
||||
|
||||
const handleStatusChange = async (item, nextStatus) => {
|
||||
try {
|
||||
await actions.listingsData.setListingStatus(item.id, nextStatus);
|
||||
Toast.success(nextStatus ? `Marked as ${nextStatus}` : 'Status cleared');
|
||||
loadData();
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
Toast.error('Failed to update status');
|
||||
}
|
||||
};
|
||||
|
||||
const handleDelete = (id) => {
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmDeletion(listingDeletionPref.hardDelete, false, id);
|
||||
return;
|
||||
}
|
||||
setListingToDelete(id);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
|
||||
const handleNavigate = (id) => navigate(`/listings/listing/${id}`);
|
||||
|
||||
const confirmDeletion = async (hardDelete, remember, id = listingToDelete) => {
|
||||
try {
|
||||
if (remember) {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
await xhrDelete('/api/listings/', { ids: [id], hardDelete });
|
||||
Toast.success('Listing successfully removed');
|
||||
loadData();
|
||||
} catch (error) {
|
||||
Toast.error(error.message || 'Error deleting listing');
|
||||
} finally {
|
||||
setDeleteModalVisible(false);
|
||||
setListingToDelete(null);
|
||||
}
|
||||
};
|
||||
|
||||
const listings = listingsData?.result || [];
|
||||
|
||||
return (
|
||||
<div className="listingsOverview">
|
||||
<div className="listingsOverview__topbar">
|
||||
<Input
|
||||
className="listingsOverview__topbar__search"
|
||||
prefix={<IconSearch />}
|
||||
showClear
|
||||
placeholder="Search"
|
||||
defaultValue={freeTextFilter ?? ''}
|
||||
onChange={handleFilterChange}
|
||||
/>
|
||||
|
||||
<RadioGroup
|
||||
type="button"
|
||||
buttonSize="middle"
|
||||
value={activityFilter === null ? 'all' : String(activityFilter)}
|
||||
onChange={(e) => {
|
||||
const v = e.target.value;
|
||||
setActivityFilter(v === 'all' ? null : v === 'true');
|
||||
setPage(1);
|
||||
}}
|
||||
>
|
||||
<Radio value="all">All</Radio>
|
||||
<Radio value="true">Active</Radio>
|
||||
<Radio value="false">Inactive</Radio>
|
||||
</RadioGroup>
|
||||
|
||||
{!isWatchlistMode && (
|
||||
<RadioGroup
|
||||
type="button"
|
||||
buttonSize="middle"
|
||||
value={watchListFilter === null ? 'all' : String(watchListFilter)}
|
||||
onChange={(e) => {
|
||||
const v = e.target.value;
|
||||
setWatchListFilter(v === 'all' ? null : v === 'true');
|
||||
setPage(1);
|
||||
}}
|
||||
>
|
||||
<Radio value="all">All</Radio>
|
||||
<Radio value="true">Watched</Radio>
|
||||
<Radio value="false">Unwatched</Radio>
|
||||
</RadioGroup>
|
||||
)}
|
||||
|
||||
<Select
|
||||
placeholder="Status"
|
||||
showClear
|
||||
onChange={(val) => {
|
||||
setStatusFilter(val ?? null);
|
||||
setPage(1);
|
||||
}}
|
||||
value={statusFilter}
|
||||
style={{ width: 150 }}
|
||||
>
|
||||
<Select.Option value="applied">Applied</Select.Option>
|
||||
<Select.Option value="rejected">Rejected</Select.Option>
|
||||
<Select.Option value="accepted">Accepted</Select.Option>
|
||||
<Select.Option value="none">No status</Select.Option>
|
||||
</Select>
|
||||
|
||||
<Select
|
||||
placeholder="Provider"
|
||||
showClear
|
||||
onChange={(val) => {
|
||||
setProviderFilter(val);
|
||||
setPage(1);
|
||||
}}
|
||||
value={providerFilter}
|
||||
style={{ width: 130 }}
|
||||
>
|
||||
{providers?.map((p) => (
|
||||
<Select.Option key={p.id} value={p.id}>
|
||||
{p.name}
|
||||
</Select.Option>
|
||||
))}
|
||||
</Select>
|
||||
|
||||
<Select
|
||||
placeholder="Job"
|
||||
showClear
|
||||
onChange={(val) => {
|
||||
setJobNameFilter(val);
|
||||
setPage(1);
|
||||
}}
|
||||
value={jobNameFilter}
|
||||
style={{ width: 130 }}
|
||||
>
|
||||
{jobs?.map((j) => (
|
||||
<Select.Option key={j.id} value={j.id}>
|
||||
{j.name}
|
||||
</Select.Option>
|
||||
))}
|
||||
</Select>
|
||||
|
||||
<Select
|
||||
prefix="Sort by"
|
||||
className="listingsOverview__topbar__sort"
|
||||
style={{ width: 220 }}
|
||||
value={sortField}
|
||||
onChange={(val) => setSortField(val)}
|
||||
>
|
||||
<Select.Option value="job_name">Job Name</Select.Option>
|
||||
<Select.Option value="created_at">Listing Date</Select.Option>
|
||||
<Select.Option value="price">Price</Select.Option>
|
||||
<Select.Option value="provider">Provider</Select.Option>
|
||||
</Select>
|
||||
|
||||
<Button
|
||||
icon={sortDir === 'asc' ? <IconArrowUp /> : <IconArrowDown />}
|
||||
onClick={() => setSortDir(sortDir === 'asc' ? 'desc' : 'asc')}
|
||||
title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
|
||||
/>
|
||||
|
||||
<div className="listingsOverview__topbar__view-toggle">
|
||||
<Tooltip content="Grid view">
|
||||
<Button
|
||||
icon={<IconGridView />}
|
||||
theme={viewMode === 'grid' ? 'solid' : 'borderless'}
|
||||
onClick={() => actions.userSettings.setListingsViewMode('grid')}
|
||||
aria-label="Grid view"
|
||||
aria-pressed={viewMode === 'grid'}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Table view">
|
||||
<Button
|
||||
icon={<IconList />}
|
||||
theme={viewMode === 'table' ? 'solid' : 'borderless'}
|
||||
onClick={() => actions.userSettings.setListingsViewMode('table')}
|
||||
aria-label="Table view"
|
||||
aria-pressed={viewMode === 'table'}
|
||||
/>
|
||||
</Tooltip>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{listings.length === 0 && (
|
||||
<Empty
|
||||
image={<IllustrationNoResult />}
|
||||
darkModeImage={<IllustrationNoResultDark />}
|
||||
description="No listings available yet..."
|
||||
/>
|
||||
)}
|
||||
|
||||
{viewMode === 'grid' ? (
|
||||
<ListingsGrid
|
||||
listings={listings}
|
||||
onWatch={handleWatch}
|
||||
onNavigate={handleNavigate}
|
||||
onDelete={handleDelete}
|
||||
onStatusChange={handleStatusChange}
|
||||
/>
|
||||
) : (
|
||||
<ListingsTable
|
||||
listings={listings}
|
||||
onWatch={handleWatch}
|
||||
onNavigate={handleNavigate}
|
||||
onDelete={handleDelete}
|
||||
onStatusChange={handleStatusChange}
|
||||
/>
|
||||
)}
|
||||
|
||||
{listings.length > 0 && (
|
||||
<div className="listingsOverview__pagination">
|
||||
<Pagination
|
||||
currentPage={page}
|
||||
pageSize={pageSize}
|
||||
total={listingsData?.totalNumber || 0}
|
||||
onPageChange={setPage}
|
||||
showSizeChanger={false}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<ListingDeletionModal
|
||||
visible={deleteModalVisible}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
onConfirm={confirmDeletion}
|
||||
onCancel={() => {
|
||||
setDeleteModalVisible(false);
|
||||
setListingToDelete(null);
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
export default ListingsOverview;
|
||||
53
ui/src/components/listings/ListingsOverview.less
Normal file
53
ui/src/components/listings/ListingsOverview.less
Normal file
@@ -0,0 +1,53 @@
|
||||
@import '../../tokens.less';
|
||||
|
||||
.listingsOverview {
|
||||
&__topbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: @space-3;
|
||||
margin-bottom: @space-4;
|
||||
flex-wrap: wrap;
|
||||
|
||||
&__search {
|
||||
min-width: 200px;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
&__view-toggle {
|
||||
display: flex;
|
||||
gap: 2px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
&__sort {
|
||||
flex-shrink: 0;
|
||||
|
||||
.semi-select-prefix {
|
||||
white-space: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.listingsOverview__topbar__search {
|
||||
width: 100%;
|
||||
flex: unset;
|
||||
}
|
||||
|
||||
.semi-radio-group {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.semi-select {
|
||||
flex: 1;
|
||||
min-width: 100px;
|
||||
width: auto !important;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
&__pagination {
|
||||
margin-top: @space-4;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
}
|
||||
}
|
||||
115
ui/src/components/listings/StatusControl.jsx
Normal file
115
ui/src/components/listings/StatusControl.jsx
Normal file
@@ -0,0 +1,115 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { useState } from 'react';
|
||||
import { Dropdown, Button, Tooltip } from '@douyinfe/semi-ui-19';
|
||||
import { IconChevronDown } from '@douyinfe/semi-icons';
|
||||
|
||||
import './StatusControl.less';
|
||||
|
||||
const STATUS_TOOLTIP =
|
||||
'Track where you stand with this listing: Applied once you have reached out, Rejected if it did not work out, or Accepted if you got it.';
|
||||
|
||||
/**
|
||||
* @typedef {('applied'|'rejected'|'accepted'|null)} ListingStatus
|
||||
*/
|
||||
|
||||
const STATUS_OPTIONS = [
|
||||
{ value: null, label: 'None' },
|
||||
{ value: 'applied', label: 'Applied' },
|
||||
{ value: 'rejected', label: 'Rejected' },
|
||||
{ value: 'accepted', label: 'Accepted' },
|
||||
];
|
||||
|
||||
/**
|
||||
* Look up the option metadata for a status value.
|
||||
* @param {ListingStatus} status
|
||||
*/
|
||||
const optionFor = (status) => STATUS_OPTIONS.find((o) => o.value === status) ?? STATUS_OPTIONS[0];
|
||||
|
||||
/**
|
||||
* Shared control for setting a listing's user-decision status
|
||||
* (Applied / Rejected / Accepted).
|
||||
*
|
||||
* Both compact (table/grid rows) and full (listing detail header) modes
|
||||
* render a Button that picks up the project's CI tokens via the
|
||||
* .status-btn classes, with a small size variant for compact contexts.
|
||||
*
|
||||
* @param {Object} props
|
||||
* @param {ListingStatus} props.status - The current status value.
|
||||
* @param {(next: ListingStatus) => void} props.onChange - Called with the new status when the user picks one.
|
||||
* @param {boolean} [props.compact=false] - When true, renders smaller for table/grid rows; full size otherwise.
|
||||
* @param {(e: React.MouseEvent) => void} [props.onTriggerClick] - Optional click handler to stop propagation on the trigger.
|
||||
*/
|
||||
export default function StatusControl({ status = null, onChange, compact = false, onTriggerClick }) {
|
||||
const [open, setOpen] = useState(false);
|
||||
const [tooltipOpen, setTooltipOpen] = useState(false);
|
||||
const current = optionFor(status);
|
||||
|
||||
const handlePick = (next) => {
|
||||
setOpen(false);
|
||||
if (next === status) return;
|
||||
onChange?.(next);
|
||||
};
|
||||
|
||||
const menu = (
|
||||
<Dropdown.Menu>
|
||||
{STATUS_OPTIONS.map((opt) => (
|
||||
<Dropdown.Item
|
||||
key={opt.value ?? '__none__'}
|
||||
active={opt.value === status}
|
||||
onClick={() => handlePick(opt.value)}
|
||||
>
|
||||
{opt.label}
|
||||
</Dropdown.Item>
|
||||
))}
|
||||
</Dropdown.Menu>
|
||||
);
|
||||
|
||||
const className = ['status-btn', compact ? 'status-btn--compact' : null, status ? `status-btn--${status}` : null]
|
||||
.filter(Boolean)
|
||||
.join(' ');
|
||||
|
||||
const trigger = (
|
||||
<Tooltip
|
||||
content={STATUS_TOOLTIP}
|
||||
position="top"
|
||||
trigger="custom"
|
||||
visible={tooltipOpen && !open}
|
||||
onVisibleChange={setTooltipOpen}
|
||||
>
|
||||
<Button
|
||||
size={compact ? 'small' : 'default'}
|
||||
theme="borderless"
|
||||
icon={<IconChevronDown />}
|
||||
iconPosition="right"
|
||||
onMouseEnter={() => setTooltipOpen(true)}
|
||||
onMouseLeave={() => setTooltipOpen(false)}
|
||||
onClick={(e) => {
|
||||
onTriggerClick?.(e);
|
||||
setTooltipOpen(false);
|
||||
setOpen((o) => !o);
|
||||
}}
|
||||
className={className}
|
||||
>
|
||||
{status ? current.label : 'Status'}
|
||||
</Button>
|
||||
</Tooltip>
|
||||
);
|
||||
|
||||
return (
|
||||
<Dropdown
|
||||
trigger="custom"
|
||||
visible={open}
|
||||
onVisibleChange={setOpen}
|
||||
onClickOutSide={() => setOpen(false)}
|
||||
position="bottom"
|
||||
render={menu}
|
||||
stopPropagation
|
||||
>
|
||||
<span className="status-btn__anchor">{trigger}</span>
|
||||
</Dropdown>
|
||||
);
|
||||
}
|
||||
64
ui/src/components/listings/StatusControl.less
Normal file
64
ui/src/components/listings/StatusControl.less
Normal file
@@ -0,0 +1,64 @@
|
||||
@import '../../tokens.less';
|
||||
|
||||
// Wrapper span used as the Dropdown's positioning anchor so the menu opens
|
||||
// directly below the visible button rather than the implicit wrapper of the
|
||||
// hover tooltip (which can have a different bounding box).
|
||||
.status-btn__anchor {
|
||||
display: inline-block;
|
||||
line-height: 0;
|
||||
}
|
||||
|
||||
// StatusControl shared base. Matches dimensions and border treatment
|
||||
// of the surrounding Watched / Open listing / Delete buttons in the
|
||||
// detail view, and shrinks via the --compact modifier for table rows
|
||||
// and grid cards.
|
||||
.status-btn {
|
||||
color: @color-muted !important;
|
||||
border: 1px solid @color-border-bright !important;
|
||||
border-radius: @radius-btn !important;
|
||||
background: transparent !important;
|
||||
transition: color @transition-fast, border-color @transition-fast, background @transition-fast;
|
||||
|
||||
&:hover {
|
||||
color: @color-text !important;
|
||||
background: rgba(255, 255, 255, 0.06) !important;
|
||||
}
|
||||
|
||||
&--compact {
|
||||
height: 24px !important;
|
||||
padding: 0 8px !important;
|
||||
font-size: @text-sm !important;
|
||||
border-radius: @radius-chip !important;
|
||||
|
||||
.semi-icon {
|
||||
font-size: 12px !important;
|
||||
}
|
||||
}
|
||||
|
||||
&--applied {
|
||||
color: @color-info !important;
|
||||
border-color: rgba(96, 165, 250, 0.4) !important;
|
||||
background: rgba(96, 165, 250, 0.08) !important;
|
||||
&:hover {
|
||||
background: rgba(96, 165, 250, 0.14) !important;
|
||||
}
|
||||
}
|
||||
|
||||
&--rejected {
|
||||
color: @color-error !important;
|
||||
border-color: rgba(251, 113, 133, 0.4) !important;
|
||||
background: rgba(251, 113, 133, 0.08) !important;
|
||||
&:hover {
|
||||
background: rgba(251, 113, 133, 0.14) !important;
|
||||
}
|
||||
}
|
||||
|
||||
&--accepted {
|
||||
color: @color-success !important;
|
||||
border-color: rgba(52, 211, 153, 0.4) !important;
|
||||
background: rgba(52, 211, 153, 0.08) !important;
|
||||
&:hover {
|
||||
background: rgba(52, 211, 153, 0.14) !important;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import maplibregl from 'maplibre-gl';
|
||||
import 'maplibre-gl/dist/maplibre-gl.css';
|
||||
import '@mapbox/mapbox-gl-draw/dist/mapbox-gl-draw.css';
|
||||
import { fixMapboxDrawCompatibility, addDrawingControl, setupAreaFilterEventListeners } from './MapDrawingExtension.js';
|
||||
import { getBoundsFromCoords } from '../../views/listings/mapUtils.js';
|
||||
import './Map.less';
|
||||
|
||||
export const GERMANY_BOUNDS = [
|
||||
@@ -66,6 +67,7 @@ export default function Map({
|
||||
const mapContainerRef = useRef(null);
|
||||
const mapRef = useRef(null);
|
||||
const drawRef = useRef(null);
|
||||
const hasFittedToInitialAreaRef = useRef(false);
|
||||
|
||||
// Initialize map - ONLY when container changes, never reinitialize
|
||||
useEffect(() => {
|
||||
@@ -128,6 +130,17 @@ export default function Map({
|
||||
} catch (error) {
|
||||
console.error('Error loading spatial filter:', error);
|
||||
}
|
||||
|
||||
if (!hasFittedToInitialAreaRef.current) {
|
||||
const coords = initialSpatialFilter.features.flatMap((feature) =>
|
||||
feature.geometry?.type === 'Polygon' ? feature.geometry.coordinates.flat() : [],
|
||||
);
|
||||
const bounds = getBoundsFromCoords(coords);
|
||||
if (bounds) {
|
||||
mapRef.current.fitBounds(bounds, { padding: 50, maxZoom: 15, duration: 0 });
|
||||
hasFittedToInitialAreaRef.current = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Setup drawing event listeners
|
||||
|
||||
@@ -37,6 +37,7 @@ export default function Navigation({ isAdmin }) {
|
||||
items: [
|
||||
{ itemKey: '/listings', text: 'Overview' },
|
||||
{ itemKey: '/map', text: 'Map View' },
|
||||
{ itemKey: '/listings/watchlist', text: 'Watchlist' },
|
||||
],
|
||||
},
|
||||
];
|
||||
@@ -61,6 +62,22 @@ export default function Navigation({ isAdmin }) {
|
||||
}
|
||||
|
||||
function parsePathName(name) {
|
||||
// Collect every leaf itemKey that looks like a route (starts with '/').
|
||||
// Prefer the longest exact-prefix match so nested routes like
|
||||
// '/listings/watchlist' resolve to themselves instead of being collapsed
|
||||
// to '/listings'.
|
||||
const allKeys = [];
|
||||
const collect = (nodes) => {
|
||||
for (const n of nodes) {
|
||||
if (typeof n.itemKey === 'string' && n.itemKey.startsWith('/')) allKeys.push(n.itemKey);
|
||||
if (Array.isArray(n.items)) collect(n.items);
|
||||
}
|
||||
};
|
||||
collect(items);
|
||||
const longestMatch = allKeys
|
||||
.filter((k) => name === k || name.startsWith(k + '/'))
|
||||
.sort((a, b) => b.length - a.length)[0];
|
||||
if (longestMatch) return longestMatch;
|
||||
const split = name.split('/').filter((s) => s.length !== 0);
|
||||
return '/' + split[0];
|
||||
}
|
||||
|
||||
128
ui/src/components/table/JobsTable.jsx
Normal file
128
ui/src/components/table/JobsTable.jsx
Normal file
@@ -0,0 +1,128 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { Button, Tag, Tooltip, Switch } from '@douyinfe/semi-ui-19';
|
||||
import {
|
||||
IconAlertTriangle,
|
||||
IconBell,
|
||||
IconBriefcase,
|
||||
IconCopy,
|
||||
IconDelete,
|
||||
IconDescend2,
|
||||
IconEdit,
|
||||
IconHome,
|
||||
IconPlayCircle,
|
||||
} from '@douyinfe/semi-icons';
|
||||
|
||||
import './JobsTable.less';
|
||||
|
||||
/**
|
||||
* @param {{ jobs: object[], onRun: Function, onEdit: Function, onClone: Function, onDeleteListings: Function, onDeleteJob: Function, onStatusChange: Function }} props
|
||||
*/
|
||||
const JobsTable = ({ jobs, onRun, onEdit, onClone, onDeleteListings, onDeleteJob, onStatusChange }) => (
|
||||
<div className="jobsTable">
|
||||
{jobs.map((job) => (
|
||||
<div key={job.id} className={`jobsTable__row${!job.enabled ? ' jobsTable__row--inactive' : ''}`}>
|
||||
<div className="jobsTable__row__dot">
|
||||
<span
|
||||
className={`jobsTable__row__dot__indicator${job.enabled ? ' jobsTable__row__dot__indicator--active' : ''}`}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="jobsTable__row__name" title={job.name}>
|
||||
{job.name}
|
||||
</div>
|
||||
|
||||
<div className="jobsTable__row__stat jobsTable__row__stat--blue">
|
||||
<IconHome size="small" />
|
||||
{job.numberOfFoundListings || 0}
|
||||
</div>
|
||||
|
||||
<div className="jobsTable__row__stat jobsTable__row__stat--orange">
|
||||
<IconBriefcase size="small" />
|
||||
{job.provider?.length || 0}
|
||||
</div>
|
||||
|
||||
<div className="jobsTable__row__stat jobsTable__row__stat--purple">
|
||||
<IconBell size="small" />
|
||||
{job.notificationAdapter?.length || 0}
|
||||
</div>
|
||||
|
||||
<div className="jobsTable__row__badges">
|
||||
<Switch
|
||||
size="small"
|
||||
checked={job.enabled}
|
||||
disabled={job.isOnlyShared}
|
||||
onChange={(checked) => onStatusChange(job.id, checked)}
|
||||
/>
|
||||
{job.running && (
|
||||
<Tag color="green" variant="light" size="small">
|
||||
RUNNING
|
||||
</Tag>
|
||||
)}
|
||||
{job.isOnlyShared && (
|
||||
<Tooltip content="Shared with you - read only">
|
||||
<span style={{ display: 'flex', alignItems: 'center' }}>
|
||||
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
|
||||
</span>
|
||||
</Tooltip>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="jobsTable__row__actions">
|
||||
<Tooltip content="Run Job">
|
||||
<Button
|
||||
type="primary"
|
||||
style={{ background: '#21aa21b5' }}
|
||||
size="small"
|
||||
theme="solid"
|
||||
icon={<IconPlayCircle />}
|
||||
disabled={job.isOnlyShared || job.running}
|
||||
onClick={() => onRun(job.id)}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Edit Job">
|
||||
<Button
|
||||
type="secondary"
|
||||
size="small"
|
||||
icon={<IconEdit />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onEdit(job.id)}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Clone Job">
|
||||
<Button
|
||||
type="tertiary"
|
||||
size="small"
|
||||
icon={<IconCopy />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onClone(job.id)}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Delete all found Listings">
|
||||
<Button
|
||||
type="danger"
|
||||
size="small"
|
||||
icon={<IconDescend2 />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onDeleteListings(job.id)}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Delete Job">
|
||||
<Button
|
||||
type="danger"
|
||||
size="small"
|
||||
icon={<IconDelete />}
|
||||
disabled={job.isOnlyShared}
|
||||
onClick={() => onDeleteJob(job.id)}
|
||||
/>
|
||||
</Tooltip>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
|
||||
export default JobsTable;
|
||||
105
ui/src/components/table/JobsTable.less
Normal file
105
ui/src/components/table/JobsTable.less
Normal file
@@ -0,0 +1,105 @@
|
||||
@import '../../tokens.less';
|
||||
|
||||
.jobsTable {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
|
||||
&__row {
|
||||
display: grid;
|
||||
grid-template-columns: 24px 1fr 80px 80px 80px auto auto;
|
||||
align-items: center;
|
||||
gap: @space-3;
|
||||
padding: 8px 12px;
|
||||
background: @color-elevated;
|
||||
border: 1px solid @color-border;
|
||||
border-radius: @radius-chip;
|
||||
transition: background @transition-fast;
|
||||
|
||||
&:hover {
|
||||
background: #252525;
|
||||
}
|
||||
|
||||
&--inactive {
|
||||
opacity: 0.6;
|
||||
}
|
||||
|
||||
&__dot {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
|
||||
&__indicator {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 50%;
|
||||
flex-shrink: 0;
|
||||
background-color: rgba(251, 113, 133, 0.7);
|
||||
|
||||
&--active {
|
||||
background-color: rgba(52, 211, 153, 0.8);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
&__name {
|
||||
font-weight: 600;
|
||||
font-size: @text-sm;
|
||||
color: @color-text;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__stat {
|
||||
font-size: @text-sm;
|
||||
font-weight: 600;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
white-space: nowrap;
|
||||
|
||||
&--blue {
|
||||
color: @color-blue-text;
|
||||
}
|
||||
|
||||
&--orange {
|
||||
color: @color-orange-text;
|
||||
}
|
||||
|
||||
&--purple {
|
||||
color: @color-purple-text;
|
||||
}
|
||||
}
|
||||
|
||||
&__badges {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
&__actions {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 2px;
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
grid-template-columns: 24px 1fr 80px auto auto;
|
||||
|
||||
.jobsTable__row__stat--orange,
|
||||
.jobsTable__row__stat--purple {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 560px) {
|
||||
grid-template-columns: 24px 1fr auto auto;
|
||||
|
||||
.jobsTable__row__stat--blue {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
141
ui/src/components/table/ListingsTable.jsx
Normal file
141
ui/src/components/table/ListingsTable.jsx
Normal file
@@ -0,0 +1,141 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { Button, Tooltip } from '@douyinfe/semi-ui-19';
|
||||
import {
|
||||
IconBriefcase,
|
||||
IconCart,
|
||||
IconDelete,
|
||||
IconLink,
|
||||
IconMapPin,
|
||||
IconStar,
|
||||
IconStarStroked,
|
||||
IconEyeOpened,
|
||||
} from '@douyinfe/semi-icons';
|
||||
import no_image from '../../assets/no_image.png';
|
||||
import * as timeService from '../../services/time/timeService.js';
|
||||
import StatusControl from '../listings/StatusControl.jsx';
|
||||
|
||||
import './ListingsTable.less';
|
||||
|
||||
/**
|
||||
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function, onStatusChange: Function }} props
|
||||
*/
|
||||
const ListingsTable = ({ listings, onWatch, onNavigate, onDelete, onStatusChange }) => (
|
||||
<div className="listingsTable">
|
||||
{listings.map((item) => (
|
||||
<div
|
||||
key={item.id}
|
||||
className={`listingsTable__row${!item.is_active ? ' listingsTable__row--inactive' : ''}`}
|
||||
role="button"
|
||||
tabIndex={0}
|
||||
onClick={() => onNavigate(item.id)}
|
||||
onKeyDown={(e) => {
|
||||
if (e.key === 'Enter' || e.key === ' ') onNavigate(item.id);
|
||||
}}
|
||||
>
|
||||
<div className="listingsTable__row__thumb">
|
||||
<img
|
||||
src={item.image_url || no_image}
|
||||
alt={item.title}
|
||||
onError={(e) => {
|
||||
e.target.src = no_image;
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="listingsTable__row__title" title={item.title}>
|
||||
{item.title}
|
||||
</div>
|
||||
|
||||
<div className="listingsTable__row__price">
|
||||
{item.price ? (
|
||||
<>
|
||||
<IconCart size="small" />
|
||||
{item.price}
|
||||
</>
|
||||
) : (
|
||||
<span className="listingsTable__row__empty">---</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="listingsTable__row__address">
|
||||
{item.address ? (
|
||||
<>
|
||||
<IconMapPin size="small" />
|
||||
{item.address}
|
||||
</>
|
||||
) : (
|
||||
<span className="listingsTable__row__empty">---</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="listingsTable__row__meta">
|
||||
<IconBriefcase size="small" />
|
||||
{item.provider}
|
||||
</div>
|
||||
|
||||
<div className="listingsTable__row__date">{timeService.format(item.created_at, false)}</div>
|
||||
|
||||
<div className="listingsTable__row__actions" onClick={(e) => e.stopPropagation()}>
|
||||
<StatusControl
|
||||
status={item.status?.status ?? null}
|
||||
compact
|
||||
onChange={(next) => onStatusChange?.(item, next)}
|
||||
onTriggerClick={(e) => e.stopPropagation()}
|
||||
/>
|
||||
<Tooltip content={item.isWatched === 1 ? 'Remove from Watchlist' : 'Add to Watchlist'}>
|
||||
<button
|
||||
type="button"
|
||||
className="listingsTable__row__star"
|
||||
onClick={(e) => onWatch(e, item)}
|
||||
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
|
||||
>
|
||||
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
|
||||
</button>
|
||||
</Tooltip>
|
||||
<Tooltip content="Original Listing">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconLink />}
|
||||
style={{ color: '#60a5fa' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
window.open(item.link, '_blank');
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="View in Fredy">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconEyeOpened />}
|
||||
style={{ color: '#34d399' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
onNavigate(item.id);
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
<Tooltip content="Remove">
|
||||
<Button
|
||||
size="small"
|
||||
icon={<IconDelete />}
|
||||
style={{ color: '#fb7185' }}
|
||||
theme="borderless"
|
||||
onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
onDelete(item.id);
|
||||
}}
|
||||
/>
|
||||
</Tooltip>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
|
||||
export default ListingsTable;
|
||||
142
ui/src/components/table/ListingsTable.less
Normal file
142
ui/src/components/table/ListingsTable.less
Normal file
@@ -0,0 +1,142 @@
|
||||
@import '../../tokens.less';
|
||||
|
||||
.listingsTable {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
|
||||
&__row {
|
||||
display: grid;
|
||||
grid-template-columns: 56px 1fr 140px 200px 120px 110px auto;
|
||||
align-items: center;
|
||||
gap: @space-3;
|
||||
padding: 8px 12px;
|
||||
background: @color-elevated;
|
||||
border: 1px solid @color-border;
|
||||
border-radius: @radius-chip;
|
||||
cursor: pointer;
|
||||
transition: background @transition-fast;
|
||||
|
||||
&:hover {
|
||||
background: #252525;
|
||||
}
|
||||
|
||||
&--inactive {
|
||||
opacity: 0.6;
|
||||
}
|
||||
|
||||
&__thumb {
|
||||
width: 56px;
|
||||
height: 40px;
|
||||
flex-shrink: 0;
|
||||
border-radius: @radius-chip;
|
||||
overflow: hidden;
|
||||
|
||||
img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
display: block;
|
||||
}
|
||||
}
|
||||
|
||||
&__title {
|
||||
font-weight: 600;
|
||||
font-size: @text-sm;
|
||||
color: @color-text;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__price {
|
||||
font-size: @text-sm;
|
||||
font-weight: 600;
|
||||
color: @color-success;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__address {
|
||||
font-size: @text-xs;
|
||||
color: @color-muted;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__meta {
|
||||
font-size: @text-xs;
|
||||
color: @color-muted;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__date {
|
||||
font-size: @text-xs;
|
||||
color: @color-faint;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__actions {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 2px;
|
||||
}
|
||||
|
||||
&__star {
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
background: transparent;
|
||||
border: none;
|
||||
border-radius: 50%;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
cursor: pointer;
|
||||
padding: 0;
|
||||
transition: background @transition-fast;
|
||||
flex-shrink: 0;
|
||||
|
||||
&:hover {
|
||||
background: rgba(0, 0, 0, 0.1);
|
||||
}
|
||||
|
||||
svg {
|
||||
color: @color-accent;
|
||||
font-size: 14px;
|
||||
}
|
||||
}
|
||||
|
||||
&__empty {
|
||||
color: @color-faint;
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
grid-template-columns: 56px 1fr 120px auto;
|
||||
|
||||
.listingsTable__row__address,
|
||||
.listingsTable__row__meta,
|
||||
.listingsTable__row__date {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 560px) {
|
||||
grid-template-columns: 56px 1fr auto;
|
||||
|
||||
.listingsTable__row__price {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -260,6 +260,22 @@ export const useFredyState = create(
|
||||
console.error('Error while trying to get resource for api/listings/map. Error:', Exception);
|
||||
}
|
||||
},
|
||||
async setListingStatus(listingId, status) {
|
||||
try {
|
||||
await xhrPost(`/api/listings/${listingId}/status`, { status });
|
||||
} catch (Exception) {
|
||||
console.error(`Error while trying to set status for listing ${listingId}. Error:`, Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
async setListingNotes(listingId, notes) {
|
||||
try {
|
||||
await xhrPost(`/api/listings/${listingId}/notes`, { notes });
|
||||
} catch (Exception) {
|
||||
console.error(`Error while trying to set notes for listing ${listingId}. Error:`, Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
},
|
||||
userSettings: {
|
||||
async getUserSettings() {
|
||||
@@ -321,6 +337,48 @@ export const useFredyState = create(
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
async setListingsViewMode(listings_view_mode) {
|
||||
try {
|
||||
await xhrPost('/api/user/settings/listings-view-mode', { listings_view_mode });
|
||||
set((state) => ({
|
||||
userSettings: {
|
||||
...state.userSettings,
|
||||
settings: { ...state.userSettings.settings, listings_view_mode },
|
||||
},
|
||||
}));
|
||||
} catch (Exception) {
|
||||
console.error('Error while trying to update listings view mode setting. Error:', Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
async setJobsViewMode(jobs_view_mode) {
|
||||
try {
|
||||
await xhrPost('/api/user/settings/jobs-view-mode', { jobs_view_mode });
|
||||
set((state) => ({
|
||||
userSettings: {
|
||||
...state.userSettings,
|
||||
settings: { ...state.userSettings.settings, jobs_view_mode },
|
||||
},
|
||||
}));
|
||||
} catch (Exception) {
|
||||
console.error('Error while trying to update jobs view mode setting. Error:', Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
async setListingDeletionPreference(listing_deletion_preference) {
|
||||
try {
|
||||
await xhrPost('/api/user/settings/listing-deletion-preference', { listing_deletion_preference });
|
||||
set((state) => ({
|
||||
userSettings: {
|
||||
...state.userSettings,
|
||||
settings: { ...state.userSettings.settings, listing_deletion_preference },
|
||||
},
|
||||
}));
|
||||
} catch (Exception) {
|
||||
console.error('Error while trying to update listing deletion preference. Error:', Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
|
||||
@@ -18,6 +18,9 @@ import {
|
||||
AutoComplete,
|
||||
Select,
|
||||
Banner,
|
||||
Radio,
|
||||
RadioGroup,
|
||||
Typography,
|
||||
} from '@douyinfe/semi-ui-19';
|
||||
import { InputNumber } from '@douyinfe/semi-ui-19';
|
||||
import { xhrPost, xhrGet } from '../../services/xhr';
|
||||
@@ -33,6 +36,8 @@ import { debounce } from '../../utils';
|
||||
import Headline from '../../components/headline/Headline.jsx';
|
||||
import './GeneralSettings.less';
|
||||
|
||||
const { Text } = Typography;
|
||||
|
||||
function formatFromTimestamp(ts) {
|
||||
const date = new Date(ts);
|
||||
return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
|
||||
@@ -54,8 +59,10 @@ const GeneralSettings = function GeneralSettings() {
|
||||
const [loading, setLoading] = React.useState(true);
|
||||
|
||||
const settings = useSelector((state) => state.generalSettings.settings);
|
||||
const currentUser = useSelector((state) => state.user.currentUser);
|
||||
|
||||
const [interval, setInterval] = React.useState('');
|
||||
const [proxyUrl, setProxyUrl] = React.useState('');
|
||||
const [port, setPort] = React.useState('');
|
||||
const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
|
||||
const [workingHourTo, setWorkingHourTo] = React.useState(null);
|
||||
@@ -72,9 +79,12 @@ const GeneralSettings = function GeneralSettings() {
|
||||
// User settings state
|
||||
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
|
||||
const providerDetails = useSelector((state) => state.userSettings.settings.provider_details);
|
||||
const listingDeletionPreference = useSelector((state) => state.userSettings.settings.listing_deletion_preference);
|
||||
const allProviders = useSelector((state) => state.provider);
|
||||
const [address, setAddress] = useState(homeAddress?.address || '');
|
||||
const [coords, setCoords] = useState(homeAddress?.coords || null);
|
||||
const [listingDeleteHard, setListingDeleteHard] = useState(false);
|
||||
const [listingDeleteSkipPrompt, setListingDeleteSkipPrompt] = useState(false);
|
||||
const saving = useIsLoading(actions.userSettings.setHomeAddress);
|
||||
const [dataSource, setDataSource] = useState([]);
|
||||
|
||||
@@ -90,6 +100,7 @@ const GeneralSettings = function GeneralSettings() {
|
||||
React.useEffect(() => {
|
||||
async function init() {
|
||||
setInterval(settings?.interval);
|
||||
setProxyUrl(settings?.proxyUrl ?? '');
|
||||
setPort(settings?.port);
|
||||
setWorkingHourFrom(settings?.workingHours?.from);
|
||||
setWorkingHourTo(settings?.workingHours?.to);
|
||||
@@ -107,6 +118,11 @@ const GeneralSettings = function GeneralSettings() {
|
||||
setCoords(homeAddress?.coords || null);
|
||||
}, [homeAddress]);
|
||||
|
||||
useEffect(() => {
|
||||
setListingDeleteHard(listingDeletionPreference?.hardDelete ?? false);
|
||||
setListingDeleteSkipPrompt(listingDeletionPreference?.skipPrompt ?? false);
|
||||
}, [listingDeletionPreference]);
|
||||
|
||||
const nullOrEmpty = (val) => val == null || val.length === 0;
|
||||
|
||||
const handleStore = async () => {
|
||||
@@ -132,6 +148,7 @@ const GeneralSettings = function GeneralSettings() {
|
||||
try {
|
||||
await xhrPost('/api/admin/generalSettings', {
|
||||
interval,
|
||||
proxyUrl: proxyUrl?.trim() ?? '',
|
||||
port,
|
||||
workingHours: {
|
||||
from: workingHourFrom,
|
||||
@@ -214,6 +231,10 @@ const GeneralSettings = function GeneralSettings() {
|
||||
try {
|
||||
const responseJson = await actions.userSettings.setHomeAddress(address);
|
||||
setCoords(responseJson.coords);
|
||||
await actions.userSettings.setListingDeletionPreference({
|
||||
skipPrompt: listingDeleteSkipPrompt,
|
||||
hardDelete: listingDeleteHard,
|
||||
});
|
||||
await actions.userSettings.getUserSettings();
|
||||
Toast.success('Settings saved. Distance calculations are running in the background.');
|
||||
} catch (error) {
|
||||
@@ -375,6 +396,18 @@ const GeneralSettings = function GeneralSettings() {
|
||||
</div>
|
||||
</SegmentPart>
|
||||
|
||||
<SegmentPart
|
||||
name="Proxy URL"
|
||||
helpText="Optional. Routes the scraping browser through a proxy. Server/datacenter IPs are frequently blocked by providers (e.g. immowelt) regardless of browser fingerprint, a German residential proxy makes requests look like a normal household and is the most effective fix. Format: http://user:pass@host:port or socks5://user:pass@host:port. Leave empty to disable."
|
||||
>
|
||||
<Input
|
||||
type="text"
|
||||
placeholder="http://user:pass@host:port"
|
||||
value={proxyUrl}
|
||||
onChange={(value) => setProxyUrl(value)}
|
||||
/>
|
||||
</SegmentPart>
|
||||
|
||||
<div className="generalSettings__save-row">
|
||||
<Button type="primary" theme="solid" onClick={handleStore} icon={<IconSave />}>
|
||||
Save
|
||||
@@ -443,6 +476,48 @@ const GeneralSettings = function GeneralSettings() {
|
||||
/>
|
||||
</SegmentPart>
|
||||
|
||||
<SegmentPart
|
||||
name="Listing deletion"
|
||||
helpText="Choose the default deletion mode. Soft delete hides them without re-scraping; hard delete removes them from the database."
|
||||
>
|
||||
<RadioGroup
|
||||
value={listingDeleteHard ? 'hard' : 'soft'}
|
||||
onChange={(e) => setListingDeleteHard(e.target.value === 'hard')}
|
||||
>
|
||||
<Radio value="soft">
|
||||
<div>
|
||||
<Text strong>Mark as deleted (Soft Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during
|
||||
the next scraping session.
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
<Radio value="hard">
|
||||
<div>
|
||||
<Text strong>Remove from database (Hard Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are completely removed from the database.
|
||||
<br />
|
||||
<Text type="warning">
|
||||
Consequence: They might re-appear when scraping the next time because Fredy won't know they
|
||||
were previously found.
|
||||
</Text>
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
</RadioGroup>
|
||||
<Checkbox
|
||||
checked={listingDeleteSkipPrompt}
|
||||
onChange={(e) => setListingDeleteSkipPrompt(e.target.checked)}
|
||||
style={{ marginTop: 12 }}
|
||||
>
|
||||
Skip confirmation dialog
|
||||
</Checkbox>
|
||||
</SegmentPart>
|
||||
|
||||
<div className="generalSettings__save-row">
|
||||
<Button
|
||||
icon={<IconSave />}
|
||||
@@ -467,12 +542,26 @@ const GeneralSettings = function GeneralSettings() {
|
||||
itemKey="backup"
|
||||
>
|
||||
<div className="generalSettings__tab-content">
|
||||
{demoMode && !currentUser?.isAdmin && (
|
||||
<Banner
|
||||
fullMode={false}
|
||||
type="warning"
|
||||
closeIcon={null}
|
||||
style={{ marginBottom: '12px' }}
|
||||
description="Backup and restore are not available in demo mode."
|
||||
/>
|
||||
)}
|
||||
<SegmentPart
|
||||
name="Backup & Restore"
|
||||
helpText="Download a zipped backup of your database or restore from a backup zip."
|
||||
>
|
||||
<div style={{ display: 'flex', gap: '0.5rem', alignItems: 'center', flexWrap: 'wrap' }}>
|
||||
<Button theme="solid" icon={<IconSave />} onClick={handleDownloadBackup}>
|
||||
<Button
|
||||
theme="solid"
|
||||
icon={<IconSave />}
|
||||
onClick={handleDownloadBackup}
|
||||
disabled={demoMode && !currentUser?.isAdmin}
|
||||
>
|
||||
Download Backup
|
||||
</Button>
|
||||
<input
|
||||
@@ -482,7 +571,12 @@ const GeneralSettings = function GeneralSettings() {
|
||||
style={{ display: 'none' }}
|
||||
onChange={handleSelectRestoreFile}
|
||||
/>
|
||||
<Button onClick={handleOpenFilePicker} theme="light" icon={<IconFolder />}>
|
||||
<Button
|
||||
onClick={handleOpenFilePicker}
|
||||
theme="light"
|
||||
icon={<IconFolder />}
|
||||
disabled={demoMode && !currentUser?.isAdmin}
|
||||
>
|
||||
Restore from Zip
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
@@ -73,7 +73,9 @@ export default function NotificationAdapterMutator({
|
||||
const adapter = useSelector((state) => state.notificationAdapter);
|
||||
|
||||
const preFilledSelectedAdapter =
|
||||
editNotificationAdapter == null ? null : adapter.find((a) => a.id === editNotificationAdapter.id);
|
||||
editNotificationAdapter == null
|
||||
? null
|
||||
: adapter.filter((a) => a != null).find((a) => a.id === editNotificationAdapter.id);
|
||||
|
||||
spreadPrefilledAdapterWithValues(preFilledSelectedAdapter, editNotificationAdapter?.fields);
|
||||
|
||||
@@ -227,9 +229,9 @@ export default function NotificationAdapterMutator({
|
||||
className="providerMutator__fields"
|
||||
value={selectedAdapter == null ? '' : selectedAdapter.id}
|
||||
optionList={adapter
|
||||
.filter((a) => a != null)
|
||||
.map((a) => {
|
||||
return {
|
||||
otherKey: a.id,
|
||||
value: a.id,
|
||||
label: a.name,
|
||||
};
|
||||
@@ -238,7 +240,7 @@ export default function NotificationAdapterMutator({
|
||||
.filter((option) =>
|
||||
editNotificationAdapter != null
|
||||
? true
|
||||
: selected.find((selectedOption) => selectedOption.id === option.key) == null,
|
||||
: selected.find((selectedOption) => selectedOption.id === option.value) == null,
|
||||
)
|
||||
.sort(sortAdapter)}
|
||||
onChange={(value) => {
|
||||
|
||||
@@ -141,21 +141,6 @@ export default function ProviderMutator({
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
<Banner
|
||||
fullMode={false}
|
||||
type="warning"
|
||||
closeIcon={null}
|
||||
title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Warning</div>}
|
||||
style={{ marginBottom: '1rem' }}
|
||||
description={
|
||||
<div>
|
||||
<p>
|
||||
Currently, our Immoscout implementation does not support drawing shapes on a map. Use a radius instead.
|
||||
</p>
|
||||
</div>
|
||||
}
|
||||
/>
|
||||
|
||||
<Select
|
||||
filter
|
||||
placeholder="Select a provider"
|
||||
|
||||
@@ -20,6 +20,8 @@ import {
|
||||
Banner,
|
||||
Spin,
|
||||
Toast,
|
||||
TextArea,
|
||||
Tooltip,
|
||||
} from '@douyinfe/semi-ui-19';
|
||||
import {
|
||||
IconArrowLeft,
|
||||
@@ -31,6 +33,7 @@ import {
|
||||
IconLink,
|
||||
IconStar,
|
||||
IconStarStroked,
|
||||
IconDelete,
|
||||
IconExpand,
|
||||
IconGridView,
|
||||
} from '@douyinfe/semi-icons';
|
||||
@@ -39,9 +42,11 @@ import 'maplibre-gl/dist/maplibre-gl.css';
|
||||
import no_image from '../../assets/no_image.png';
|
||||
import * as timeService from '../../services/time/timeService.js';
|
||||
import { distanceMeters, getBoundsFromCoords } from './mapUtils.js';
|
||||
import { xhrPost } from '../../services/xhr.js';
|
||||
import { xhrPost, xhrDelete } from '../../services/xhr.js';
|
||||
import ListingDeletionModal from '../../components/ListingDeletionModal.jsx';
|
||||
|
||||
import Headline from '../../components/headline/Headline.jsx';
|
||||
import StatusControl from '../../components/listings/StatusControl.jsx';
|
||||
import './ListingDetail.less';
|
||||
|
||||
const { Title, Text } = Typography;
|
||||
@@ -55,10 +60,16 @@ export default function ListingDetail() {
|
||||
const navigate = useNavigate();
|
||||
const actions = useActions();
|
||||
const listing = useSelector((state) => state.listingsData.currentListing);
|
||||
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
|
||||
const userSettings = useSelector((state) => state.userSettings.settings);
|
||||
const homeAddress = userSettings?.home_address;
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
const mapContainer = useRef(null);
|
||||
const map = useRef(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
|
||||
const [notesDraft, setNotesDraft] = useState('');
|
||||
const [notesSaving, setNotesSaving] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
async function fetchListing() {
|
||||
@@ -76,6 +87,10 @@ export default function ListingDetail() {
|
||||
fetchListing();
|
||||
}, [listingId]);
|
||||
|
||||
useEffect(() => {
|
||||
setNotesDraft(listing?.notes ?? '');
|
||||
}, [listing?.id, listing?.notes]);
|
||||
|
||||
const hasGeo =
|
||||
listing?.latitude != null && listing?.longitude != null && listing?.latitude !== -1 && listing?.longitude !== -1;
|
||||
|
||||
@@ -239,6 +254,21 @@ export default function ListingDetail() {
|
||||
};
|
||||
}, [listing, loading, homeAddress]);
|
||||
|
||||
const confirmDeletion = async (hardDelete, remember) => {
|
||||
try {
|
||||
if (remember) {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
await xhrDelete('/api/listings/', { ids: [listing.id], hardDelete });
|
||||
Toast.success('Listing successfully removed');
|
||||
navigate('/listings');
|
||||
} catch (e) {
|
||||
Toast.error(e.message || 'Error deleting listing');
|
||||
} finally {
|
||||
setDeleteModalVisible(false);
|
||||
}
|
||||
};
|
||||
|
||||
const handleWatch = async () => {
|
||||
try {
|
||||
await xhrPost('/api/listings/watch', { listingId: listing.id });
|
||||
@@ -250,6 +280,32 @@ export default function ListingDetail() {
|
||||
}
|
||||
};
|
||||
|
||||
const handleStatusChange = async (next) => {
|
||||
try {
|
||||
await actions.listingsData.setListingStatus(listing.id, next);
|
||||
await actions.listingsData.getListing(listingId);
|
||||
Toast.success(next ? `Marked as ${next}` : 'Status cleared');
|
||||
} catch (e) {
|
||||
console.error('Failed to update status:', e);
|
||||
Toast.error('Failed to update status');
|
||||
}
|
||||
};
|
||||
|
||||
const handleSaveNotes = async () => {
|
||||
if (!listing) return;
|
||||
setNotesSaving(true);
|
||||
try {
|
||||
await actions.listingsData.setListingNotes(listing.id, notesDraft);
|
||||
await actions.listingsData.getListing(listingId);
|
||||
Toast.success('Notes saved');
|
||||
} catch (e) {
|
||||
console.error('Failed to save notes:', e);
|
||||
Toast.error('Failed to save notes');
|
||||
} finally {
|
||||
setNotesSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
if (loading) {
|
||||
return (
|
||||
<div style={{ display: 'flex', justifyContent: 'center', alignItems: 'center', height: '100%' }}>
|
||||
@@ -260,35 +316,58 @@ export default function ListingDetail() {
|
||||
|
||||
if (!listing) return null;
|
||||
|
||||
const statusLabel = listing.status?.status
|
||||
? listing.status.status.charAt(0).toUpperCase() + listing.status.status.slice(1)
|
||||
: null;
|
||||
|
||||
const data = [
|
||||
{ key: 'Price', value: `${listing.price} €`, Icon: <IconCart /> },
|
||||
{
|
||||
key: 'Price',
|
||||
value: `${listing.price} €`,
|
||||
Icon: <IconCart />,
|
||||
helpText: 'The asking price of this listing, as reported by the provider.',
|
||||
},
|
||||
{
|
||||
key: 'Size',
|
||||
value: listing.size ? `${listing.size} m²` : 'N/A',
|
||||
Icon: <IconExpand />,
|
||||
helpText: 'Living space of the listing in square meters.',
|
||||
},
|
||||
{
|
||||
key: 'Rooms',
|
||||
value: listing.rooms ? `${listing.rooms} Rooms` : 'N/A',
|
||||
Icon: <IconGridView />,
|
||||
helpText: 'Number of rooms in the listing.',
|
||||
},
|
||||
{
|
||||
key: 'Job',
|
||||
value: listing.job_name,
|
||||
Icon: <IconBriefcase />,
|
||||
helpText: 'The Fredy job that found this listing.',
|
||||
},
|
||||
{
|
||||
key: 'Provider',
|
||||
value: listing.provider ? listing.provider.charAt(0).toUpperCase() + listing.provider.slice(1) : 'Unknown',
|
||||
Icon: <IconBriefcase />,
|
||||
helpText: 'The real estate portal where this listing was scraped from.',
|
||||
},
|
||||
{
|
||||
key: 'Added',
|
||||
value: timeService.format(listing.created_at),
|
||||
Icon: <IconClock />,
|
||||
helpText: 'When Fredy first added this listing to your database.',
|
||||
},
|
||||
];
|
||||
|
||||
if (statusLabel) {
|
||||
data.push({
|
||||
key: 'Status',
|
||||
value: listing.status?.setAt ? `${statusLabel} (set ${timeService.format(listing.status.setAt)})` : statusLabel,
|
||||
Icon: <IconActivity />,
|
||||
helpText: 'The status you marked for this listing and when you set it.',
|
||||
});
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="listing-detail">
|
||||
<Headline
|
||||
@@ -326,10 +405,25 @@ export default function ListingDetail() {
|
||||
>
|
||||
{listing.isWatched === 1 ? 'Watched' : 'Watch'}
|
||||
</Button>
|
||||
<StatusControl status={listing.status?.status ?? null} onChange={handleStatusChange} />
|
||||
<a href={listing.link} target="_blank" rel="noopener noreferrer" className="listing-detail__open-btn">
|
||||
<IconLink style={{ marginRight: 6 }} />
|
||||
Open listing
|
||||
</a>
|
||||
<Button
|
||||
icon={<IconDelete />}
|
||||
onClick={() => {
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmDeletion(listingDeletionPref.hardDelete);
|
||||
return;
|
||||
}
|
||||
setDeleteModalVisible(true);
|
||||
}}
|
||||
theme="light"
|
||||
type="danger"
|
||||
>
|
||||
Delete
|
||||
</Button>
|
||||
</Space>
|
||||
</div>
|
||||
|
||||
@@ -345,6 +439,32 @@ export default function ListingDetail() {
|
||||
preview={!!listing.image_url}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="listing-detail__notes">
|
||||
<Title heading={4} className="listing-detail__notes-title">
|
||||
Notes
|
||||
</Title>
|
||||
<TextArea
|
||||
value={notesDraft}
|
||||
onChange={(val) => setNotesDraft(val)}
|
||||
placeholder="Your private notes about this listing…"
|
||||
rows={5}
|
||||
autosize={{ minRows: 4, maxRows: 12 }}
|
||||
className="listing-detail__notes-textarea"
|
||||
showClear
|
||||
/>
|
||||
<Space className="listing-detail__notes-actions">
|
||||
<Button
|
||||
theme="solid"
|
||||
type="primary"
|
||||
loading={notesSaving}
|
||||
disabled={notesSaving || (notesDraft ?? '') === (listing.notes ?? '')}
|
||||
onClick={handleSaveNotes}
|
||||
>
|
||||
Store notes
|
||||
</Button>
|
||||
</Space>
|
||||
</div>
|
||||
</Col>
|
||||
<Col span={24} lg={12}>
|
||||
<div className="listing-detail__info-section">
|
||||
@@ -354,10 +474,12 @@ export default function ListingDetail() {
|
||||
<Descriptions column={1}>
|
||||
{data.map((item, index) => (
|
||||
<Descriptions.Item key={index}>
|
||||
<Space>
|
||||
{item.Icon}
|
||||
{item.value}
|
||||
</Space>
|
||||
<Tooltip content={item.helpText} position="left">
|
||||
<span className="listing-detail__details-item">
|
||||
{item.Icon}
|
||||
{item.value}
|
||||
</span>
|
||||
</Tooltip>
|
||||
</Descriptions.Item>
|
||||
))}
|
||||
</Descriptions>
|
||||
@@ -397,6 +519,13 @@ export default function ListingDetail() {
|
||||
<div ref={mapContainer} className="listing-detail__map-container" />
|
||||
)}
|
||||
</div>
|
||||
|
||||
<ListingDeletionModal
|
||||
visible={deleteModalVisible}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
onConfirm={confirmDeletion}
|
||||
onCancel={() => setDeleteModalVisible(false)}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -89,6 +89,49 @@
|
||||
}
|
||||
}
|
||||
|
||||
&__notes {
|
||||
padding: 1.5rem;
|
||||
border-top: 1px solid var(--semi-color-border);
|
||||
}
|
||||
|
||||
&__notes-title {
|
||||
margin-bottom: 1rem !important;
|
||||
}
|
||||
|
||||
&__notes-actions {
|
||||
margin-top: 0.75rem;
|
||||
width: 100%;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
&__notes-textarea {
|
||||
background: #2a2a2a !important;
|
||||
border: 1px solid @color-border-bright !important;
|
||||
border-radius: @radius-input !important;
|
||||
box-shadow: none !important;
|
||||
outline: none !important;
|
||||
transition: border-color @transition-fast, background @transition-fast !important;
|
||||
|
||||
textarea {
|
||||
background: transparent !important;
|
||||
border: 0 !important;
|
||||
box-shadow: none !important;
|
||||
outline: none !important;
|
||||
color: @color-text !important;
|
||||
font-family: @font-ui !important;
|
||||
font-size: @text-base !important;
|
||||
}
|
||||
|
||||
&.semi-input-textarea-focus,
|
||||
&:focus,
|
||||
&:focus-within {
|
||||
border-color: @color-accent !important;
|
||||
box-shadow: none !important;
|
||||
outline: none !important;
|
||||
background: #2f2f2f !important;
|
||||
}
|
||||
}
|
||||
|
||||
&__watch-btn {
|
||||
color: @color-muted !important;
|
||||
border: 1px solid @color-border-bright !important;
|
||||
@@ -128,6 +171,13 @@
|
||||
padding: 1.5rem;
|
||||
}
|
||||
|
||||
&__details-item {
|
||||
cursor: help;
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
&__map-container {
|
||||
height: 400px;
|
||||
width: 100%;
|
||||
|
||||
@@ -3,14 +3,18 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import ListingsGrid from '../../components/grid/listings/ListingsGrid.jsx';
|
||||
import ListingsOverview from '../../components/listings/ListingsOverview.jsx';
|
||||
import Headline from '../../components/headline/Headline.jsx';
|
||||
|
||||
export default function Listings() {
|
||||
/**
|
||||
* @param {{ mode?: 'all' | 'watchlist' }} props
|
||||
*/
|
||||
export default function Listings({ mode = 'all' }) {
|
||||
const title = mode === 'watchlist' ? 'Watchlist' : 'Listings';
|
||||
return (
|
||||
<>
|
||||
<Headline text="Listings" />
|
||||
<ListingsGrid />
|
||||
<Headline text={title} />
|
||||
<ListingsOverview mode={mode} />
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -37,7 +37,10 @@ export default function MapView() {
|
||||
const sp = useSearchParams();
|
||||
const [searchParams, setSearchParams] = sp;
|
||||
const listings = useSelector((state) => state.listingsData.mapListings);
|
||||
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
|
||||
const userSettings = useSelector((state) => state.userSettings.settings);
|
||||
const homeAddress = userSettings?.home_address;
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
|
||||
const jobs = useSelector((state) => state.jobsData.jobs);
|
||||
const [jobId, setJobId] = useSearchParamState(sp, 'job', null, parseString);
|
||||
@@ -52,10 +55,14 @@ export default function MapView() {
|
||||
|
||||
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
|
||||
const [listingToDelete, setListingToDelete] = useState(null);
|
||||
const deleteListingRef = useRef(null);
|
||||
|
||||
const confirmListingDeletion = async (hardDelete) => {
|
||||
const confirmListingDeletion = async (hardDelete, remember, id = listingToDelete) => {
|
||||
try {
|
||||
await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
|
||||
if (remember) {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
await xhrDelete('/api/listings/', { ids: [id], hardDelete });
|
||||
Toast.success('Listing successfully removed');
|
||||
fetchListings();
|
||||
} catch (error) {
|
||||
@@ -66,6 +73,15 @@ export default function MapView() {
|
||||
}
|
||||
};
|
||||
|
||||
deleteListingRef.current = (id) => {
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmListingDeletion(listingDeletionPref.hardDelete, false, id);
|
||||
return;
|
||||
}
|
||||
setListingToDelete(id);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
// Only reset to full range when no URL override is set
|
||||
if (urlPriceMax === null) {
|
||||
@@ -88,10 +104,7 @@ export default function MapView() {
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
window.deleteListing = (id) => {
|
||||
setListingToDelete(id);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
window.deleteListing = (id) => deleteListingRef.current(id);
|
||||
|
||||
window.viewDetails = (id) => {
|
||||
navigate(`/listings/listing/${id}`);
|
||||
@@ -472,6 +485,7 @@ export default function MapView() {
|
||||
|
||||
<ListingDeletionModal
|
||||
visible={deleteModalVisible}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
onConfirm={confirmListingDeletion}
|
||||
onCancel={() => {
|
||||
setDeleteModalVisible(false);
|
||||
|
||||
@@ -37,7 +37,7 @@ const Users = function Users() {
|
||||
await actions.jobsData.getJobs();
|
||||
await actions.user.getUsers();
|
||||
} catch (error) {
|
||||
Toast.error(error);
|
||||
Toast.error(error.error);
|
||||
setUserIdToBeRemoved(null);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -10,6 +10,7 @@ export default defineConfig({
|
||||
globals: true,
|
||||
environment: 'node',
|
||||
include: ['test/**/*.test.js'],
|
||||
globalSetup: ['./test/globalSetup.js'],
|
||||
testTimeout: 60000,
|
||||
reporters: ['verbose'],
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user