mirror of
https://github.com/orangecoding/fredy.git
synced 2026-06-16 12:31:07 +00:00
Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
44edf47393 | ||
|
|
5ceac25aa6 | ||
|
|
34b68e1f52 | ||
|
|
6428e7ad78 | ||
|
|
2bcec04d55 | ||
|
|
ee2112a24d | ||
|
|
5a54448288 | ||
|
|
f1b8709ab7 | ||
|
|
b56e13aa16 | ||
|
|
a834abc31c | ||
|
|
573868eccb | ||
|
|
1a210d7c1c | ||
|
|
996b841cfb | ||
|
|
b2e294e38c | ||
|
|
8afeaa05d9 | ||
|
|
ec47137b89 | ||
|
|
33161de087 | ||
|
|
acab23207e | ||
|
|
2896d531e4 | ||
|
|
0cbfa25062 | ||
|
|
bcd3042026 | ||
|
|
0ce93acaf6 |
36
README.md
36
README.md
@@ -23,7 +23,7 @@
|
||||
|
||||
|
||||
|
||||
# Fredy 🏡 – Your Self-Hosted Real Estate Finder for Germany
|
||||
# Fredy 🏡 - Your Self-Hosted Real Estate Finder for Germany
|
||||
|
||||
Finding an apartment or house in Germany can be stressful and
|
||||
time-consuming.\
|
||||
@@ -167,6 +167,40 @@ For more information on how to set it up and use it, please refer to the [MCP Re
|
||||
|
||||
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
|
||||
|
||||
## 🛡️ Bot Detection & Proxies
|
||||
|
||||
Most browser-based providers (immowelt, immonet, kleinanzeigen, ...) are scraped through a hardened headless browser ([CloakBrowser](https://www.npmjs.com/package/cloakbrowser)). It makes the **browser fingerprint** indistinguishable from a real Chrome, which is enough when you run Fredy on a normal home connection.
|
||||
|
||||
On a **server / VPS the requests usually originate from a datacenter IP**, and providers behind anti-bot systems (e.g. AWS CloudFront/WAF) block those based on **IP reputation alone**, no matter how perfect the fingerprint is. The typical symptom: it works locally but you get `We have been detected as a bot :-/` on the server.
|
||||
|
||||
### The fix: a residential proxy
|
||||
|
||||
A **residential proxy** routes Fredy's browser through the internet connection of a real household, so the provider sees a "normal user" IP instead of a datacenter. For German portals, use a **German (DE) residential** (or mobile/4G) proxy. Plain VPNs and **datacenter proxies do not help** here, they share the same bad reputation as your server.
|
||||
|
||||
**Configure it** under **Settings → Execution → Proxy URL**. Supported formats:
|
||||
|
||||
```
|
||||
http://user:pass@host:port
|
||||
socks5://user:pass@host:port
|
||||
```
|
||||
|
||||
Leave the field empty to disable. The proxy applies to all headless-browser providers and takes effect on the next job run (no restart needed). Immoscout uses a separate mobile API and is not affected.
|
||||
|
||||
### Where to get a residential proxy
|
||||
|
||||
Residential proxies are a paid service (usually billed per GB, Fredy's traffic is small). Well-known providers offering German residential IPs include:
|
||||
|
||||
| Provider | Notes |
|
||||
|---|---|
|
||||
| [IPRoyal](https://iproyal.com) | Pay-as-you-go, no monthly minimum, good for low volume |
|
||||
| [Webshare](https://www.webshare.io) | Cheap entry tier, has a small free plan to test with |
|
||||
| [Decodo (formerly Smartproxy)](https://decodo.com) | Easy setup, country/city targeting |
|
||||
| [SOAX](https://soax.com) | Residential + mobile, fine-grained geo-targeting |
|
||||
| [Bright Data](https://brightdata.com) | Largest pool, most features, higher complexity/price |
|
||||
| [Oxylabs](https://oxylabs.io) | Enterprise-grade, larger plans |
|
||||
|
||||
This is not an endorsement, pick whatever fits your budget. For low-volume use like Fredy, a pay-as-you-go plan (e.g. IPRoyal) or a cheap entry tier (e.g. Webshare) is usually plenty. Make sure to select **Germany** as the proxy location and keep the search interval reasonable (the higher the interval, the less you look like a bot).
|
||||
|
||||
## Analytics
|
||||
|
||||
Fredy is completely free (and will always remain free). However, it would be a huge help if you’d allow me to collect some analytical data.
|
||||
|
||||
@@ -11,6 +11,8 @@
|
||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||
|
||||
<title>Fredy || Real Estate Finder</title>
|
||||
<link rel="icon" type="image/png" href="/ui/src/assets/heart.png" />
|
||||
<link rel="apple-touch-icon" href="/ui/src/assets/heart.png" />
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link href="https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
|
||||
|
||||
@@ -5,9 +5,10 @@
|
||||
|
||||
import { NoNewListingsWarning } from './errors.js';
|
||||
import {
|
||||
storeListings,
|
||||
getKnownListingHashesForJobAndProvider,
|
||||
deleteListingsById,
|
||||
getKnownListingHashesForJobAndProvider,
|
||||
storeListings,
|
||||
updateListingDistance,
|
||||
} from './services/storage/listingsStorage.js';
|
||||
import { getJob } from './services/storage/jobStorage.js';
|
||||
import * as notify from './notification/notify.js';
|
||||
@@ -16,8 +17,7 @@ import urlModifier from './services/queryStringMutator.js';
|
||||
import logger from './services/logger.js';
|
||||
import { geocodeAddress } from './services/geocoding/geoCodingService.js';
|
||||
import { distanceMeters } from './services/listings/distanceCalculator.js';
|
||||
import { getUserSettings, getSettings } from './services/storage/settingsStorage.js';
|
||||
import { updateListingDistance } from './services/storage/listingsStorage.js';
|
||||
import { getSettings, getUserSettings } from './services/storage/settingsStorage.js';
|
||||
import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
|
||||
import { formatListing } from './utils/formatListing.js';
|
||||
|
||||
@@ -97,9 +97,9 @@ class FredyPipelineExecutioner {
|
||||
}
|
||||
|
||||
/**
|
||||
* Optionally enrich new listings with data from their detail pages.
|
||||
* Optionally, enrich new listings with data from their detail pages.
|
||||
* Only called when the provider config defines a `fetchDetails` function.
|
||||
* Runs all fetches in parallel. Each individual fetch must handle its own errors
|
||||
* Runs all fetches in parallel. Each fetch must handle its own errors
|
||||
* and always resolve (never reject) to avoid aborting other listings.
|
||||
*
|
||||
* @param {Listing[]} newListings New listings to enrich.
|
||||
@@ -132,7 +132,7 @@ class FredyPipelineExecutioner {
|
||||
for (const listing of newListings) {
|
||||
if (listing.address) {
|
||||
const coords = await geocodeAddress(listing.address);
|
||||
if (coords) {
|
||||
if (coords && coords.lat !== -1 && coords.lng !== -1) {
|
||||
listing.latitude = coords.lat;
|
||||
listing.longitude = coords.lng;
|
||||
}
|
||||
@@ -227,7 +227,7 @@ class FredyPipelineExecutioner {
|
||||
const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
|
||||
return new Promise((resolve, reject) => {
|
||||
extractor
|
||||
.execute(url, this._providerConfig.waitForSelector)
|
||||
.execute(url, this._providerConfig.waitForSelector, this._providerId)
|
||||
.then(() => {
|
||||
const listings = extractor.parseResponseText(
|
||||
this._providerConfig.crawlContainer,
|
||||
@@ -264,15 +264,15 @@ class FredyPipelineExecutioner {
|
||||
const requiredKeys = this._providerConfig.requiredFieldNames;
|
||||
const requireValues = ['id', 'link', 'title'];
|
||||
|
||||
const filteredListings = listings
|
||||
// this should never filter some listings out, because the normalize function should always extract all fields.
|
||||
.filter((item) => requiredKeys.every((key) => key in item))
|
||||
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
|
||||
.filter(this._providerConfig.filter)
|
||||
// filter out listings that are missing required fields
|
||||
.filter((item) => requireValues.every((key) => item[key] != null));
|
||||
|
||||
return filteredListings;
|
||||
return (
|
||||
listings
|
||||
// this should never filter some listings out, because the normalize function should always extract all fields.
|
||||
.filter((item) => requiredKeys.every((key) => key in item))
|
||||
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
|
||||
.filter(this._providerConfig.filter)
|
||||
// filter out listings that are missing required fields
|
||||
.filter((item) => requireValues.every((key) => item[key] != null))
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -10,5 +10,8 @@ export const TRACKING_POIS = {
|
||||
JOBS_TABLE_VIEW: 'JOBS_TABLE_VIEW',
|
||||
LISTING_TABLE_VIEW: 'LISTING_TABLE_VIEW',
|
||||
BASE_URL_SETTING: 'BASE_URL_SETTING',
|
||||
SET_PROXY_SETTING: 'SET_PROXY_SETTING',
|
||||
DETECTED_AS_BOT: 'DETECTED_AS_BOT',
|
||||
NOTES_CREATE: 'NOTES_CREATE',
|
||||
USING_LISTING_STATUS: 'USING_LISTING_STATUS',
|
||||
};
|
||||
|
||||
@@ -76,13 +76,13 @@ fastify.register(async (app) => {
|
||||
app.register(dashboardPlugin, { prefix: '/api/dashboard' });
|
||||
app.register(userSettingsPlugin, { prefix: '/api/user/settings' });
|
||||
app.register(trackingPlugin, { prefix: '/api/tracking' });
|
||||
app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
|
||||
});
|
||||
|
||||
// Admin-only routes
|
||||
fastify.register(async (app) => {
|
||||
app.addHook('preHandler', authHook);
|
||||
app.addHook('preHandler', adminHook);
|
||||
app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
|
||||
app.register(backupPlugin, { prefix: '/api/admin/backup' });
|
||||
app.register(userPlugin, { prefix: '/api/admin/users' });
|
||||
});
|
||||
|
||||
@@ -27,8 +27,11 @@ export default async function generalSettingsPlugin(fastify) {
|
||||
}
|
||||
const localSettings = await getSettings();
|
||||
|
||||
if (localSettings.demoMode && !isAdmin(request)) {
|
||||
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change these settings.' });
|
||||
if (!isAdmin(request)) {
|
||||
const reason = localSettings.demoMode
|
||||
? 'In demo mode, it is not allowed to change these settings.'
|
||||
: 'Only admins can change these settings.';
|
||||
return reply.code(403).send({ error: reason });
|
||||
}
|
||||
|
||||
try {
|
||||
@@ -41,6 +44,9 @@ export default async function generalSettingsPlugin(fastify) {
|
||||
if (appSettings.baseUrl != null) {
|
||||
await trackPoi(TRACKING_POIS.BASE_URL_SETTING);
|
||||
}
|
||||
if (appSettings.proxyUrl != null) {
|
||||
await trackPoi(TRACKING_POIS.SET_PROXY_SETTING);
|
||||
}
|
||||
} catch (err) {
|
||||
logger.error(err);
|
||||
return reply.code(500).send({ error: 'Error while trying to write settings.' });
|
||||
|
||||
@@ -10,6 +10,8 @@ import logger from '../../services/logger.js';
|
||||
import { nullOrEmpty } from '../../utils.js';
|
||||
import { getJobs } from '../../services/storage/jobStorage.js';
|
||||
import { getSettings } from '../../services/storage/settingsStorage.js';
|
||||
import { trackPoi } from '../../services/tracking/Tracker.js';
|
||||
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
|
||||
|
||||
/**
|
||||
* @param {import('fastify').FastifyInstance} fastify
|
||||
@@ -23,6 +25,7 @@ export default async function listingsPlugin(fastify) {
|
||||
jobNameFilter,
|
||||
providerFilter,
|
||||
watchListFilter,
|
||||
statusFilter,
|
||||
sortfield = null,
|
||||
sortdir = 'asc',
|
||||
freeTextFilter,
|
||||
@@ -35,6 +38,11 @@ export default async function listingsPlugin(fastify) {
|
||||
};
|
||||
const normalizedActivity = toBool(activityFilter);
|
||||
const normalizedWatch = toBool(watchListFilter);
|
||||
const allowedStatuses = ['applied', 'rejected', 'accepted', 'none'];
|
||||
const normalizedStatus =
|
||||
typeof statusFilter === 'string' && allowedStatuses.includes(statusFilter.toLowerCase())
|
||||
? statusFilter.toLowerCase()
|
||||
: undefined;
|
||||
|
||||
let jobFilter = null;
|
||||
let jobIdFilter = null;
|
||||
@@ -54,6 +62,7 @@ export default async function listingsPlugin(fastify) {
|
||||
jobIdFilter: jobIdFilter,
|
||||
providerFilter,
|
||||
watchListFilter: normalizedWatch,
|
||||
statusFilter: normalizedStatus,
|
||||
sortField: sortfield || null,
|
||||
sortDir: sortdir === 'desc' ? 'desc' : 'asc',
|
||||
userId: request.session.currentUser,
|
||||
@@ -94,6 +103,55 @@ export default async function listingsPlugin(fastify) {
|
||||
return reply.send();
|
||||
});
|
||||
|
||||
fastify.post('/:listingId/notes', async (request, reply) => {
|
||||
const { listingId } = request.params || {};
|
||||
const { notes } = request.body || {};
|
||||
const userId = request.session?.currentUser;
|
||||
if (!listingId || !userId) {
|
||||
return reply.code(400).send({ message: 'listingId or user not provided' });
|
||||
}
|
||||
try {
|
||||
const changes = listingStorage.setListingNotes(listingId, typeof notes === 'string' ? notes : null);
|
||||
if (changes === 0) {
|
||||
return reply.code(404).send({ message: 'Listing not found' });
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(error);
|
||||
return reply.code(500).send({ message: 'Failed to update listing notes' });
|
||||
}
|
||||
|
||||
await trackPoi(TRACKING_POIS.NOTES_CREATE);
|
||||
return reply.send();
|
||||
});
|
||||
|
||||
fastify.post('/:listingId/status', async (request, reply) => {
|
||||
const { listingId } = request.params || {};
|
||||
const { status } = request.body || {};
|
||||
const userId = request.session?.currentUser;
|
||||
if (!listingId || !userId) {
|
||||
return reply.code(400).send({ message: 'listingId or user not provided' });
|
||||
}
|
||||
const allowed = ['applied', 'rejected', 'accepted'];
|
||||
const normalized = status == null ? null : String(status).toLowerCase();
|
||||
if (normalized != null && !allowed.includes(normalized)) {
|
||||
return reply.code(400).send({ message: `Invalid status: ${status}` });
|
||||
}
|
||||
try {
|
||||
const changes = listingStorage.setListingStatus(listingId, normalized);
|
||||
await trackPoi(TRACKING_POIS.USING_LISTING_STATUS);
|
||||
if (changes === 0) {
|
||||
return reply.code(404).send({ message: 'Listing not found' });
|
||||
}
|
||||
if (normalized != null) {
|
||||
watchListStorage.ensureWatch(listingId, userId);
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(error);
|
||||
return reply.code(500).send({ message: 'Failed to update listing status' });
|
||||
}
|
||||
return reply.send();
|
||||
});
|
||||
|
||||
fastify.delete('/job', async (request, reply) => {
|
||||
const { jobId, hardDelete = false } = request.body;
|
||||
const settings = await getSettings();
|
||||
|
||||
@@ -18,7 +18,7 @@ const notificationAdapter = await Promise.all(
|
||||
*/
|
||||
export default async function notificationAdapterPlugin(fastify) {
|
||||
fastify.get('/', async () => {
|
||||
return notificationAdapter.map((adapter) => adapter.config);
|
||||
return notificationAdapter.map((adapter) => adapter.config).filter(Boolean);
|
||||
});
|
||||
|
||||
fastify.post('/try', async (request, reply) => {
|
||||
|
||||
@@ -151,4 +151,28 @@ export default async function userSettingsPlugin(fastify) {
|
||||
return reply.code(500).send({ error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
fastify.post('/listing-deletion-preference', async (request, reply) => {
|
||||
const userId = request.session.currentUser;
|
||||
const { listing_deletion_preference } = request.body;
|
||||
|
||||
const globalSettings = await getSettings();
|
||||
if (globalSettings.demoMode && !isAdmin(request)) {
|
||||
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
|
||||
}
|
||||
|
||||
if (listing_deletion_preference == null) {
|
||||
return reply.code(400).send({ error: 'listing_deletion_preference is required.' });
|
||||
}
|
||||
|
||||
const { skipPrompt, hardDelete } = listing_deletion_preference;
|
||||
|
||||
try {
|
||||
upsertSettings({ listing_deletion_preference: { skipPrompt, hardDelete } }, userId);
|
||||
return { success: true };
|
||||
} catch (error) {
|
||||
logger.error('Error updating listing deletion preference', error);
|
||||
return reply.code(500).send({ error: error.message });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -155,6 +155,12 @@ export function createMcpServer() {
|
||||
),
|
||||
sortField: z.string().optional().describe('Sort by: created_at, price, size, provider, title, is_active'),
|
||||
sortDir: z.string().optional().describe('Sort direction: asc or desc'),
|
||||
status: z
|
||||
.enum(['applied', 'rejected', 'accepted', 'none'])
|
||||
.optional()
|
||||
.describe(
|
||||
'Filter by user-set status. "applied", "rejected", or "accepted" return only listings with that status; "none" returns only listings without a status set.',
|
||||
),
|
||||
},
|
||||
async (
|
||||
{
|
||||
@@ -170,6 +176,7 @@ export function createMcpServer() {
|
||||
maxPrice,
|
||||
sortField,
|
||||
sortDir,
|
||||
status,
|
||||
},
|
||||
extra,
|
||||
) => {
|
||||
@@ -192,6 +199,7 @@ export function createMcpServer() {
|
||||
maxPrice: maxPrice ?? null,
|
||||
sortField: sortField ?? null,
|
||||
sortDir: sortDir ?? 'desc',
|
||||
statusFilter: status,
|
||||
userId: user.id,
|
||||
isAdmin: user.isAdmin,
|
||||
});
|
||||
|
||||
@@ -124,10 +124,10 @@ export function normalizeListListings(queryResult, { page, pageSize }) {
|
||||
md += '\n\n';
|
||||
|
||||
if (listings.length > 0) {
|
||||
md += `| ID | Title | Address | Price | Size | Provider | Active | Created | Job |\n`;
|
||||
md += `|----|-------|---------|-------|------|----------|--------|---------|-----|\n`;
|
||||
md += `| ID | Title | Address | Price | Size | Provider | Active | Status | Created | Job |\n`;
|
||||
md += `|----|-------|---------|-------|------|----------|--------|--------|---------|-----|\n`;
|
||||
for (const l of listings) {
|
||||
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
|
||||
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${cell(l.status?.status)} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
|
||||
}
|
||||
md += `\nUse **get_listing** with an ID for full details (description, link, image).\n`;
|
||||
} else {
|
||||
@@ -156,6 +156,10 @@ export function normalizeGetListing(listing) {
|
||||
md += `- **Link:** ${listing.link || '–'}\n`;
|
||||
md += `- **Image:** ${listing.image_url || '–'}\n`;
|
||||
md += `- **Active:** ${listing.is_active ? 'yes' : 'no'}\n`;
|
||||
md += `- **Status:** ${listing.status?.status || '–'}\n`;
|
||||
if (listing.status?.setAt) {
|
||||
md += `- **Status set at:** ${formatDate(listing.status.setAt)}\n`;
|
||||
}
|
||||
md += `- **Created:** ${formatDate(listing.created_at)}\n`;
|
||||
md += `- **Job:** ${listing.job_name || '–'}\n`;
|
||||
if (listing.latitude != null && listing.longitude != null) {
|
||||
|
||||
@@ -9,6 +9,7 @@ import fetch from 'node-fetch';
|
||||
import pThrottle from 'p-throttle';
|
||||
import { normalizeImageUrl } from '../../utils.js';
|
||||
import logger from '../../services/logger.js';
|
||||
import { shouldUseMultipart, buildPhotoFormData } from './telegramPhotoUploader.js';
|
||||
|
||||
const RATE_LIMIT_INTERVAL = 1000;
|
||||
const chatThrottleMap = new Map();
|
||||
@@ -177,11 +178,13 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
|
||||
const jobName = job == null ? jobKey : job.name;
|
||||
|
||||
const throttledCall = getThrottled(chatId, async function (endpoint, body) {
|
||||
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, {
|
||||
method: 'post',
|
||||
body: JSON.stringify(body),
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
// FormData (multipart) vs JSON. node-fetch sets its own multipart boundary
|
||||
// header, so we must NOT supply Content-Type ourselves in that case.
|
||||
const isFormData = body instanceof FormData;
|
||||
const opts = isFormData
|
||||
? { method: 'post', body }
|
||||
: { method: 'post', body: JSON.stringify(body), headers: { 'Content-Type': 'application/json' } };
|
||||
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, opts);
|
||||
|
||||
if (!res.ok) {
|
||||
const errorBody = await res.text();
|
||||
@@ -208,16 +211,28 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
|
||||
});
|
||||
}
|
||||
|
||||
return await throttledCall('sendPhoto', {
|
||||
chat_id: chatId,
|
||||
photo: img,
|
||||
caption: plainText
|
||||
? buildCaptionPlain(jobName, serviceName, o, baseUrl)
|
||||
: buildCaption(jobName, serviceName, o, baseUrl),
|
||||
...(plainText ? {} : { parse_mode: 'HTML' }),
|
||||
...(message_thread_id ? { message_thread_id } : {}),
|
||||
}).catch(async (e) => {
|
||||
logger.error(`Error sending photo to Telegram and use a fallback: ${e.message}`);
|
||||
const caption = plainText
|
||||
? buildCaptionPlain(jobName, serviceName, o, baseUrl)
|
||||
: buildCaption(jobName, serviceName, o, baseUrl);
|
||||
const parseMode = plainText ? undefined : 'HTML';
|
||||
|
||||
// .webp URLs (Immowelt/Cloudimage) fail Telegram's URL-based sendPhoto with
|
||||
// "failed to get HTTP URL content". Upload the bytes via multipart instead;
|
||||
// the rendered chat message is identical.
|
||||
const photoCall = shouldUseMultipart(img)
|
||||
? buildPhotoFormData({ chatId, imageUrl: img, caption, parseMode, messageThreadId: message_thread_id }).then(
|
||||
(fd) => throttledCall('sendPhoto', fd),
|
||||
)
|
||||
: throttledCall('sendPhoto', {
|
||||
chat_id: chatId,
|
||||
photo: img,
|
||||
caption,
|
||||
...(parseMode ? { parse_mode: parseMode } : {}),
|
||||
...(message_thread_id ? { message_thread_id } : {}),
|
||||
});
|
||||
|
||||
return await photoCall.catch(async (e) => {
|
||||
logger.warn(`Error sending photo to Telegram and use a fallback: ${e.message}`);
|
||||
return await throttledCall('sendMessage', textPayload).catch((e) => {
|
||||
logger.error(`Error sending message to Telegram: ${e.message}`);
|
||||
throw e;
|
||||
|
||||
106
lib/notification/adapter/telegramPhotoUploader.js
Normal file
106
lib/notification/adapter/telegramPhotoUploader.js
Normal file
@@ -0,0 +1,106 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
/**
|
||||
* Helpers for sending photos to Telegram via `multipart/form-data` instead of
|
||||
* the HTTP-URL path. Used when the URL is one that Telegram's URL-fetcher will
|
||||
* reject - notably `.webp` images from Cloudimage (mms.immowelt.de), which
|
||||
* Telegram refuses with "Bad Request: failed to get HTTP URL content".
|
||||
*
|
||||
* The HTTP-URL path is faster and is still the default in telegram.js; this
|
||||
* module is the fallback for URLs whose extension makes Telegram fail.
|
||||
*/
|
||||
|
||||
/** Telegram's sendPhoto limit when uploading bytes via multipart/form-data. */
|
||||
const TELEGRAM_MULTIPART_MAX_BYTES = 10 * 1024 * 1024;
|
||||
|
||||
/** Accept header used when re-fetching the image ourselves.
|
||||
* Deliberately excludes `image/webp` so CDNs that content-negotiate
|
||||
* (like Cloudimage on mms.immowelt.de) transcode WEBP to JPEG. */
|
||||
const NON_WEBP_ACCEPT = 'image/jpeg,image/png,image/*;q=0.8';
|
||||
|
||||
/**
|
||||
* Returns true if the URL's path ends in a `.webp` extension. Such URLs need
|
||||
* multipart upload because Telegram identifies media types from the URL path
|
||||
* and rejects `.webp` in sendPhoto via HTTP URL.
|
||||
*
|
||||
* Conservative: returns false for null/empty/non-string input, malformed URLs,
|
||||
* and non-https schemes.
|
||||
*
|
||||
* @param {string|null|undefined} url
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function shouldUseMultipart(url) {
|
||||
if (typeof url !== 'string' || url.length === 0) return false;
|
||||
let parsed;
|
||||
try {
|
||||
parsed = new URL(url);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (parsed.protocol !== 'https:') return false;
|
||||
return /\.webp$/i.test(parsed.pathname);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch an image from `imageUrl` and build a `FormData` body suitable for
|
||||
* POSTing to `https://api.telegram.org/bot<token>/sendPhoto`.
|
||||
*
|
||||
* - Sends an `Accept` header that excludes `image/webp` so origin/CDN servers
|
||||
* that content-negotiate return JPEG bytes.
|
||||
* - Rejects images larger than Telegram's 10 MB multipart limit, both
|
||||
* advertised via `Content-Length` and (defensively) after download.
|
||||
* - The `photo` field is named with a `.jpg` extension because Telegram
|
||||
* identifies file type from the filename.
|
||||
*
|
||||
* Throws if the image fetch fails, the size limit is exceeded, or the URL is
|
||||
* unreachable. The caller is responsible for catching and falling back.
|
||||
*
|
||||
* @param {Object} args
|
||||
* @param {string|number} args.chatId
|
||||
* @param {string} args.imageUrl
|
||||
* @param {string} args.caption
|
||||
* @param {string} [args.parseMode] - Telegram parse_mode, e.g. 'HTML'.
|
||||
* @param {number} [args.messageThreadId] - Telegram supergroup topic id.
|
||||
* @returns {Promise<FormData>}
|
||||
*/
|
||||
export async function buildPhotoFormData({ chatId, imageUrl, caption, parseMode, messageThreadId }) {
|
||||
const res = await fetch(imageUrl, {
|
||||
method: 'GET',
|
||||
headers: { Accept: NON_WEBP_ACCEPT },
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`Failed to fetch image for multipart upload (${res.status}): ${imageUrl}`);
|
||||
}
|
||||
|
||||
const advertised = Number(res.headers.get('content-length'));
|
||||
if (Number.isFinite(advertised) && advertised > TELEGRAM_MULTIPART_MAX_BYTES) {
|
||||
throw new Error(
|
||||
`Image exceeds Telegram multipart size limit (advertised ${advertised} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
|
||||
);
|
||||
}
|
||||
|
||||
const buf = await res.arrayBuffer();
|
||||
if (buf.byteLength > TELEGRAM_MULTIPART_MAX_BYTES) {
|
||||
throw new Error(
|
||||
`Image exceeds Telegram multipart size limit (downloaded ${buf.byteLength} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Telegram identifies the media type from the filename extension. We always
|
||||
// upload as .jpg because the Accept header forces JPEG bytes from CDNs that
|
||||
// honor it; for the rare CDN that ignores Accept and still returns WEBP, the
|
||||
// .jpg filename is a small lie but Telegram's image pipeline accepts it.
|
||||
const blob = new Blob([buf], { type: 'image/jpeg' });
|
||||
|
||||
const fd = new FormData();
|
||||
fd.append('chat_id', String(chatId));
|
||||
fd.append('caption', caption);
|
||||
if (parseMode) fd.append('parse_mode', parseMode);
|
||||
if (messageThreadId != null) fd.append('message_thread_id', String(messageThreadId));
|
||||
fd.append('photo', blob, 'photo.jpg');
|
||||
return fd;
|
||||
}
|
||||
@@ -26,7 +26,7 @@ function parseId(shortenedLink) {
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser });
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immobilienDe_details' });
|
||||
if (!html) return listing;
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
|
||||
@@ -16,7 +16,7 @@ let appliedBlackList = [];
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser });
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immowelt_details' });
|
||||
if (!html) return listing;
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
|
||||
@@ -128,7 +128,7 @@ async function enrichListingFromDetails(listing, browser) {
|
||||
if (!absoluteLink) return listing;
|
||||
|
||||
try {
|
||||
const html = await puppeteerExtractor(absoluteLink, null, { browser });
|
||||
const html = await puppeteerExtractor(absoluteLink, null, { browser, name: 'kleinanzeigen_details' });
|
||||
if (!html) return { ...listing, link: absoluteLink };
|
||||
|
||||
const { detailAddress, detailDescription } = extractDetailFromHtml(html);
|
||||
@@ -196,8 +196,8 @@ const config = {
|
||||
id: '.aditem@data-adid',
|
||||
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
|
||||
tags: '.aditem-main--middle--tags | removeNewline | trim',
|
||||
title: '.aditem-main .text-module-begin a | removeNewline | trim',
|
||||
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
|
||||
title: '.aditem-main .text-module-begin | removeNewline | trim',
|
||||
link: '.aditem@data-href',
|
||||
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
|
||||
address: '.aditem-main--top--left | trim | removeNewline',
|
||||
image: 'img@src',
|
||||
|
||||
@@ -16,7 +16,7 @@ let appliedBlackList = [];
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, 'body', { browser });
|
||||
const html = await puppeteerExtractor(listing.link, 'body', { browser, name: 'sparkasse_details' });
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
const nextDataRaw = $('#__NEXT_DATA__').text;
|
||||
|
||||
@@ -16,7 +16,7 @@ let appliedBlackList = [];
|
||||
|
||||
async function fetchDetails(listing, browser) {
|
||||
try {
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser });
|
||||
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'wgGesucht_details' });
|
||||
if (!html) return listing;
|
||||
|
||||
const $ = cheerio.load(html);
|
||||
|
||||
@@ -29,11 +29,12 @@ export default class Extractor {
|
||||
* your response will never contain what you are really looking for
|
||||
* @param url
|
||||
* @param waitForSelector
|
||||
* @param jobKey
|
||||
*/
|
||||
execute = async (url, waitForSelector = null) => {
|
||||
execute = async (url, waitForSelector = null, jobKey = null) => {
|
||||
this.responseText = null;
|
||||
try {
|
||||
this.responseText = await puppeteerExtractor(url, waitForSelector, this.options);
|
||||
this.responseText = await puppeteerExtractor(url, waitForSelector, { ...this.options, name: jobKey });
|
||||
if (this.responseText != null) {
|
||||
loadParser(this.responseText);
|
||||
}
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
*/
|
||||
|
||||
import { launch } from 'cloakbrowser/puppeteer';
|
||||
import { debug, botDetected } from './utils.js';
|
||||
import { botDetected, debug } from './utils.js';
|
||||
import { getPreLaunchConfig } from './botPrevention.js';
|
||||
import logger from '../logger.js';
|
||||
import { trackPoi } from '../tracking/Tracker.js';
|
||||
@@ -50,7 +50,7 @@ export async function launchBrowser(url, options) {
|
||||
preCfg.windowSizeArg,
|
||||
];
|
||||
|
||||
const browser = await launch({
|
||||
return await launch({
|
||||
headless: options?.puppeteerHeadless ?? true,
|
||||
humanize: true,
|
||||
args,
|
||||
@@ -59,8 +59,6 @@ export async function launchBrowser(url, options) {
|
||||
...(options?.proxyUrl ? { proxy: options.proxyUrl } : {}),
|
||||
...(preCfg.timezone ? { timezone: preCfg.timezone } : {}),
|
||||
});
|
||||
|
||||
return browser;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -148,7 +146,11 @@ export default async function execute(url, waitForSelector, options) {
|
||||
if (botDetected(pageSource, statusCode)) {
|
||||
logger.warn('We have been detected as a bot :-/ Tried url: => ', url);
|
||||
|
||||
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
|
||||
if (options != null && options.name != null) {
|
||||
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT + '_' + options.name);
|
||||
} else {
|
||||
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
|
||||
}
|
||||
|
||||
result = null;
|
||||
} else {
|
||||
|
||||
@@ -141,6 +141,43 @@ const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
|
||||
'barrierefreie-wohnung-mieten': { equipment: ['handicappedaccessible'] },
|
||||
};
|
||||
|
||||
// SEO-optimized rental paths used by the ImmoScout web UI when the user
|
||||
// configures a maximum warmrent. Example: "wohnung-bis-800-euro-warm" means
|
||||
// "apartment for rent up to 800 EUR warmrent". The web UI generates these
|
||||
// paths instead of explicit `price` / `pricetype` query parameters.
|
||||
// Note: only the warmrent variant uses an SEO slug; max coldrent searches
|
||||
// use the regular "wohnung-mieten" path with explicit `price` and
|
||||
// `pricetype=rentpermonth` query params, which the existing translator
|
||||
// already handles.
|
||||
const SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE = {
|
||||
wohnung: 'apartmentrent',
|
||||
haus: 'houserent',
|
||||
};
|
||||
const SEO_MAX_WARMRENT_PATH_PATTERN = /^(?<type>wohnung|haus)-bis-(?<price>\d+)-euro-warm$/;
|
||||
|
||||
/**
|
||||
* Parses SEO-optimized ImmoScout web paths that encode a maximum warmrent, such
|
||||
* as "wohnung-bis-800-euro-warm". Returns the corresponding mobile API real
|
||||
* estate type and the implicit price/pricetype parameters, or null if the path
|
||||
* does not match the known SEO max-warmrent pattern.
|
||||
*
|
||||
* @param {string} realTypeKey The last segment of the URL path.
|
||||
* @returns {{ realType: string, additionalParams: Record<string, string> } | null}
|
||||
*/
|
||||
function parseSeoMaxWarmrentPath(realTypeKey) {
|
||||
const match = realTypeKey.match(SEO_MAX_WARMRENT_PATH_PATTERN);
|
||||
if (!match) return null;
|
||||
|
||||
const { type, price } = match.groups;
|
||||
return {
|
||||
realType: SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE[type],
|
||||
additionalParams: {
|
||||
price: `-${price}`,
|
||||
pricetype: 'calculatedtotalrent',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function convertWebToMobile(webUrl) {
|
||||
let url;
|
||||
try {
|
||||
@@ -164,7 +201,14 @@ export function convertWebToMobile(webUrl) {
|
||||
additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey];
|
||||
realType = REAL_ESTATE_TYPE['wohnung-mieten'];
|
||||
} else {
|
||||
throw new Error(`Real estate type not found: ${realTypeKey}`);
|
||||
// Test for SEO max-warmrent path, e.g. "wohnung-bis-800-euro-warm"
|
||||
const seoMaxWarmrent = parseSeoMaxWarmrentPath(realTypeKey);
|
||||
if (seoMaxWarmrent) {
|
||||
realType = seoMaxWarmrent.realType;
|
||||
additionalParamsFromWebPath = seoMaxWarmrent.additionalParams;
|
||||
} else {
|
||||
throw new Error(`Real estate type not found: ${realTypeKey}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ import * as similarityCache from '../similarity-check/similarityCache.js';
|
||||
import { isRunning, markFinished, markRunning } from './run-state.js';
|
||||
import { sendToUsers } from '../sse/sse-broker.js';
|
||||
import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
|
||||
import { getSettings } from '../storage/settingsStorage.js';
|
||||
|
||||
/**
|
||||
* Initializes the job execution service.
|
||||
@@ -160,6 +161,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
|
||||
}
|
||||
let browser;
|
||||
try {
|
||||
// Read the proxy live (not from the startup snapshot) so changing it in the
|
||||
// UI takes effect on the next run without a backend restart. An empty value
|
||||
// disables the proxy. Routing the headless browser through a (German
|
||||
// residential) proxy avoids datacenter-IP based bot detection on the
|
||||
// Puppeteer-based providers (immowelt, immonet, kleinanzeigen, ...).
|
||||
const liveSettings = await getSettings();
|
||||
const proxyUrl = typeof liveSettings?.proxyUrl === 'string' ? liveSettings.proxyUrl.trim() : '';
|
||||
|
||||
const jobProviders = job.provider.filter(
|
||||
(p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
|
||||
);
|
||||
@@ -168,14 +177,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
|
||||
const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
|
||||
matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
|
||||
|
||||
if (browser && !browser.isConnected()) {
|
||||
if (browser && !browser.connected) {
|
||||
logger.debug('Browser is disconnected, nullifying to launch a new one.');
|
||||
await puppeteerExtractor.closeBrowser(browser);
|
||||
browser = null;
|
||||
}
|
||||
|
||||
if (!browser && matchedProvider.config.getListings == null) {
|
||||
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
|
||||
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, proxyUrl ? { proxyUrl } : {});
|
||||
}
|
||||
|
||||
await new FredyPipelineExecutioner(matchedProvider.config, job, prov.id, similarityCache, browser).execute();
|
||||
|
||||
@@ -3,10 +3,27 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { nullOrEmpty } from '../../utils.js';
|
||||
import { nullOrEmpty, fromJson } from '../../utils.js';
|
||||
import SqliteConnection from './SqliteConnection.js';
|
||||
import { nanoid } from 'nanoid';
|
||||
|
||||
/**
|
||||
* Parse the JSON `status` column of a listing row in place.
|
||||
*
|
||||
* The DB stores status as a JSON payload `{ status, setAt }` (or NULL).
|
||||
* Consumers expect an object/null, so we normalize before returning.
|
||||
*
|
||||
* @param {Object|null|undefined} row - A raw row from the listings table.
|
||||
* @returns {Object|null|undefined} The same row with `status` parsed.
|
||||
*/
|
||||
const parseListingStatus = (row) => {
|
||||
if (row == null) return row;
|
||||
if (typeof row.status === 'string') {
|
||||
row.status = fromJson(row.status, null);
|
||||
}
|
||||
return row;
|
||||
};
|
||||
|
||||
/**
|
||||
* Return a list of known listing hashes for a given job and provider.
|
||||
* Useful to de-duplicate before inserting new listings.
|
||||
@@ -214,6 +231,8 @@ export const storeListings = (jobId, providerId, listings) => {
|
||||
longitude: item.longitude || null,
|
||||
};
|
||||
stmt.run(params);
|
||||
// Propagate the DB primary key back so downstream pipeline steps use the correct id
|
||||
item.id = params.id;
|
||||
}
|
||||
});
|
||||
|
||||
@@ -242,6 +261,7 @@ export const storeListings = (jobId, providerId, listings) => {
|
||||
* @param {object} [params.jobNameFilter]
|
||||
* @param {object} [params.providerFilter]
|
||||
* @param {object} [params.watchListFilter]
|
||||
* @param {('applied'|'rejected'|'accepted'|'none')} [params.statusFilter] - Filter by listing status. 'none' matches NULL.
|
||||
* @param {string|null} [params.sortField=null] - One of: 'created_at','price','size','provider','title'.
|
||||
* @param {('asc'|'desc')} [params.sortDir='asc']
|
||||
* @param {number} [params.createdAfter] - Only include listings created at or after this unix timestamp (ms).
|
||||
@@ -258,6 +278,7 @@ export const queryListings = ({
|
||||
jobIdFilter,
|
||||
providerFilter,
|
||||
watchListFilter,
|
||||
statusFilter,
|
||||
freeTextFilter,
|
||||
sortField = null,
|
||||
sortDir = 'asc',
|
||||
@@ -315,6 +336,18 @@ export const queryListings = ({
|
||||
} else if (watchListFilter === false) {
|
||||
whereParts.push('(wl.id IS NULL)');
|
||||
}
|
||||
// statusFilter: 'applied'|'rejected'|'accepted' -> equality on JSON status field; 'none' -> NULL.
|
||||
// The status column is a JSON payload `{ status, setAt }`, so we extract the inner
|
||||
// status string for comparison instead of matching the raw text.
|
||||
if (statusFilter === 'none') {
|
||||
whereParts.push('(l.status IS NULL)');
|
||||
} else if (
|
||||
typeof statusFilter === 'string' &&
|
||||
['applied', 'rejected', 'accepted'].includes(statusFilter.toLowerCase())
|
||||
) {
|
||||
params.statusValue = statusFilter.toLowerCase();
|
||||
whereParts.push(`(json_extract(l.status, '$.status') = @statusValue)`);
|
||||
}
|
||||
// Time range filters (unix timestamps in milliseconds)
|
||||
if (Number.isFinite(createdAfter) && createdAfter > 0) {
|
||||
params.createdAfter = createdAfter;
|
||||
@@ -389,7 +422,7 @@ export const queryListings = ({
|
||||
params,
|
||||
);
|
||||
|
||||
return { totalNumber, page: safePage, result: rows };
|
||||
return { totalNumber, page: safePage, result: rows.map(parseListingStatus) };
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -417,9 +450,10 @@ export const deleteListingsByJobId = (jobId, hardDelete = false) => {
|
||||
};
|
||||
|
||||
/**
|
||||
* Delete listings by a list of listing IDs.
|
||||
* Delete listings by a list of listing IDs (the nanoid primary key stored in the `id` column).
|
||||
* Used by API routes that receive row IDs from the client.
|
||||
*
|
||||
* @param {string[]} ids - Array of listing IDs to delete.
|
||||
* @param {string[]} ids - Array of DB row IDs to delete.
|
||||
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
|
||||
* @returns {any} The result from SqliteConnection.execute.
|
||||
*/
|
||||
@@ -623,7 +657,7 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
|
||||
if (!isAdmin) {
|
||||
whereScoping = `AND (j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`;
|
||||
}
|
||||
return (
|
||||
return parseListingStatus(
|
||||
SqliteConnection.query(
|
||||
`SELECT l.*, j.name AS job_name, CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END AS isWatched
|
||||
FROM listings l
|
||||
@@ -631,10 +665,57 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
|
||||
LEFT JOIN watch_list wl ON wl.listing_id = l.id AND wl.user_id = @userId
|
||||
WHERE l.id = @id AND l.manually_deleted = 0 ${whereScoping}`,
|
||||
params,
|
||||
)[0] || null
|
||||
)[0] || null,
|
||||
);
|
||||
};
|
||||
|
||||
/**
|
||||
* Set or clear the notes attached to a single listing.
|
||||
*
|
||||
* Empty strings are normalized to NULL so the DB doesn't keep meaningless
|
||||
* whitespace and queries can filter "has notes" with a simple IS NOT NULL.
|
||||
*
|
||||
* @param {string} id - The listing ID.
|
||||
* @param {string|null} notes - The note text to store, or null/empty to clear.
|
||||
* @returns {number} Number of rows affected (0 if listing not found).
|
||||
*/
|
||||
export const setListingNotes = (id, notes) => {
|
||||
if (!id) return 0;
|
||||
const trimmed = typeof notes === 'string' ? notes.trim() : null;
|
||||
const value = trimmed && trimmed.length > 0 ? trimmed : null;
|
||||
const res = SqliteConnection.execute(`UPDATE listings SET notes = @notes WHERE id = @id`, {
|
||||
id,
|
||||
notes: value,
|
||||
});
|
||||
return res?.changes ?? 0;
|
||||
};
|
||||
|
||||
/**
|
||||
* Set or clear the status of a single listing.
|
||||
*
|
||||
* The status column stores a JSON payload `{ status, setAt }` so consumers
|
||||
* can show both the user's decision and when it was made. Passing `null`
|
||||
* clears the column.
|
||||
*
|
||||
* @param {string} id - The listing ID.
|
||||
* @param {('applied'|'rejected'|'accepted'|null)} status - New status, or null to clear.
|
||||
* @returns {number} Number of rows affected (0 if listing not found).
|
||||
*/
|
||||
export const setListingStatus = (id, status) => {
|
||||
if (!id) return 0;
|
||||
const allowed = ['applied', 'rejected', 'accepted'];
|
||||
const normalized = status == null ? null : String(status).toLowerCase();
|
||||
if (normalized != null && !allowed.includes(normalized)) {
|
||||
throw new Error(`Invalid listing status: ${status}`);
|
||||
}
|
||||
const payload = normalized == null ? null : JSON.stringify({ status: normalized, setAt: Date.now() });
|
||||
const res = SqliteConnection.execute(`UPDATE listings SET status = @status WHERE id = @id`, {
|
||||
id,
|
||||
status: payload,
|
||||
});
|
||||
return res?.changes ?? 0;
|
||||
};
|
||||
|
||||
/**
|
||||
* Resets geocoordinates and distance for all listings related to a user.
|
||||
*
|
||||
|
||||
11
lib/services/storage/migrations/sql/18.add-listing-status.js
Normal file
11
lib/services/storage/migrations/sql/18.add-listing-status.js
Normal file
@@ -0,0 +1,11 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
export function up(db) {
|
||||
db.exec(`
|
||||
ALTER TABLE listings ADD COLUMN status JSON;
|
||||
CREATE INDEX IF NOT EXISTS idx_listings_status ON listings (json_extract(status, '$.status'));
|
||||
`);
|
||||
}
|
||||
10
lib/services/storage/migrations/sql/19.add-listing-notes.js
Normal file
10
lib/services/storage/migrations/sql/19.add-listing-notes.js
Normal file
@@ -0,0 +1,10 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
export function up(db) {
|
||||
db.exec(`
|
||||
ALTER TABLE listings ADD COLUMN notes TEXT;
|
||||
`);
|
||||
}
|
||||
@@ -47,6 +47,25 @@ export const deleteWatch = (listingId, userId) => {
|
||||
return { deleted: Boolean(res?.changes) };
|
||||
};
|
||||
|
||||
/**
|
||||
* Ensure a watch entry exists. Does not toggle; safe to call when row may already exist.
|
||||
* Used by the status endpoint to auto-watch a listing when a status is set.
|
||||
* @param {string} listingId
|
||||
* @param {string} userId
|
||||
* @returns {{watched:boolean}}
|
||||
*/
|
||||
export const ensureWatch = (listingId, userId) => {
|
||||
if (!listingId || !userId) return { watched: false };
|
||||
const { created } = createWatch(listingId, userId);
|
||||
if (created) return { watched: true };
|
||||
const exists =
|
||||
SqliteConnection.query(
|
||||
`SELECT 1 AS ok FROM watch_list WHERE listing_id = @listing_id AND user_id = @user_id LIMIT 1`,
|
||||
{ listing_id: listingId, user_id: userId },
|
||||
).length > 0;
|
||||
return { watched: exists };
|
||||
};
|
||||
|
||||
/**
|
||||
* Toggle a watch entry. If exists -> delete, otherwise create.
|
||||
* @param {string} listingId
|
||||
|
||||
50
package.json
50
package.json
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "fredy",
|
||||
"version": "22.0.4",
|
||||
"version": "22.3.0",
|
||||
"description": "[F]ind [R]eal [E]states [d]amn eas[y].",
|
||||
"scripts": {
|
||||
"prepare": "husky",
|
||||
@@ -62,9 +62,9 @@
|
||||
"Firefox ESR"
|
||||
],
|
||||
"dependencies": {
|
||||
"@douyinfe/semi-icons": "^2.97.0",
|
||||
"@douyinfe/semi-ui": "2.97.0",
|
||||
"@douyinfe/semi-ui-19": "^2.97.0",
|
||||
"@douyinfe/semi-icons": "^2.99.3",
|
||||
"@douyinfe/semi-ui": "2.99.3",
|
||||
"@douyinfe/semi-ui-19": "^2.99.3",
|
||||
"@fastify/cookie": "^11.0.2",
|
||||
"@fastify/helmet": "^13.0.2",
|
||||
"@fastify/session": "^11.1.1",
|
||||
@@ -73,12 +73,12 @@
|
||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||
"@sendgrid/mail": "8.1.6",
|
||||
"@turf/boolean-point-in-polygon": "^7.3.5",
|
||||
"@vitejs/plugin-react": "6.0.1",
|
||||
"@vitejs/plugin-react": "6.0.2",
|
||||
"adm-zip": "^0.5.17",
|
||||
"better-sqlite3": "^12.9.0",
|
||||
"better-sqlite3": "^12.10.0",
|
||||
"chart.js": "^4.5.1",
|
||||
"cheerio": "^1.2.0",
|
||||
"cloakbrowser": "^0.3.28",
|
||||
"cloakbrowser": "^0.3.31",
|
||||
"fastify": "^5.8.5",
|
||||
"handlebars": "4.7.9",
|
||||
"maplibre-gl": "^5.24.0",
|
||||
@@ -86,41 +86,41 @@
|
||||
"node-cron": "^4.2.1",
|
||||
"node-fetch": "3.3.2",
|
||||
"node-mailjet": "6.0.11",
|
||||
"nodemailer": "^8.0.7",
|
||||
"nodemailer": "^8.0.10",
|
||||
"p-throttle": "^8.1.0",
|
||||
"package-up": "^5.0.0",
|
||||
"puppeteer-core": "^24.43.1",
|
||||
"query-string": "9.3.1",
|
||||
"react": "19.2.6",
|
||||
"puppeteer-core": "^25.1.0",
|
||||
"query-string": "9.4.0",
|
||||
"react": "19.2.7",
|
||||
"react-chartjs-2": "^5.3.1",
|
||||
"react-dom": "19.2.6",
|
||||
"react-dom": "19.2.7",
|
||||
"react-range-slider-input": "^3.3.5",
|
||||
"react-router": "7.15.0",
|
||||
"react-router-dom": "7.15.0",
|
||||
"resend": "^6.12.3",
|
||||
"semver": "^7.8.0",
|
||||
"react-router": "7.16.0",
|
||||
"react-router-dom": "7.16.0",
|
||||
"resend": "^6.12.4",
|
||||
"semver": "^7.8.1",
|
||||
"slack": "11.0.2",
|
||||
"vite": "8.0.12",
|
||||
"vite": "8.0.16",
|
||||
"x-var": "^3.0.1",
|
||||
"zustand": "^5.0.13"
|
||||
"zustand": "^5.0.14"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@babel/core": "7.29.0",
|
||||
"@babel/eslint-parser": "7.28.6",
|
||||
"@babel/preset-env": "7.29.5",
|
||||
"@babel/preset-react": "7.28.5",
|
||||
"@babel/core": "7.29.7",
|
||||
"@babel/eslint-parser": "7.29.7",
|
||||
"@babel/preset-env": "7.29.7",
|
||||
"@babel/preset-react": "7.29.7",
|
||||
"@eslint/js": "^10.0.1",
|
||||
"chalk": "^5.6.2",
|
||||
"eslint": "10.3.0",
|
||||
"eslint": "10.4.1",
|
||||
"eslint-config-prettier": "10.1.8",
|
||||
"eslint-plugin-react": "7.37.5",
|
||||
"globals": "^17.6.0",
|
||||
"history": "5.3.0",
|
||||
"husky": "9.1.7",
|
||||
"less": "4.6.4",
|
||||
"lint-staged": "17.0.4",
|
||||
"lint-staged": "17.0.7",
|
||||
"nodemon": "^3.1.14",
|
||||
"prettier": "3.8.3",
|
||||
"vitest": "^4.1.6"
|
||||
"vitest": "^4.1.8"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,4 +32,7 @@ export const deletedIds = [];
|
||||
export const deleteListingsById = (ids) => {
|
||||
deletedIds.push(...ids);
|
||||
};
|
||||
export const deleteListingsByHash = (hashes) => {
|
||||
deletedIds.push(...hashes);
|
||||
};
|
||||
/* eslint-enable no-unused-vars */
|
||||
|
||||
360
test/notification/telegram.test.js
Normal file
360
test/notification/telegram.test.js
Normal file
@@ -0,0 +1,360 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
// Mock external deps BEFORE importing the module under test.
|
||||
vi.mock('node-fetch', () => ({ default: vi.fn() }));
|
||||
vi.mock('../../lib/services/storage/jobStorage.js', () => ({
|
||||
getJob: (jobKey) => ({ id: jobKey, name: jobKey }),
|
||||
}));
|
||||
vi.mock('../../lib/services/markdown.js', () => ({
|
||||
markdown2Html: () => '',
|
||||
}));
|
||||
|
||||
// Helpers to build mock fetch responses.
|
||||
function jsonOk(body = { ok: true }) {
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
text: async () => JSON.stringify(body),
|
||||
};
|
||||
}
|
||||
|
||||
function jsonErr(status, body) {
|
||||
return {
|
||||
ok: false,
|
||||
status,
|
||||
text: async () => JSON.stringify(body),
|
||||
};
|
||||
}
|
||||
|
||||
function imageOk(bytes = new Uint8Array([0xff, 0xd8, 0xff])) {
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: {
|
||||
get: (h) => {
|
||||
const k = h.toLowerCase();
|
||||
if (k === 'content-type') return 'image/jpeg';
|
||||
if (k === 'content-length') return String(bytes.byteLength);
|
||||
return null;
|
||||
},
|
||||
},
|
||||
arrayBuffer: async () => bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength),
|
||||
};
|
||||
}
|
||||
|
||||
// Globals are mocked too so buildPhotoFormData (which uses global fetch) can be
|
||||
// intercepted by the same single mock.
|
||||
let mockNodeFetch;
|
||||
let mockGlobalFetch;
|
||||
let send;
|
||||
|
||||
beforeEach(async () => {
|
||||
// Reset modules to get a fresh import with our mocks applied.
|
||||
vi.resetModules();
|
||||
const nodeFetchMod = await import('node-fetch');
|
||||
mockNodeFetch = nodeFetchMod.default;
|
||||
mockNodeFetch.mockReset();
|
||||
|
||||
mockGlobalFetch = vi.fn();
|
||||
vi.stubGlobal('fetch', mockGlobalFetch);
|
||||
|
||||
({ send } = await import('../../lib/notification/adapter/telegram.js'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
const baseConfig = {
|
||||
id: 'telegram',
|
||||
fields: { token: 'TKN', chatId: '999' },
|
||||
};
|
||||
|
||||
describe('telegram send() - HTTP URL path (default for .jpg / .png)', () => {
|
||||
it('POSTs JSON to sendPhoto for a .jpg image URL', async () => {
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 'Listing',
|
||||
link: 'https://example.com/a',
|
||||
address: 'Addr',
|
||||
price: '500€',
|
||||
size: '50m²',
|
||||
image: 'https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
const [url, opts] = mockNodeFetch.mock.calls[0];
|
||||
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
|
||||
expect(opts.method).toBe('post');
|
||||
expect(opts.headers?.['Content-Type']).toBe('application/json');
|
||||
const body = JSON.parse(opts.body);
|
||||
expect(body.chat_id).toBe('999');
|
||||
expect(body.photo).toBe('https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394');
|
||||
expect(body.parse_mode).toBe('HTML');
|
||||
});
|
||||
|
||||
it('does NOT pre-fetch the image when using HTTP URL path', async () => {
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/x.jpg',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
// global fetch (used by buildPhotoFormData) must not be called
|
||||
expect(mockGlobalFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('falls back to sendMessage when sendPhoto fails', async () => {
|
||||
mockNodeFetch
|
||||
.mockResolvedValueOnce(jsonErr(400, { ok: false, description: 'boom' }))
|
||||
.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/x.jpg',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
|
||||
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendPhoto');
|
||||
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
});
|
||||
});
|
||||
|
||||
describe('telegram send() - multipart path (.webp URLs)', () => {
|
||||
it('pre-fetches the image then POSTs FormData to sendPhoto for a .webp URL', async () => {
|
||||
// 1st: GET image via global fetch
|
||||
mockGlobalFetch.mockResolvedValueOnce(imageOk());
|
||||
// 2nd: POST sendPhoto via node-fetch
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 'Listing',
|
||||
link: 'https://example.com/a',
|
||||
address: 'Addr',
|
||||
price: '500€',
|
||||
size: '50m²',
|
||||
image: 'https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
// image was fetched
|
||||
expect(mockGlobalFetch).toHaveBeenCalledTimes(1);
|
||||
expect(mockGlobalFetch.mock.calls[0][0]).toBe('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394');
|
||||
|
||||
// sendPhoto called via node-fetch with FormData
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
const [url, opts] = mockNodeFetch.mock.calls[0];
|
||||
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
|
||||
expect(opts.method).toBe('post');
|
||||
expect(opts.body).toBeInstanceOf(FormData);
|
||||
// No explicit Content-Type header - fetch sets multipart boundary itself
|
||||
expect(opts.headers).toBeUndefined();
|
||||
expect(opts.body.get('chat_id')).toBe('999');
|
||||
expect(opts.body.get('parse_mode')).toBe('HTML');
|
||||
const photo = opts.body.get('photo');
|
||||
expect(photo).toBeTruthy();
|
||||
expect(photo.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('falls back to sendMessage when the image pre-fetch fails for a .webp URL', async () => {
|
||||
// image fetch fails (404 from CDN)
|
||||
mockGlobalFetch.mockResolvedValueOnce({
|
||||
ok: false,
|
||||
status: 404,
|
||||
headers: { get: () => null },
|
||||
arrayBuffer: async () => new ArrayBuffer(0),
|
||||
});
|
||||
// then sendMessage succeeds via node-fetch
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/gone.webp',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
});
|
||||
|
||||
it('falls back to sendMessage when multipart sendPhoto returns a Telegram error', async () => {
|
||||
mockGlobalFetch.mockResolvedValueOnce(imageOk());
|
||||
mockNodeFetch
|
||||
.mockResolvedValueOnce(jsonErr(400, { description: 'broke' })) // multipart sendPhoto
|
||||
.mockResolvedValueOnce(jsonOk()); // sendMessage fallback
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/x.webp',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
|
||||
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
});
|
||||
});
|
||||
|
||||
describe('telegram send() - mixed batch (regression-safety)', () => {
|
||||
it('handles a batch with both .jpg and .webp - jpg uses URL, webp uses multipart', async () => {
|
||||
// .webp image fetch
|
||||
mockGlobalFetch.mockResolvedValueOnce(imageOk());
|
||||
// both sendPhoto calls succeed
|
||||
mockNodeFetch
|
||||
.mockResolvedValueOnce(jsonOk()) // could be either listing first
|
||||
.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'jpg-listing',
|
||||
title: 'a',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/a.jpg',
|
||||
},
|
||||
{
|
||||
id: 'webp-listing',
|
||||
title: 'b',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: 'https://example.com/b.webp',
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockGlobalFetch).toHaveBeenCalledTimes(1); // only webp pre-fetches
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
|
||||
|
||||
// Verify one call had FormData and one had JSON body
|
||||
const bodies = mockNodeFetch.mock.calls.map((c) => c[1].body);
|
||||
const hasFormData = bodies.some((b) => b instanceof FormData);
|
||||
const hasJson = bodies.some((b) => typeof b === 'string' && b.startsWith('{'));
|
||||
expect(hasFormData).toBe(true);
|
||||
expect(hasJson).toBe(true);
|
||||
});
|
||||
|
||||
it('uses sendMessage (not sendPhoto) when image is null', async () => {
|
||||
mockNodeFetch.mockResolvedValueOnce(jsonOk());
|
||||
|
||||
await send({
|
||||
serviceName: 'immowelt',
|
||||
newListings: [
|
||||
{
|
||||
id: 'a',
|
||||
title: 't',
|
||||
link: 'l',
|
||||
address: 'a',
|
||||
price: '',
|
||||
size: '',
|
||||
image: null,
|
||||
},
|
||||
],
|
||||
notificationConfig: [baseConfig],
|
||||
jobKey: 'Berlin',
|
||||
});
|
||||
|
||||
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
|
||||
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
|
||||
expect(mockGlobalFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('telegram send() - config validation', () => {
|
||||
it('throws when telegram adapter config is missing', () => {
|
||||
expect(() =>
|
||||
send({
|
||||
serviceName: 's',
|
||||
newListings: [],
|
||||
notificationConfig: [],
|
||||
jobKey: 'k',
|
||||
}),
|
||||
).toThrow(/configuration missing/);
|
||||
});
|
||||
|
||||
it('throws when token or chatId is missing', () => {
|
||||
expect(() =>
|
||||
send({
|
||||
serviceName: 's',
|
||||
newListings: [],
|
||||
notificationConfig: [{ id: 'telegram', fields: { token: '' } }],
|
||||
jobKey: 'k',
|
||||
}),
|
||||
).toThrow(/token.*chatId/);
|
||||
});
|
||||
});
|
||||
287
test/notification/telegramPhotoUploader.test.js
Normal file
287
test/notification/telegramPhotoUploader.test.js
Normal file
@@ -0,0 +1,287 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
import { shouldUseMultipart, buildPhotoFormData } from '../../lib/notification/adapter/telegramPhotoUploader.js';
|
||||
|
||||
describe('shouldUseMultipart', () => {
|
||||
it('returns true for .webp URL with query string', () => {
|
||||
expect(shouldUseMultipart('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true for .webp URL without query string', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.webp')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true for uppercase .WEBP extension', () => {
|
||||
expect(shouldUseMultipart('https://example.com/IMG.WEBP?x=1')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false for .jpg URL with query string', () => {
|
||||
expect(shouldUseMultipart('https://mms.immowelt.de/a/b/c/d/xyz.jpg?ci_seal=hash&w=525&h=394')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for .jpeg URL', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.jpeg')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for .png URL with query string', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.png?w=100')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for .gif URL', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.gif')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for null', () => {
|
||||
expect(shouldUseMultipart(null)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for undefined', () => {
|
||||
expect(shouldUseMultipart(undefined)).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for empty string', () => {
|
||||
expect(shouldUseMultipart('')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for malformed URL', () => {
|
||||
expect(shouldUseMultipart('not a url')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for URL where webp is in the query but not the path', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo.jpg?format=webp')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for URL with no extension at all', () => {
|
||||
expect(shouldUseMultipart('https://example.com/photo')).toBe(false);
|
||||
});
|
||||
|
||||
it('returns false for non-https schemes', () => {
|
||||
// file/data/ftp URLs should not be relevant; safer to skip multipart
|
||||
expect(shouldUseMultipart('http://example.com/photo.webp')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildPhotoFormData', () => {
|
||||
let mockFetch;
|
||||
|
||||
beforeEach(() => {
|
||||
mockFetch = vi.fn();
|
||||
vi.stubGlobal('fetch', mockFetch);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
function makeImageResponse({ contentType = 'image/jpeg', bytes = new Uint8Array([0xff, 0xd8, 0xff]) } = {}) {
|
||||
const buf = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: {
|
||||
get: (h) =>
|
||||
h.toLowerCase() === 'content-type'
|
||||
? contentType
|
||||
: h.toLowerCase() === 'content-length'
|
||||
? String(bytes.byteLength)
|
||||
: null,
|
||||
},
|
||||
arrayBuffer: async () => buf,
|
||||
};
|
||||
}
|
||||
|
||||
it('fetches image with Accept header that excludes webp so the CDN transcodes to JPEG', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
await buildPhotoFormData({
|
||||
chatId: '123',
|
||||
imageUrl: 'https://example.com/photo.webp',
|
||||
caption: 'hi',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(mockFetch).toHaveBeenCalledTimes(1);
|
||||
const [url, opts] = mockFetch.mock.calls[0];
|
||||
expect(url).toBe('https://example.com/photo.webp');
|
||||
expect(opts?.headers?.Accept || opts?.headers?.accept).toMatch(/image\/jpeg/);
|
||||
expect(opts?.headers?.Accept || opts?.headers?.accept).not.toMatch(/image\/webp/);
|
||||
});
|
||||
|
||||
it('returns FormData containing chat_id, caption, parse_mode, and photo fields', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '12345',
|
||||
imageUrl: 'https://example.com/abc.webp',
|
||||
caption: 'My caption',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd).toBeInstanceOf(FormData);
|
||||
expect(fd.get('chat_id')).toBe('12345');
|
||||
expect(fd.get('caption')).toBe('My caption');
|
||||
expect(fd.get('parse_mode')).toBe('HTML');
|
||||
const photo = fd.get('photo');
|
||||
expect(photo).toBeTruthy();
|
||||
// File-like (Blob); has a name and a size
|
||||
expect(typeof photo.name).toBe('string');
|
||||
expect(photo.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('uses a .jpg filename (Telegram uses URL/filename extension for type detection)', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
const photo = fd.get('photo');
|
||||
expect(photo.name).toMatch(/\.jpg$/i);
|
||||
});
|
||||
|
||||
it('includes message_thread_id when provided', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
messageThreadId: 42,
|
||||
});
|
||||
|
||||
expect(fd.get('message_thread_id')).toBe('42');
|
||||
});
|
||||
|
||||
it('omits message_thread_id when not provided', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd.get('message_thread_id')).toBeNull();
|
||||
});
|
||||
|
||||
it('omits parse_mode when not provided (plain text mode)', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/source.webp',
|
||||
caption: 'c',
|
||||
});
|
||||
|
||||
expect(fd.get('parse_mode')).toBeNull();
|
||||
});
|
||||
|
||||
it('throws when the image fetch returns non-200', async () => {
|
||||
mockFetch.mockResolvedValueOnce({
|
||||
ok: false,
|
||||
status: 404,
|
||||
headers: { get: () => null },
|
||||
arrayBuffer: async () => new ArrayBuffer(0),
|
||||
});
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/gone.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/404/);
|
||||
});
|
||||
|
||||
it('throws when the image fetch throws (network error)', async () => {
|
||||
mockFetch.mockRejectedValueOnce(new Error('ECONNREFUSED'));
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/x.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/ECONNREFUSED/);
|
||||
});
|
||||
|
||||
it('throws when the image exceeds 10 MB (Telegram multipart limit)', async () => {
|
||||
// 11 MB
|
||||
const big = new Uint8Array(11 * 1024 * 1024);
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes: big }));
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/huge.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/size|large|10/i);
|
||||
});
|
||||
|
||||
it('rejects early when content-length header advertises > 10 MB (avoids download)', async () => {
|
||||
mockFetch.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
status: 200,
|
||||
headers: {
|
||||
get: (h) => {
|
||||
const k = h.toLowerCase();
|
||||
if (k === 'content-type') return 'image/jpeg';
|
||||
if (k === 'content-length') return String(50 * 1024 * 1024);
|
||||
return null;
|
||||
},
|
||||
},
|
||||
arrayBuffer: async () => {
|
||||
throw new Error('should not be called - size check should reject first');
|
||||
},
|
||||
});
|
||||
|
||||
await expect(
|
||||
buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/huge.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
}),
|
||||
).rejects.toThrow(/size|large|10/i);
|
||||
});
|
||||
|
||||
it('accepts exactly 10 MB images (boundary)', async () => {
|
||||
const bytes = new Uint8Array(10 * 1024 * 1024);
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes }));
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: '1',
|
||||
imageUrl: 'https://example.com/exact.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd.get('photo').size).toBe(10 * 1024 * 1024);
|
||||
});
|
||||
|
||||
it('coerces non-string chatId (number) to string in form data', async () => {
|
||||
mockFetch.mockResolvedValueOnce(makeImageResponse());
|
||||
|
||||
const fd = await buildPhotoFormData({
|
||||
chatId: 999,
|
||||
imageUrl: 'https://example.com/x.webp',
|
||||
caption: 'c',
|
||||
parseMode: 'HTML',
|
||||
});
|
||||
|
||||
expect(fd.get('chat_id')).toBe('999');
|
||||
});
|
||||
});
|
||||
@@ -38,6 +38,20 @@ async function tryReadFile(filepath) {
|
||||
}
|
||||
}
|
||||
|
||||
function withRealEstateType(data, realEstateType) {
|
||||
if (!realEstateType?.length || !Array.isArray(data?.resultListItems)) {
|
||||
return data;
|
||||
}
|
||||
|
||||
const cloned = typeof structuredClone === 'function' ? structuredClone(data) : JSON.parse(JSON.stringify(data));
|
||||
for (const item of cloned.resultListItems) {
|
||||
if (item?.type === 'EXPOSE_RESULT' && item?.item) {
|
||||
item.item.realEstateType = realEstateType;
|
||||
}
|
||||
}
|
||||
return cloned;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns fixture HTML for the given URL by mapping hostname → provider name,
|
||||
* then distinguishing list vs detail pages by comparing the URL path against
|
||||
@@ -83,7 +97,10 @@ export function buildFetchMock() {
|
||||
const raw = await tryReadFile(path.join(FIXTURES_DIR, 'immoscout_list.json'));
|
||||
listData = raw ? JSON.parse(raw) : { resultListItems: [] };
|
||||
}
|
||||
return { ok: true, status: 200, json: () => Promise.resolve(listData) };
|
||||
|
||||
const requestedType = new URL(urlStr).searchParams.get('realestatetype');
|
||||
const responseData = withRealEstateType(listData, requestedType);
|
||||
return { ok: true, status: 200, json: () => Promise.resolve(responseData) };
|
||||
}
|
||||
|
||||
if (urlStr.includes('api.mobile.immobilienscout24.de/expose/')) {
|
||||
|
||||
37
test/services/extractor/puppeteerExtractor.test.js
Normal file
37
test/services/extractor/puppeteerExtractor.test.js
Normal file
@@ -0,0 +1,37 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { vi, describe, it, expect, beforeEach } from 'vitest';
|
||||
|
||||
// Mock the CloakBrowser launcher so no real Chromium binary is needed and we can
|
||||
// assert which options get forwarded to it.
|
||||
const { launchMock } = vi.hoisted(() => ({ launchMock: vi.fn() }));
|
||||
|
||||
vi.mock('cloakbrowser/puppeteer', () => ({
|
||||
launch: launchMock,
|
||||
}));
|
||||
|
||||
const { launchBrowser } = await import('../../../lib/services/extractor/puppeteerExtractor.js');
|
||||
|
||||
describe('launchBrowser proxy forwarding', () => {
|
||||
beforeEach(() => {
|
||||
launchMock.mockReset();
|
||||
launchMock.mockResolvedValue({ close: async () => {} });
|
||||
});
|
||||
|
||||
it('forwards proxyUrl to CloakBrowser as the proxy option', async () => {
|
||||
await launchBrowser('https://www.immowelt.de/', { proxyUrl: 'http://user:pass@host:8080' });
|
||||
|
||||
expect(launchMock).toHaveBeenCalledTimes(1);
|
||||
expect(launchMock.mock.calls[0][0]).toMatchObject({ proxy: 'http://user:pass@host:8080' });
|
||||
});
|
||||
|
||||
it('does not set a proxy when no proxyUrl is given', async () => {
|
||||
await launchBrowser('https://www.immowelt.de/', {});
|
||||
|
||||
expect(launchMock).toHaveBeenCalledTimes(1);
|
||||
expect(launchMock.mock.calls[0][0].proxy).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -4,11 +4,16 @@
|
||||
*/
|
||||
|
||||
import { convertWebToMobile } from '../../../lib/services/immoscout/immoscout-web-translator.js';
|
||||
import { expect } from 'vitest';
|
||||
import { expect, vi } from 'vitest';
|
||||
import { readFile } from 'fs/promises';
|
||||
import { buildFetchMock } from '../../offlineFixtures.js';
|
||||
|
||||
export const testData = JSON.parse(await readFile(new URL('./testdata.json', import.meta.url)));
|
||||
|
||||
if (process.env.TEST_MODE === 'offline') {
|
||||
vi.stubGlobal('fetch', buildFetchMock());
|
||||
}
|
||||
|
||||
describe('#immoscout-mobile URL conversion', () => {
|
||||
// Test shape URL conversion
|
||||
it('should convert a full web URL with shape to mobile URL', () => {
|
||||
@@ -41,6 +46,60 @@ describe('#immoscout-mobile URL conversion', () => {
|
||||
expect(queryParams.get('equipment').split(',')).toEqual(expect.arrayContaining(['garden', 'balcony']));
|
||||
});
|
||||
|
||||
// Test URL conversion of SEO web path for max warmrent. The ImmoScout web UI
|
||||
// generates this special SEO slug instead of explicit price/pricetype params
|
||||
// when the user configures a "Warmmiete" filter (real-world URL).
|
||||
it('should convert a SEO apartment max warmrent path to rent + price + pricetype', () => {
|
||||
const webUrl =
|
||||
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-bis-800-euro-warm?livingspace=-800.0&enteredFrom=result_list';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
|
||||
expect(queryParams.get('price')).toBe('-800');
|
||||
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
|
||||
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
|
||||
expect(queryParams.get('livingspace')).toBe('-800.0');
|
||||
});
|
||||
|
||||
// Same SEO pattern for houses ("haus-bis-X-euro-warm" → houserent).
|
||||
it('should convert a SEO house max warmrent path to rent + price + pricetype', () => {
|
||||
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/haus-bis-1500-euro-warm';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('houserent');
|
||||
expect(queryParams.get('price')).toBe('-1500');
|
||||
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
|
||||
});
|
||||
|
||||
// Sanity check: max coldrent ("Kaltmiete") does NOT use an SEO slug. The web
|
||||
// UI keeps the regular "wohnung-mieten" path and passes explicit
|
||||
// price + pricetype query params, which the existing translator already
|
||||
// handles (real-world URL).
|
||||
it('should convert a max coldrent search via the regular wohnung-mieten path', () => {
|
||||
const webUrl =
|
||||
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-mieten?price=-800.0&livingspace=-800.0&pricetype=rentpermonth&enteredFrom=result_list';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
|
||||
expect(queryParams.get('price')).toBe('-800.0');
|
||||
expect(queryParams.get('pricetype')).toBe('rentpermonth');
|
||||
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
|
||||
});
|
||||
|
||||
// Explicit query params win over the SEO slug's implicit defaults.
|
||||
it('should let explicit query params override SEO path price defaults', () => {
|
||||
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-bis-800-euro-warm?price=100-500';
|
||||
|
||||
const converted = convertWebToMobile(webUrl);
|
||||
const queryParams = new URL(converted).searchParams;
|
||||
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
|
||||
expect(queryParams.get('price')).toBe('100-500');
|
||||
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
|
||||
});
|
||||
|
||||
// Test URL conversion with unsupported query parameters
|
||||
it('should remove unsupported query parameters', () => {
|
||||
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?minimuminternetspeed=100000';
|
||||
|
||||
@@ -18,6 +18,7 @@ describe('services/jobs/jobExecutionService', () => {
|
||||
const busPath = root + '/lib/services/events/event-bus.js';
|
||||
const jobStoragePath = root + '/lib/services/storage/jobStorage.js';
|
||||
const userStoragePath = root + '/lib/services/storage/userStorage.js';
|
||||
const settingsStoragePath = root + '/lib/services/storage/settingsStorage.js';
|
||||
const brokerPath = root + '/lib/services/sse/sse-broker.js';
|
||||
const utilsPath = root + '/lib/utils.js';
|
||||
const loggerPath = root + '/lib/services/logger.js';
|
||||
@@ -33,6 +34,9 @@ describe('services/jobs/jobExecutionService', () => {
|
||||
getUsers: () => state.users.slice(),
|
||||
getUser: (id) => state.users.find((u) => u.id === id) || null,
|
||||
}));
|
||||
vi.doMock(settingsStoragePath, () => ({
|
||||
getSettings: async () => ({}),
|
||||
}));
|
||||
vi.doMock(brokerPath, () => ({
|
||||
sendToUsers: (...args) => calls.sent.push(args),
|
||||
}));
|
||||
|
||||
193
test/storage/listingStatus.test.js
Normal file
193
test/storage/listingStatus.test.js
Normal file
@@ -0,0 +1,193 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { vi, describe, it, expect, beforeEach } from 'vitest';
|
||||
|
||||
// We mock SqliteConnection so we can assert which SQL the storage layer
|
||||
// runs and with which params, without spinning up a real SQLite DB.
|
||||
|
||||
const calls = {
|
||||
execute: [],
|
||||
query: [],
|
||||
};
|
||||
|
||||
const sqliteMock = {
|
||||
execute: (sql, params) => {
|
||||
calls.execute.push({ sql, params });
|
||||
// Default: pretend 1 row was affected (so setListingStatus reports success).
|
||||
return { changes: 1 };
|
||||
},
|
||||
query: (sql, params) => {
|
||||
calls.query.push({ sql, params });
|
||||
// Return shape varies by test — overridden via queryHandler when needed.
|
||||
if (sqliteMock.__queryHandler) return sqliteMock.__queryHandler(sql, params);
|
||||
return [];
|
||||
},
|
||||
__queryHandler: null,
|
||||
};
|
||||
|
||||
vi.mock('../../lib/services/storage/SqliteConnection.js', () => ({
|
||||
default: sqliteMock,
|
||||
}));
|
||||
|
||||
describe('listingsStorage.setListingStatus', () => {
|
||||
let listingsStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
sqliteMock.__queryHandler = null;
|
||||
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
|
||||
});
|
||||
|
||||
it('runs an UPDATE storing a JSON payload with status and setAt', () => {
|
||||
const before = Date.now();
|
||||
const changes = listingsStorage.setListingStatus('listing-1', 'Applied');
|
||||
const after = Date.now();
|
||||
expect(changes).toBe(1);
|
||||
expect(calls.execute).toHaveLength(1);
|
||||
expect(calls.execute[0].sql).toMatch(/UPDATE listings SET status = @status WHERE id = @id/);
|
||||
expect(calls.execute[0].params.id).toBe('listing-1');
|
||||
const parsed = JSON.parse(calls.execute[0].params.status);
|
||||
expect(parsed.status).toBe('applied');
|
||||
expect(parsed.setAt).toBeGreaterThanOrEqual(before);
|
||||
expect(parsed.setAt).toBeLessThanOrEqual(after);
|
||||
});
|
||||
|
||||
it('accepts null to clear the status (no JSON wrapping)', () => {
|
||||
listingsStorage.setListingStatus('listing-2', null);
|
||||
expect(calls.execute[0].params).toEqual({ id: 'listing-2', status: null });
|
||||
});
|
||||
|
||||
it('rejects invalid statuses', () => {
|
||||
expect(() => listingsStorage.setListingStatus('listing-3', 'maybe')).toThrow(/Invalid listing status/);
|
||||
expect(calls.execute).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('returns 0 when no id is supplied (no SQL is run)', () => {
|
||||
const result = listingsStorage.setListingStatus(null, 'applied');
|
||||
expect(result).toBe(0);
|
||||
expect(calls.execute).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('listingsStorage.queryListings statusFilter', () => {
|
||||
let listingsStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
// Return empty rows for both the count and the page-fetch queries.
|
||||
sqliteMock.__queryHandler = (sql) => {
|
||||
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 0 }];
|
||||
return [];
|
||||
};
|
||||
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
|
||||
});
|
||||
|
||||
it("adds 'l.status IS NULL' to WHERE when statusFilter is 'none'", () => {
|
||||
listingsStorage.queryListings({ statusFilter: 'none', userId: 'u1', isAdmin: true });
|
||||
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
|
||||
expect(pageQuery.sql).toMatch(/\(l\.status IS NULL\)/);
|
||||
});
|
||||
|
||||
it('extracts the inner status field via json_extract for a concrete status', () => {
|
||||
listingsStorage.queryListings({ statusFilter: 'applied', userId: 'u1', isAdmin: true });
|
||||
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
|
||||
expect(pageQuery.sql).toMatch(/json_extract\(l\.status, '\$\.status'\) = @statusValue/);
|
||||
expect(pageQuery.params.statusValue).toBe('applied');
|
||||
});
|
||||
|
||||
it('ignores unknown statusFilter values silently', () => {
|
||||
listingsStorage.queryListings({ statusFilter: 'bogus', userId: 'u1', isAdmin: true });
|
||||
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
|
||||
expect(pageQuery.sql).not.toMatch(/status/i);
|
||||
});
|
||||
|
||||
it('parses the JSON status payload of returned rows into an object', () => {
|
||||
sqliteMock.__queryHandler = (sql) => {
|
||||
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 2 }];
|
||||
return [
|
||||
{ id: 'a', status: JSON.stringify({ status: 'applied', setAt: 1700000000000 }) },
|
||||
{ id: 'b', status: null },
|
||||
];
|
||||
};
|
||||
const result = listingsStorage.queryListings({ userId: 'u1', isAdmin: true });
|
||||
expect(result.result[0].status).toEqual({ status: 'applied', setAt: 1700000000000 });
|
||||
expect(result.result[1].status).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('listingsStorage.getListingById', () => {
|
||||
let listingsStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
|
||||
});
|
||||
|
||||
it('parses the JSON status payload of the returned row', () => {
|
||||
sqliteMock.__queryHandler = () => [
|
||||
{ id: 'a', status: JSON.stringify({ status: 'rejected', setAt: 1700000000001 }) },
|
||||
];
|
||||
const row = listingsStorage.getListingById('a', 'u1', true);
|
||||
expect(row.status).toEqual({ status: 'rejected', setAt: 1700000000001 });
|
||||
});
|
||||
|
||||
it('returns null status untouched', () => {
|
||||
sqliteMock.__queryHandler = () => [{ id: 'a', status: null }];
|
||||
const row = listingsStorage.getListingById('a', 'u1', true);
|
||||
expect(row.status).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null when no row is found', () => {
|
||||
sqliteMock.__queryHandler = () => [];
|
||||
const row = listingsStorage.getListingById('missing', 'u1', true);
|
||||
expect(row).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('watchListStorage.ensureWatch', () => {
|
||||
let watchListStorage;
|
||||
|
||||
beforeEach(async () => {
|
||||
calls.execute.length = 0;
|
||||
calls.query.length = 0;
|
||||
sqliteMock.__queryHandler = null;
|
||||
watchListStorage = await import('../../lib/services/storage/watchListStorage.js');
|
||||
});
|
||||
|
||||
it('inserts and reports watched=true on first call', () => {
|
||||
// After INSERT, createWatch queries for existence and gets a row back.
|
||||
sqliteMock.__queryHandler = () => [{ ok: 1 }];
|
||||
const result = watchListStorage.ensureWatch('listing-1', 'user-1');
|
||||
expect(result).toEqual({ watched: true });
|
||||
// INSERT should have been issued.
|
||||
expect(calls.execute.some((c) => /INSERT INTO watch_list/.test(c.sql))).toBe(true);
|
||||
});
|
||||
|
||||
it('returns watched=true when an entry already exists', () => {
|
||||
// Simulate ON CONFLICT being a no-op: execute reports no changes, then SELECT confirms row exists.
|
||||
sqliteMock.execute = (sql, params) => {
|
||||
calls.execute.push({ sql, params });
|
||||
return { changes: 0 };
|
||||
};
|
||||
sqliteMock.__queryHandler = () => [{ ok: 1 }];
|
||||
const result = watchListStorage.ensureWatch('listing-2', 'user-2');
|
||||
expect(result).toEqual({ watched: true });
|
||||
// Restore execute to default for subsequent tests.
|
||||
sqliteMock.execute = (sql, params) => {
|
||||
calls.execute.push({ sql, params });
|
||||
return { changes: 1 };
|
||||
};
|
||||
});
|
||||
|
||||
it('returns watched=false when listingId or userId is missing', () => {
|
||||
expect(watchListStorage.ensureWatch(null, 'u')).toEqual({ watched: false });
|
||||
expect(watchListStorage.ensureWatch('l', null)).toEqual({ watched: false });
|
||||
expect(calls.execute).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
@@ -155,6 +155,7 @@ const routes = {
|
||||
'GET /api/dashboard': dashboard,
|
||||
'GET /api/demo': { demoMode: false },
|
||||
'POST /api/user/settings/news-hash': {},
|
||||
'POST /api/user/settings/listing-deletion-preference': {},
|
||||
};
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
|
||||
@@ -95,7 +95,10 @@ async function downloadHtmlProvider(name, providerConfig, launchBrowser, closeBr
|
||||
|
||||
const browser = await launchBrowser(providerConfig.url, {});
|
||||
try {
|
||||
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, { browser });
|
||||
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, {
|
||||
browser,
|
||||
name: 'dowload_fixtures',
|
||||
});
|
||||
|
||||
if (!html) {
|
||||
console.warn(` Failed to download ${name}`);
|
||||
|
||||
@@ -97,6 +97,7 @@ export default function FredyApp() {
|
||||
<Route path="/dashboard" element={<Dashboard />} />
|
||||
<Route path="/jobs" element={<Jobs />} />
|
||||
<Route path="/listings" element={<Listings />} />
|
||||
<Route path="/listings/watchlist" element={<Listings mode="watchlist" />} />
|
||||
<Route path="/listings/listing/:listingId" element={<ListingDetail />} />
|
||||
<Route path="/map" element={<MapView />} />
|
||||
<Route path="/watchlistManagement" element={<WatchlistManagement />} />
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { useState } from 'react';
|
||||
import { Modal, Radio, RadioGroup, Typography } from '@douyinfe/semi-ui-19';
|
||||
import { useState, useEffect } from 'react';
|
||||
import { Modal, Radio, RadioGroup, Typography, Checkbox } from '@douyinfe/semi-ui-19';
|
||||
|
||||
const { Text } = Typography;
|
||||
|
||||
@@ -15,11 +15,24 @@ const ListingDeletionModal = ({
|
||||
title = 'Delete Listings',
|
||||
showOptions = true,
|
||||
message = 'How would you like to delete the selected listing(s)?',
|
||||
defaultDeleteType = 'soft',
|
||||
}) => {
|
||||
const [deleteType, setDeleteType] = useState('soft');
|
||||
const [remember, setRemember] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
if (visible) {
|
||||
setDeleteType(defaultDeleteType);
|
||||
setRemember(false);
|
||||
}
|
||||
}, [visible, defaultDeleteType]);
|
||||
|
||||
const handleOk = () => {
|
||||
onConfirm(!showOptions || deleteType === 'hard');
|
||||
if (showOptions) {
|
||||
onConfirm(deleteType === 'hard', remember);
|
||||
} else {
|
||||
onConfirm(true);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
@@ -36,32 +49,37 @@ const ListingDeletionModal = ({
|
||||
<Text>{message}</Text>
|
||||
</div>
|
||||
{showOptions && (
|
||||
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
|
||||
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Mark as deleted (Soft Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
|
||||
scraping session.
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Remove from database (Hard Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are completely removed from the database.
|
||||
<>
|
||||
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
|
||||
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Mark as deleted (Soft Delete)</Text>
|
||||
<br />
|
||||
<Text type="warning">
|
||||
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
|
||||
previously found.
|
||||
<Text type="secondary">
|
||||
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
|
||||
scraping session.
|
||||
</Text>
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
</RadioGroup>
|
||||
</div>
|
||||
</Radio>
|
||||
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
|
||||
<div style={{ marginLeft: 8 }}>
|
||||
<Text strong>Remove from database (Hard Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are completely removed from the database.
|
||||
<br />
|
||||
<Text type="warning">
|
||||
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
|
||||
previously found.
|
||||
</Text>
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
</RadioGroup>
|
||||
<Checkbox checked={remember} onChange={(e) => setRemember(e.target.checked)} style={{ marginTop: 16 }}>
|
||||
Remember my choice and skip this dialog next time
|
||||
</Checkbox>
|
||||
</>
|
||||
)}
|
||||
</Modal>
|
||||
);
|
||||
|
||||
@@ -60,6 +60,8 @@ const JobGrid = () => {
|
||||
|
||||
const userSettings = useSelector((state) => state.userSettings.settings);
|
||||
const viewMode = userSettings?.jobs_view_mode ?? 'grid';
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
|
||||
const [page, setPage] = useState(1);
|
||||
const pageSize = 12;
|
||||
@@ -142,13 +144,21 @@ const JobGrid = () => {
|
||||
};
|
||||
|
||||
const onListingRemoval = (jobId) => {
|
||||
setPendingDeletion({ type: 'listings', jobId });
|
||||
const deletion = { type: 'listings', jobId };
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmDeletion(listingDeletionPref.hardDelete, false, deletion);
|
||||
return;
|
||||
}
|
||||
setPendingDeletion(deletion);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
|
||||
const confirmDeletion = async (hardDelete) => {
|
||||
const { type, jobId } = pendingDeletion;
|
||||
const confirmDeletion = async (hardDelete, remember, deletion = pendingDeletion) => {
|
||||
const { type, jobId } = deletion;
|
||||
try {
|
||||
if (remember && type === 'listings') {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
if (type === 'job') {
|
||||
await xhrDelete('/api/jobs', { jobId });
|
||||
Toast.success('Job and listings successfully removed');
|
||||
@@ -174,7 +184,7 @@ const JobGrid = () => {
|
||||
Toast.success('Job status successfully changed');
|
||||
loadData();
|
||||
} catch (error) {
|
||||
Toast.error(error);
|
||||
Toast.error(error.error);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -425,6 +435,7 @@ const JobGrid = () => {
|
||||
visible={deleteModalVisible}
|
||||
title={pendingDeletion?.type === 'job' ? 'Delete Job' : 'Delete Listings'}
|
||||
showOptions={pendingDeletion?.type !== 'job'}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
message={
|
||||
pendingDeletion?.type === 'job'
|
||||
? 'Are you sure you want to delete this job? All associated listings will be removed from the database.'
|
||||
|
||||
@@ -16,13 +16,14 @@ import {
|
||||
} from '@douyinfe/semi-icons';
|
||||
import no_image from '../../../assets/no_image.png';
|
||||
import * as timeService from '../../../services/time/timeService.js';
|
||||
import StatusControl from '../../listings/StatusControl.jsx';
|
||||
|
||||
import './ListingsGrid.less';
|
||||
|
||||
/**
|
||||
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function }} props
|
||||
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function, onStatusChange: Function }} props
|
||||
*/
|
||||
const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete }) => (
|
||||
const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete, onStatusChange }) => (
|
||||
<div className="listingsGrid__grid">
|
||||
{listings.map((item) => (
|
||||
<div
|
||||
@@ -49,14 +50,16 @@ const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete }) => (
|
||||
<span>Inactive</span>
|
||||
</div>
|
||||
)}
|
||||
<button
|
||||
type="button"
|
||||
className="listingsGrid__card__star"
|
||||
onClick={(e) => onWatch(e, item)}
|
||||
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
|
||||
>
|
||||
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
|
||||
</button>
|
||||
<Tooltip content={item.isWatched === 1 ? 'Remove from Watchlist' : 'Add to Watchlist'}>
|
||||
<button
|
||||
type="button"
|
||||
className="listingsGrid__card__star"
|
||||
onClick={(e) => onWatch(e, item)}
|
||||
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
|
||||
>
|
||||
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
|
||||
</button>
|
||||
</Tooltip>
|
||||
</div>
|
||||
|
||||
<div className="listingsGrid__card__body">
|
||||
@@ -83,6 +86,12 @@ const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete }) => (
|
||||
</div>
|
||||
|
||||
<div className="listingsGrid__card__actions" onClick={(e) => e.stopPropagation()}>
|
||||
<StatusControl
|
||||
status={item.status?.status ?? null}
|
||||
compact
|
||||
onChange={(next) => onStatusChange?.(item, next)}
|
||||
onTriggerClick={(e) => e.stopPropagation()}
|
||||
/>
|
||||
<Tooltip content="Original Listing">
|
||||
<Button
|
||||
size="small"
|
||||
|
||||
@@ -11,12 +11,11 @@
|
||||
border: 1px solid @color-border !important;
|
||||
border-radius: @radius-card !important;
|
||||
overflow: hidden;
|
||||
transition: transform @transition-card, box-shadow @transition-card;
|
||||
transition: box-shadow @transition-card;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
|
||||
&:hover {
|
||||
transform: translateY(-2px);
|
||||
box-shadow: 0 6px 24px -4px rgba(0,0,0,0.6);
|
||||
}
|
||||
|
||||
|
||||
@@ -23,7 +23,8 @@ import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-i
|
||||
|
||||
import './ListingsOverview.less';
|
||||
|
||||
const ListingsOverview = () => {
|
||||
const ListingsOverview = ({ mode = 'all' }) => {
|
||||
const isWatchlistMode = mode === 'watchlist';
|
||||
const listingsData = useSelector((state) => state.listingsData);
|
||||
const providers = useSelector((state) => state.provider);
|
||||
const jobs = useSelector((state) => state.jobsData.jobs);
|
||||
@@ -33,6 +34,8 @@ const ListingsOverview = () => {
|
||||
const sp = useSearchParams();
|
||||
|
||||
const viewMode = userSettings?.listings_view_mode ?? 'grid';
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
|
||||
const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
|
||||
const pageSize = 40;
|
||||
@@ -44,9 +47,13 @@ const ListingsOverview = () => {
|
||||
const [jobNameFilter, setJobNameFilter] = useSearchParamState(sp, 'job', null, parseString);
|
||||
const [activityFilter, setActivityFilter] = useSearchParamState(sp, 'active', null, parseNullableBoolean);
|
||||
const [providerFilter, setProviderFilter] = useSearchParamState(sp, 'provider', null, parseString);
|
||||
const [statusFilter, setStatusFilter] = useSearchParamState(sp, 'status', null, parseString);
|
||||
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
|
||||
const [listingToDelete, setListingToDelete] = useState(null);
|
||||
|
||||
// In watchlist mode the watch filter is forced to "watched only" — regardless of the URL.
|
||||
const effectiveWatchListFilter = isWatchlistMode ? true : watchListFilter;
|
||||
|
||||
const loadData = () => {
|
||||
actions.listingsData.getListingsData({
|
||||
page,
|
||||
@@ -54,13 +61,30 @@ const ListingsOverview = () => {
|
||||
sortfield: sortField,
|
||||
sortdir: sortDir,
|
||||
freeTextFilter,
|
||||
filter: { watchListFilter, jobNameFilter, activityFilter, providerFilter },
|
||||
filter: {
|
||||
watchListFilter: effectiveWatchListFilter,
|
||||
jobNameFilter,
|
||||
activityFilter,
|
||||
providerFilter,
|
||||
statusFilter,
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
loadData();
|
||||
}, [page, sortField, sortDir, freeTextFilter, providerFilter, activityFilter, jobNameFilter, watchListFilter]);
|
||||
}, [
|
||||
page,
|
||||
sortField,
|
||||
sortDir,
|
||||
freeTextFilter,
|
||||
providerFilter,
|
||||
activityFilter,
|
||||
jobNameFilter,
|
||||
watchListFilter,
|
||||
statusFilter,
|
||||
isWatchlistMode,
|
||||
]);
|
||||
|
||||
const handleFilterChange = useMemo(
|
||||
() =>
|
||||
@@ -90,16 +114,34 @@ const ListingsOverview = () => {
|
||||
}
|
||||
};
|
||||
|
||||
const handleStatusChange = async (item, nextStatus) => {
|
||||
try {
|
||||
await actions.listingsData.setListingStatus(item.id, nextStatus);
|
||||
Toast.success(nextStatus ? `Marked as ${nextStatus}` : 'Status cleared');
|
||||
loadData();
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
Toast.error('Failed to update status');
|
||||
}
|
||||
};
|
||||
|
||||
const handleDelete = (id) => {
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmDeletion(listingDeletionPref.hardDelete, false, id);
|
||||
return;
|
||||
}
|
||||
setListingToDelete(id);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
|
||||
const handleNavigate = (id) => navigate(`/listings/listing/${id}`);
|
||||
|
||||
const confirmDeletion = async (hardDelete) => {
|
||||
const confirmDeletion = async (hardDelete, remember, id = listingToDelete) => {
|
||||
try {
|
||||
await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
|
||||
if (remember) {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
await xhrDelete('/api/listings/', { ids: [id], hardDelete });
|
||||
Toast.success('Listing successfully removed');
|
||||
loadData();
|
||||
} catch (error) {
|
||||
@@ -139,20 +181,38 @@ const ListingsOverview = () => {
|
||||
<Radio value="false">Inactive</Radio>
|
||||
</RadioGroup>
|
||||
|
||||
<RadioGroup
|
||||
type="button"
|
||||
buttonSize="middle"
|
||||
value={watchListFilter === null ? 'all' : String(watchListFilter)}
|
||||
onChange={(e) => {
|
||||
const v = e.target.value;
|
||||
setWatchListFilter(v === 'all' ? null : v === 'true');
|
||||
{!isWatchlistMode && (
|
||||
<RadioGroup
|
||||
type="button"
|
||||
buttonSize="middle"
|
||||
value={watchListFilter === null ? 'all' : String(watchListFilter)}
|
||||
onChange={(e) => {
|
||||
const v = e.target.value;
|
||||
setWatchListFilter(v === 'all' ? null : v === 'true');
|
||||
setPage(1);
|
||||
}}
|
||||
>
|
||||
<Radio value="all">All</Radio>
|
||||
<Radio value="true">Watched</Radio>
|
||||
<Radio value="false">Unwatched</Radio>
|
||||
</RadioGroup>
|
||||
)}
|
||||
|
||||
<Select
|
||||
placeholder="Status"
|
||||
showClear
|
||||
onChange={(val) => {
|
||||
setStatusFilter(val ?? null);
|
||||
setPage(1);
|
||||
}}
|
||||
value={statusFilter}
|
||||
style={{ width: 150 }}
|
||||
>
|
||||
<Radio value="all">All</Radio>
|
||||
<Radio value="true">Watched</Radio>
|
||||
<Radio value="false">Unwatched</Radio>
|
||||
</RadioGroup>
|
||||
<Select.Option value="applied">Applied</Select.Option>
|
||||
<Select.Option value="rejected">Rejected</Select.Option>
|
||||
<Select.Option value="accepted">Accepted</Select.Option>
|
||||
<Select.Option value="none">No status</Select.Option>
|
||||
</Select>
|
||||
|
||||
<Select
|
||||
placeholder="Provider"
|
||||
@@ -188,7 +248,13 @@ const ListingsOverview = () => {
|
||||
))}
|
||||
</Select>
|
||||
|
||||
<Select prefix="Sort by" style={{ width: 185 }} value={sortField} onChange={(val) => setSortField(val)}>
|
||||
<Select
|
||||
prefix="Sort by"
|
||||
className="listingsOverview__topbar__sort"
|
||||
style={{ width: 220 }}
|
||||
value={sortField}
|
||||
onChange={(val) => setSortField(val)}
|
||||
>
|
||||
<Select.Option value="job_name">Job Name</Select.Option>
|
||||
<Select.Option value="created_at">Listing Date</Select.Option>
|
||||
<Select.Option value="price">Price</Select.Option>
|
||||
@@ -232,9 +298,21 @@ const ListingsOverview = () => {
|
||||
)}
|
||||
|
||||
{viewMode === 'grid' ? (
|
||||
<ListingsGrid listings={listings} onWatch={handleWatch} onNavigate={handleNavigate} onDelete={handleDelete} />
|
||||
<ListingsGrid
|
||||
listings={listings}
|
||||
onWatch={handleWatch}
|
||||
onNavigate={handleNavigate}
|
||||
onDelete={handleDelete}
|
||||
onStatusChange={handleStatusChange}
|
||||
/>
|
||||
) : (
|
||||
<ListingsTable listings={listings} onWatch={handleWatch} onNavigate={handleNavigate} onDelete={handleDelete} />
|
||||
<ListingsTable
|
||||
listings={listings}
|
||||
onWatch={handleWatch}
|
||||
onNavigate={handleNavigate}
|
||||
onDelete={handleDelete}
|
||||
onStatusChange={handleStatusChange}
|
||||
/>
|
||||
)}
|
||||
|
||||
{listings.length > 0 && (
|
||||
@@ -251,6 +329,7 @@ const ListingsOverview = () => {
|
||||
|
||||
<ListingDeletionModal
|
||||
visible={deleteModalVisible}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
onConfirm={confirmDeletion}
|
||||
onCancel={() => {
|
||||
setDeleteModalVisible(false);
|
||||
|
||||
@@ -19,6 +19,14 @@
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
&__sort {
|
||||
flex-shrink: 0;
|
||||
|
||||
.semi-select-prefix {
|
||||
white-space: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.listingsOverview__topbar__search {
|
||||
width: 100%;
|
||||
|
||||
115
ui/src/components/listings/StatusControl.jsx
Normal file
115
ui/src/components/listings/StatusControl.jsx
Normal file
@@ -0,0 +1,115 @@
|
||||
/*
|
||||
* Copyright (c) 2026 by Christian Kellner.
|
||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||
*/
|
||||
|
||||
import { useState } from 'react';
|
||||
import { Dropdown, Button, Tooltip } from '@douyinfe/semi-ui-19';
|
||||
import { IconChevronDown } from '@douyinfe/semi-icons';
|
||||
|
||||
import './StatusControl.less';
|
||||
|
||||
const STATUS_TOOLTIP =
|
||||
'Track where you stand with this listing: Applied once you have reached out, Rejected if it did not work out, or Accepted if you got it.';
|
||||
|
||||
/**
|
||||
* @typedef {('applied'|'rejected'|'accepted'|null)} ListingStatus
|
||||
*/
|
||||
|
||||
const STATUS_OPTIONS = [
|
||||
{ value: null, label: 'None' },
|
||||
{ value: 'applied', label: 'Applied' },
|
||||
{ value: 'rejected', label: 'Rejected' },
|
||||
{ value: 'accepted', label: 'Accepted' },
|
||||
];
|
||||
|
||||
/**
|
||||
* Look up the option metadata for a status value.
|
||||
* @param {ListingStatus} status
|
||||
*/
|
||||
const optionFor = (status) => STATUS_OPTIONS.find((o) => o.value === status) ?? STATUS_OPTIONS[0];
|
||||
|
||||
/**
|
||||
* Shared control for setting a listing's user-decision status
|
||||
* (Applied / Rejected / Accepted).
|
||||
*
|
||||
* Both compact (table/grid rows) and full (listing detail header) modes
|
||||
* render a Button that picks up the project's CI tokens via the
|
||||
* .status-btn classes, with a small size variant for compact contexts.
|
||||
*
|
||||
* @param {Object} props
|
||||
* @param {ListingStatus} props.status - The current status value.
|
||||
* @param {(next: ListingStatus) => void} props.onChange - Called with the new status when the user picks one.
|
||||
* @param {boolean} [props.compact=false] - When true, renders smaller for table/grid rows; full size otherwise.
|
||||
* @param {(e: React.MouseEvent) => void} [props.onTriggerClick] - Optional click handler to stop propagation on the trigger.
|
||||
*/
|
||||
export default function StatusControl({ status = null, onChange, compact = false, onTriggerClick }) {
|
||||
const [open, setOpen] = useState(false);
|
||||
const [tooltipOpen, setTooltipOpen] = useState(false);
|
||||
const current = optionFor(status);
|
||||
|
||||
const handlePick = (next) => {
|
||||
setOpen(false);
|
||||
if (next === status) return;
|
||||
onChange?.(next);
|
||||
};
|
||||
|
||||
const menu = (
|
||||
<Dropdown.Menu>
|
||||
{STATUS_OPTIONS.map((opt) => (
|
||||
<Dropdown.Item
|
||||
key={opt.value ?? '__none__'}
|
||||
active={opt.value === status}
|
||||
onClick={() => handlePick(opt.value)}
|
||||
>
|
||||
{opt.label}
|
||||
</Dropdown.Item>
|
||||
))}
|
||||
</Dropdown.Menu>
|
||||
);
|
||||
|
||||
const className = ['status-btn', compact ? 'status-btn--compact' : null, status ? `status-btn--${status}` : null]
|
||||
.filter(Boolean)
|
||||
.join(' ');
|
||||
|
||||
const trigger = (
|
||||
<Tooltip
|
||||
content={STATUS_TOOLTIP}
|
||||
position="top"
|
||||
trigger="custom"
|
||||
visible={tooltipOpen && !open}
|
||||
onVisibleChange={setTooltipOpen}
|
||||
>
|
||||
<Button
|
||||
size={compact ? 'small' : 'default'}
|
||||
theme="borderless"
|
||||
icon={<IconChevronDown />}
|
||||
iconPosition="right"
|
||||
onMouseEnter={() => setTooltipOpen(true)}
|
||||
onMouseLeave={() => setTooltipOpen(false)}
|
||||
onClick={(e) => {
|
||||
onTriggerClick?.(e);
|
||||
setTooltipOpen(false);
|
||||
setOpen((o) => !o);
|
||||
}}
|
||||
className={className}
|
||||
>
|
||||
{status ? current.label : 'Status'}
|
||||
</Button>
|
||||
</Tooltip>
|
||||
);
|
||||
|
||||
return (
|
||||
<Dropdown
|
||||
trigger="custom"
|
||||
visible={open}
|
||||
onVisibleChange={setOpen}
|
||||
onClickOutSide={() => setOpen(false)}
|
||||
position="bottom"
|
||||
render={menu}
|
||||
stopPropagation
|
||||
>
|
||||
<span className="status-btn__anchor">{trigger}</span>
|
||||
</Dropdown>
|
||||
);
|
||||
}
|
||||
64
ui/src/components/listings/StatusControl.less
Normal file
64
ui/src/components/listings/StatusControl.less
Normal file
@@ -0,0 +1,64 @@
|
||||
@import '../../tokens.less';
|
||||
|
||||
// Wrapper span used as the Dropdown's positioning anchor so the menu opens
|
||||
// directly below the visible button rather than the implicit wrapper of the
|
||||
// hover tooltip (which can have a different bounding box).
|
||||
.status-btn__anchor {
|
||||
display: inline-block;
|
||||
line-height: 0;
|
||||
}
|
||||
|
||||
// StatusControl shared base. Matches dimensions and border treatment
|
||||
// of the surrounding Watched / Open listing / Delete buttons in the
|
||||
// detail view, and shrinks via the --compact modifier for table rows
|
||||
// and grid cards.
|
||||
.status-btn {
|
||||
color: @color-muted !important;
|
||||
border: 1px solid @color-border-bright !important;
|
||||
border-radius: @radius-btn !important;
|
||||
background: transparent !important;
|
||||
transition: color @transition-fast, border-color @transition-fast, background @transition-fast;
|
||||
|
||||
&:hover {
|
||||
color: @color-text !important;
|
||||
background: rgba(255, 255, 255, 0.06) !important;
|
||||
}
|
||||
|
||||
&--compact {
|
||||
height: 24px !important;
|
||||
padding: 0 8px !important;
|
||||
font-size: @text-sm !important;
|
||||
border-radius: @radius-chip !important;
|
||||
|
||||
.semi-icon {
|
||||
font-size: 12px !important;
|
||||
}
|
||||
}
|
||||
|
||||
&--applied {
|
||||
color: @color-info !important;
|
||||
border-color: rgba(96, 165, 250, 0.4) !important;
|
||||
background: rgba(96, 165, 250, 0.08) !important;
|
||||
&:hover {
|
||||
background: rgba(96, 165, 250, 0.14) !important;
|
||||
}
|
||||
}
|
||||
|
||||
&--rejected {
|
||||
color: @color-error !important;
|
||||
border-color: rgba(251, 113, 133, 0.4) !important;
|
||||
background: rgba(251, 113, 133, 0.08) !important;
|
||||
&:hover {
|
||||
background: rgba(251, 113, 133, 0.14) !important;
|
||||
}
|
||||
}
|
||||
|
||||
&--accepted {
|
||||
color: @color-success !important;
|
||||
border-color: rgba(52, 211, 153, 0.4) !important;
|
||||
background: rgba(52, 211, 153, 0.08) !important;
|
||||
&:hover {
|
||||
background: rgba(52, 211, 153, 0.14) !important;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import maplibregl from 'maplibre-gl';
|
||||
import 'maplibre-gl/dist/maplibre-gl.css';
|
||||
import '@mapbox/mapbox-gl-draw/dist/mapbox-gl-draw.css';
|
||||
import { fixMapboxDrawCompatibility, addDrawingControl, setupAreaFilterEventListeners } from './MapDrawingExtension.js';
|
||||
import { getBoundsFromCoords } from '../../views/listings/mapUtils.js';
|
||||
import './Map.less';
|
||||
|
||||
export const GERMANY_BOUNDS = [
|
||||
@@ -66,6 +67,7 @@ export default function Map({
|
||||
const mapContainerRef = useRef(null);
|
||||
const mapRef = useRef(null);
|
||||
const drawRef = useRef(null);
|
||||
const hasFittedToInitialAreaRef = useRef(false);
|
||||
|
||||
// Initialize map - ONLY when container changes, never reinitialize
|
||||
useEffect(() => {
|
||||
@@ -128,6 +130,17 @@ export default function Map({
|
||||
} catch (error) {
|
||||
console.error('Error loading spatial filter:', error);
|
||||
}
|
||||
|
||||
if (!hasFittedToInitialAreaRef.current) {
|
||||
const coords = initialSpatialFilter.features.flatMap((feature) =>
|
||||
feature.geometry?.type === 'Polygon' ? feature.geometry.coordinates.flat() : [],
|
||||
);
|
||||
const bounds = getBoundsFromCoords(coords);
|
||||
if (bounds) {
|
||||
mapRef.current.fitBounds(bounds, { padding: 50, maxZoom: 15, duration: 0 });
|
||||
hasFittedToInitialAreaRef.current = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Setup drawing event listeners
|
||||
|
||||
@@ -37,6 +37,7 @@ export default function Navigation({ isAdmin }) {
|
||||
items: [
|
||||
{ itemKey: '/listings', text: 'Overview' },
|
||||
{ itemKey: '/map', text: 'Map View' },
|
||||
{ itemKey: '/listings/watchlist', text: 'Watchlist' },
|
||||
],
|
||||
},
|
||||
];
|
||||
@@ -61,6 +62,22 @@ export default function Navigation({ isAdmin }) {
|
||||
}
|
||||
|
||||
function parsePathName(name) {
|
||||
// Collect every leaf itemKey that looks like a route (starts with '/').
|
||||
// Prefer the longest exact-prefix match so nested routes like
|
||||
// '/listings/watchlist' resolve to themselves instead of being collapsed
|
||||
// to '/listings'.
|
||||
const allKeys = [];
|
||||
const collect = (nodes) => {
|
||||
for (const n of nodes) {
|
||||
if (typeof n.itemKey === 'string' && n.itemKey.startsWith('/')) allKeys.push(n.itemKey);
|
||||
if (Array.isArray(n.items)) collect(n.items);
|
||||
}
|
||||
};
|
||||
collect(items);
|
||||
const longestMatch = allKeys
|
||||
.filter((k) => name === k || name.startsWith(k + '/'))
|
||||
.sort((a, b) => b.length - a.length)[0];
|
||||
if (longestMatch) return longestMatch;
|
||||
const split = name.split('/').filter((s) => s.length !== 0);
|
||||
return '/' + split[0];
|
||||
}
|
||||
|
||||
@@ -63,7 +63,7 @@ const JobsTable = ({ jobs, onRun, onEdit, onClone, onDeleteListings, onDeleteJob
|
||||
</Tag>
|
||||
)}
|
||||
{job.isOnlyShared && (
|
||||
<Tooltip content="Shared with you — read only">
|
||||
<Tooltip content="Shared with you - read only">
|
||||
<span style={{ display: 'flex', alignItems: 'center' }}>
|
||||
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
|
||||
</span>
|
||||
|
||||
@@ -16,13 +16,14 @@ import {
|
||||
} from '@douyinfe/semi-icons';
|
||||
import no_image from '../../assets/no_image.png';
|
||||
import * as timeService from '../../services/time/timeService.js';
|
||||
import StatusControl from '../listings/StatusControl.jsx';
|
||||
|
||||
import './ListingsTable.less';
|
||||
|
||||
/**
|
||||
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function }} props
|
||||
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function, onStatusChange: Function }} props
|
||||
*/
|
||||
const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
|
||||
const ListingsTable = ({ listings, onWatch, onNavigate, onDelete, onStatusChange }) => (
|
||||
<div className="listingsTable">
|
||||
{listings.map((item) => (
|
||||
<div
|
||||
@@ -56,7 +57,7 @@ const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
|
||||
{item.price}
|
||||
</>
|
||||
) : (
|
||||
<span className="listingsTable__row__empty">—</span>
|
||||
<span className="listingsTable__row__empty">---</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
@@ -67,7 +68,7 @@ const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
|
||||
{item.address}
|
||||
</>
|
||||
) : (
|
||||
<span className="listingsTable__row__empty">—</span>
|
||||
<span className="listingsTable__row__empty">---</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
@@ -79,14 +80,22 @@ const ListingsTable = ({ listings, onWatch, onNavigate, onDelete }) => (
|
||||
<div className="listingsTable__row__date">{timeService.format(item.created_at, false)}</div>
|
||||
|
||||
<div className="listingsTable__row__actions" onClick={(e) => e.stopPropagation()}>
|
||||
<button
|
||||
type="button"
|
||||
className="listingsTable__row__star"
|
||||
onClick={(e) => onWatch(e, item)}
|
||||
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
|
||||
>
|
||||
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
|
||||
</button>
|
||||
<StatusControl
|
||||
status={item.status?.status ?? null}
|
||||
compact
|
||||
onChange={(next) => onStatusChange?.(item, next)}
|
||||
onTriggerClick={(e) => e.stopPropagation()}
|
||||
/>
|
||||
<Tooltip content={item.isWatched === 1 ? 'Remove from Watchlist' : 'Add to Watchlist'}>
|
||||
<button
|
||||
type="button"
|
||||
className="listingsTable__row__star"
|
||||
onClick={(e) => onWatch(e, item)}
|
||||
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
|
||||
>
|
||||
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
|
||||
</button>
|
||||
</Tooltip>
|
||||
<Tooltip content="Original Listing">
|
||||
<Button
|
||||
size="small"
|
||||
|
||||
@@ -260,6 +260,22 @@ export const useFredyState = create(
|
||||
console.error('Error while trying to get resource for api/listings/map. Error:', Exception);
|
||||
}
|
||||
},
|
||||
async setListingStatus(listingId, status) {
|
||||
try {
|
||||
await xhrPost(`/api/listings/${listingId}/status`, { status });
|
||||
} catch (Exception) {
|
||||
console.error(`Error while trying to set status for listing ${listingId}. Error:`, Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
async setListingNotes(listingId, notes) {
|
||||
try {
|
||||
await xhrPost(`/api/listings/${listingId}/notes`, { notes });
|
||||
} catch (Exception) {
|
||||
console.error(`Error while trying to set notes for listing ${listingId}. Error:`, Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
},
|
||||
userSettings: {
|
||||
async getUserSettings() {
|
||||
@@ -349,6 +365,20 @@ export const useFredyState = create(
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
async setListingDeletionPreference(listing_deletion_preference) {
|
||||
try {
|
||||
await xhrPost('/api/user/settings/listing-deletion-preference', { listing_deletion_preference });
|
||||
set((state) => ({
|
||||
userSettings: {
|
||||
...state.userSettings,
|
||||
settings: { ...state.userSettings.settings, listing_deletion_preference },
|
||||
},
|
||||
}));
|
||||
} catch (Exception) {
|
||||
console.error('Error while trying to update listing deletion preference. Error:', Exception);
|
||||
throw Exception;
|
||||
}
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
|
||||
@@ -18,6 +18,9 @@ import {
|
||||
AutoComplete,
|
||||
Select,
|
||||
Banner,
|
||||
Radio,
|
||||
RadioGroup,
|
||||
Typography,
|
||||
} from '@douyinfe/semi-ui-19';
|
||||
import { InputNumber } from '@douyinfe/semi-ui-19';
|
||||
import { xhrPost, xhrGet } from '../../services/xhr';
|
||||
@@ -33,6 +36,8 @@ import { debounce } from '../../utils';
|
||||
import Headline from '../../components/headline/Headline.jsx';
|
||||
import './GeneralSettings.less';
|
||||
|
||||
const { Text } = Typography;
|
||||
|
||||
function formatFromTimestamp(ts) {
|
||||
const date = new Date(ts);
|
||||
return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
|
||||
@@ -57,6 +62,7 @@ const GeneralSettings = function GeneralSettings() {
|
||||
const currentUser = useSelector((state) => state.user.currentUser);
|
||||
|
||||
const [interval, setInterval] = React.useState('');
|
||||
const [proxyUrl, setProxyUrl] = React.useState('');
|
||||
const [port, setPort] = React.useState('');
|
||||
const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
|
||||
const [workingHourTo, setWorkingHourTo] = React.useState(null);
|
||||
@@ -73,9 +79,12 @@ const GeneralSettings = function GeneralSettings() {
|
||||
// User settings state
|
||||
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
|
||||
const providerDetails = useSelector((state) => state.userSettings.settings.provider_details);
|
||||
const listingDeletionPreference = useSelector((state) => state.userSettings.settings.listing_deletion_preference);
|
||||
const allProviders = useSelector((state) => state.provider);
|
||||
const [address, setAddress] = useState(homeAddress?.address || '');
|
||||
const [coords, setCoords] = useState(homeAddress?.coords || null);
|
||||
const [listingDeleteHard, setListingDeleteHard] = useState(false);
|
||||
const [listingDeleteSkipPrompt, setListingDeleteSkipPrompt] = useState(false);
|
||||
const saving = useIsLoading(actions.userSettings.setHomeAddress);
|
||||
const [dataSource, setDataSource] = useState([]);
|
||||
|
||||
@@ -91,6 +100,7 @@ const GeneralSettings = function GeneralSettings() {
|
||||
React.useEffect(() => {
|
||||
async function init() {
|
||||
setInterval(settings?.interval);
|
||||
setProxyUrl(settings?.proxyUrl ?? '');
|
||||
setPort(settings?.port);
|
||||
setWorkingHourFrom(settings?.workingHours?.from);
|
||||
setWorkingHourTo(settings?.workingHours?.to);
|
||||
@@ -108,6 +118,11 @@ const GeneralSettings = function GeneralSettings() {
|
||||
setCoords(homeAddress?.coords || null);
|
||||
}, [homeAddress]);
|
||||
|
||||
useEffect(() => {
|
||||
setListingDeleteHard(listingDeletionPreference?.hardDelete ?? false);
|
||||
setListingDeleteSkipPrompt(listingDeletionPreference?.skipPrompt ?? false);
|
||||
}, [listingDeletionPreference]);
|
||||
|
||||
const nullOrEmpty = (val) => val == null || val.length === 0;
|
||||
|
||||
const handleStore = async () => {
|
||||
@@ -133,6 +148,7 @@ const GeneralSettings = function GeneralSettings() {
|
||||
try {
|
||||
await xhrPost('/api/admin/generalSettings', {
|
||||
interval,
|
||||
proxyUrl: proxyUrl?.trim() ?? '',
|
||||
port,
|
||||
workingHours: {
|
||||
from: workingHourFrom,
|
||||
@@ -215,6 +231,10 @@ const GeneralSettings = function GeneralSettings() {
|
||||
try {
|
||||
const responseJson = await actions.userSettings.setHomeAddress(address);
|
||||
setCoords(responseJson.coords);
|
||||
await actions.userSettings.setListingDeletionPreference({
|
||||
skipPrompt: listingDeleteSkipPrompt,
|
||||
hardDelete: listingDeleteHard,
|
||||
});
|
||||
await actions.userSettings.getUserSettings();
|
||||
Toast.success('Settings saved. Distance calculations are running in the background.');
|
||||
} catch (error) {
|
||||
@@ -376,6 +396,18 @@ const GeneralSettings = function GeneralSettings() {
|
||||
</div>
|
||||
</SegmentPart>
|
||||
|
||||
<SegmentPart
|
||||
name="Proxy URL"
|
||||
helpText="Optional. Routes the scraping browser through a proxy. Server/datacenter IPs are frequently blocked by providers (e.g. immowelt) regardless of browser fingerprint, a German residential proxy makes requests look like a normal household and is the most effective fix. Format: http://user:pass@host:port or socks5://user:pass@host:port. Leave empty to disable."
|
||||
>
|
||||
<Input
|
||||
type="text"
|
||||
placeholder="http://user:pass@host:port"
|
||||
value={proxyUrl}
|
||||
onChange={(value) => setProxyUrl(value)}
|
||||
/>
|
||||
</SegmentPart>
|
||||
|
||||
<div className="generalSettings__save-row">
|
||||
<Button type="primary" theme="solid" onClick={handleStore} icon={<IconSave />}>
|
||||
Save
|
||||
@@ -444,6 +476,48 @@ const GeneralSettings = function GeneralSettings() {
|
||||
/>
|
||||
</SegmentPart>
|
||||
|
||||
<SegmentPart
|
||||
name="Listing deletion"
|
||||
helpText="Choose the default deletion mode. Soft delete hides them without re-scraping; hard delete removes them from the database."
|
||||
>
|
||||
<RadioGroup
|
||||
value={listingDeleteHard ? 'hard' : 'soft'}
|
||||
onChange={(e) => setListingDeleteHard(e.target.value === 'hard')}
|
||||
>
|
||||
<Radio value="soft">
|
||||
<div>
|
||||
<Text strong>Mark as deleted (Soft Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during
|
||||
the next scraping session.
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
<Radio value="hard">
|
||||
<div>
|
||||
<Text strong>Remove from database (Hard Delete)</Text>
|
||||
<br />
|
||||
<Text type="secondary">
|
||||
Listings are completely removed from the database.
|
||||
<br />
|
||||
<Text type="warning">
|
||||
Consequence: They might re-appear when scraping the next time because Fredy won't know they
|
||||
were previously found.
|
||||
</Text>
|
||||
</Text>
|
||||
</div>
|
||||
</Radio>
|
||||
</RadioGroup>
|
||||
<Checkbox
|
||||
checked={listingDeleteSkipPrompt}
|
||||
onChange={(e) => setListingDeleteSkipPrompt(e.target.checked)}
|
||||
style={{ marginTop: 12 }}
|
||||
>
|
||||
Skip confirmation dialog
|
||||
</Checkbox>
|
||||
</SegmentPart>
|
||||
|
||||
<div className="generalSettings__save-row">
|
||||
<Button
|
||||
icon={<IconSave />}
|
||||
|
||||
@@ -73,7 +73,9 @@ export default function NotificationAdapterMutator({
|
||||
const adapter = useSelector((state) => state.notificationAdapter);
|
||||
|
||||
const preFilledSelectedAdapter =
|
||||
editNotificationAdapter == null ? null : adapter.find((a) => a.id === editNotificationAdapter.id);
|
||||
editNotificationAdapter == null
|
||||
? null
|
||||
: adapter.filter((a) => a != null).find((a) => a.id === editNotificationAdapter.id);
|
||||
|
||||
spreadPrefilledAdapterWithValues(preFilledSelectedAdapter, editNotificationAdapter?.fields);
|
||||
|
||||
@@ -227,9 +229,9 @@ export default function NotificationAdapterMutator({
|
||||
className="providerMutator__fields"
|
||||
value={selectedAdapter == null ? '' : selectedAdapter.id}
|
||||
optionList={adapter
|
||||
.filter((a) => a != null)
|
||||
.map((a) => {
|
||||
return {
|
||||
otherKey: a.id,
|
||||
value: a.id,
|
||||
label: a.name,
|
||||
};
|
||||
@@ -238,7 +240,7 @@ export default function NotificationAdapterMutator({
|
||||
.filter((option) =>
|
||||
editNotificationAdapter != null
|
||||
? true
|
||||
: selected.find((selectedOption) => selectedOption.id === option.key) == null,
|
||||
: selected.find((selectedOption) => selectedOption.id === option.value) == null,
|
||||
)
|
||||
.sort(sortAdapter)}
|
||||
onChange={(value) => {
|
||||
|
||||
@@ -20,6 +20,8 @@ import {
|
||||
Banner,
|
||||
Spin,
|
||||
Toast,
|
||||
TextArea,
|
||||
Tooltip,
|
||||
} from '@douyinfe/semi-ui-19';
|
||||
import {
|
||||
IconArrowLeft,
|
||||
@@ -44,6 +46,7 @@ import { xhrPost, xhrDelete } from '../../services/xhr.js';
|
||||
import ListingDeletionModal from '../../components/ListingDeletionModal.jsx';
|
||||
|
||||
import Headline from '../../components/headline/Headline.jsx';
|
||||
import StatusControl from '../../components/listings/StatusControl.jsx';
|
||||
import './ListingDetail.less';
|
||||
|
||||
const { Title, Text } = Typography;
|
||||
@@ -57,11 +60,16 @@ export default function ListingDetail() {
|
||||
const navigate = useNavigate();
|
||||
const actions = useActions();
|
||||
const listing = useSelector((state) => state.listingsData.currentListing);
|
||||
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
|
||||
const userSettings = useSelector((state) => state.userSettings.settings);
|
||||
const homeAddress = userSettings?.home_address;
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
const mapContainer = useRef(null);
|
||||
const map = useRef(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
|
||||
const [notesDraft, setNotesDraft] = useState('');
|
||||
const [notesSaving, setNotesSaving] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
async function fetchListing() {
|
||||
@@ -79,6 +87,10 @@ export default function ListingDetail() {
|
||||
fetchListing();
|
||||
}, [listingId]);
|
||||
|
||||
useEffect(() => {
|
||||
setNotesDraft(listing?.notes ?? '');
|
||||
}, [listing?.id, listing?.notes]);
|
||||
|
||||
const hasGeo =
|
||||
listing?.latitude != null && listing?.longitude != null && listing?.latitude !== -1 && listing?.longitude !== -1;
|
||||
|
||||
@@ -242,8 +254,11 @@ export default function ListingDetail() {
|
||||
};
|
||||
}, [listing, loading, homeAddress]);
|
||||
|
||||
const confirmDeletion = async (hardDelete) => {
|
||||
const confirmDeletion = async (hardDelete, remember) => {
|
||||
try {
|
||||
if (remember) {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
await xhrDelete('/api/listings/', { ids: [listing.id], hardDelete });
|
||||
Toast.success('Listing successfully removed');
|
||||
navigate('/listings');
|
||||
@@ -265,6 +280,32 @@ export default function ListingDetail() {
|
||||
}
|
||||
};
|
||||
|
||||
const handleStatusChange = async (next) => {
|
||||
try {
|
||||
await actions.listingsData.setListingStatus(listing.id, next);
|
||||
await actions.listingsData.getListing(listingId);
|
||||
Toast.success(next ? `Marked as ${next}` : 'Status cleared');
|
||||
} catch (e) {
|
||||
console.error('Failed to update status:', e);
|
||||
Toast.error('Failed to update status');
|
||||
}
|
||||
};
|
||||
|
||||
const handleSaveNotes = async () => {
|
||||
if (!listing) return;
|
||||
setNotesSaving(true);
|
||||
try {
|
||||
await actions.listingsData.setListingNotes(listing.id, notesDraft);
|
||||
await actions.listingsData.getListing(listingId);
|
||||
Toast.success('Notes saved');
|
||||
} catch (e) {
|
||||
console.error('Failed to save notes:', e);
|
||||
Toast.error('Failed to save notes');
|
||||
} finally {
|
||||
setNotesSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
if (loading) {
|
||||
return (
|
||||
<div style={{ display: 'flex', justifyContent: 'center', alignItems: 'center', height: '100%' }}>
|
||||
@@ -275,35 +316,58 @@ export default function ListingDetail() {
|
||||
|
||||
if (!listing) return null;
|
||||
|
||||
const statusLabel = listing.status?.status
|
||||
? listing.status.status.charAt(0).toUpperCase() + listing.status.status.slice(1)
|
||||
: null;
|
||||
|
||||
const data = [
|
||||
{ key: 'Price', value: `${listing.price} €`, Icon: <IconCart /> },
|
||||
{
|
||||
key: 'Price',
|
||||
value: `${listing.price} €`,
|
||||
Icon: <IconCart />,
|
||||
helpText: 'The asking price of this listing, as reported by the provider.',
|
||||
},
|
||||
{
|
||||
key: 'Size',
|
||||
value: listing.size ? `${listing.size} m²` : 'N/A',
|
||||
Icon: <IconExpand />,
|
||||
helpText: 'Living space of the listing in square meters.',
|
||||
},
|
||||
{
|
||||
key: 'Rooms',
|
||||
value: listing.rooms ? `${listing.rooms} Rooms` : 'N/A',
|
||||
Icon: <IconGridView />,
|
||||
helpText: 'Number of rooms in the listing.',
|
||||
},
|
||||
{
|
||||
key: 'Job',
|
||||
value: listing.job_name,
|
||||
Icon: <IconBriefcase />,
|
||||
helpText: 'The Fredy job that found this listing.',
|
||||
},
|
||||
{
|
||||
key: 'Provider',
|
||||
value: listing.provider ? listing.provider.charAt(0).toUpperCase() + listing.provider.slice(1) : 'Unknown',
|
||||
Icon: <IconBriefcase />,
|
||||
helpText: 'The real estate portal where this listing was scraped from.',
|
||||
},
|
||||
{
|
||||
key: 'Added',
|
||||
value: timeService.format(listing.created_at),
|
||||
Icon: <IconClock />,
|
||||
helpText: 'When Fredy first added this listing to your database.',
|
||||
},
|
||||
];
|
||||
|
||||
if (statusLabel) {
|
||||
data.push({
|
||||
key: 'Status',
|
||||
value: listing.status?.setAt ? `${statusLabel} (set ${timeService.format(listing.status.setAt)})` : statusLabel,
|
||||
Icon: <IconActivity />,
|
||||
helpText: 'The status you marked for this listing and when you set it.',
|
||||
});
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="listing-detail">
|
||||
<Headline
|
||||
@@ -341,13 +405,20 @@ export default function ListingDetail() {
|
||||
>
|
||||
{listing.isWatched === 1 ? 'Watched' : 'Watch'}
|
||||
</Button>
|
||||
<StatusControl status={listing.status?.status ?? null} onChange={handleStatusChange} />
|
||||
<a href={listing.link} target="_blank" rel="noopener noreferrer" className="listing-detail__open-btn">
|
||||
<IconLink style={{ marginRight: 6 }} />
|
||||
Open listing
|
||||
</a>
|
||||
<Button
|
||||
icon={<IconDelete />}
|
||||
onClick={() => setDeleteModalVisible(true)}
|
||||
onClick={() => {
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmDeletion(listingDeletionPref.hardDelete);
|
||||
return;
|
||||
}
|
||||
setDeleteModalVisible(true);
|
||||
}}
|
||||
theme="light"
|
||||
type="danger"
|
||||
>
|
||||
@@ -368,6 +439,32 @@ export default function ListingDetail() {
|
||||
preview={!!listing.image_url}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="listing-detail__notes">
|
||||
<Title heading={4} className="listing-detail__notes-title">
|
||||
Notes
|
||||
</Title>
|
||||
<TextArea
|
||||
value={notesDraft}
|
||||
onChange={(val) => setNotesDraft(val)}
|
||||
placeholder="Your private notes about this listing…"
|
||||
rows={5}
|
||||
autosize={{ minRows: 4, maxRows: 12 }}
|
||||
className="listing-detail__notes-textarea"
|
||||
showClear
|
||||
/>
|
||||
<Space className="listing-detail__notes-actions">
|
||||
<Button
|
||||
theme="solid"
|
||||
type="primary"
|
||||
loading={notesSaving}
|
||||
disabled={notesSaving || (notesDraft ?? '') === (listing.notes ?? '')}
|
||||
onClick={handleSaveNotes}
|
||||
>
|
||||
Store notes
|
||||
</Button>
|
||||
</Space>
|
||||
</div>
|
||||
</Col>
|
||||
<Col span={24} lg={12}>
|
||||
<div className="listing-detail__info-section">
|
||||
@@ -377,10 +474,12 @@ export default function ListingDetail() {
|
||||
<Descriptions column={1}>
|
||||
{data.map((item, index) => (
|
||||
<Descriptions.Item key={index}>
|
||||
<Space>
|
||||
{item.Icon}
|
||||
{item.value}
|
||||
</Space>
|
||||
<Tooltip content={item.helpText} position="left">
|
||||
<span className="listing-detail__details-item">
|
||||
{item.Icon}
|
||||
{item.value}
|
||||
</span>
|
||||
</Tooltip>
|
||||
</Descriptions.Item>
|
||||
))}
|
||||
</Descriptions>
|
||||
@@ -423,6 +522,7 @@ export default function ListingDetail() {
|
||||
|
||||
<ListingDeletionModal
|
||||
visible={deleteModalVisible}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
onConfirm={confirmDeletion}
|
||||
onCancel={() => setDeleteModalVisible(false)}
|
||||
/>
|
||||
|
||||
@@ -89,6 +89,49 @@
|
||||
}
|
||||
}
|
||||
|
||||
&__notes {
|
||||
padding: 1.5rem;
|
||||
border-top: 1px solid var(--semi-color-border);
|
||||
}
|
||||
|
||||
&__notes-title {
|
||||
margin-bottom: 1rem !important;
|
||||
}
|
||||
|
||||
&__notes-actions {
|
||||
margin-top: 0.75rem;
|
||||
width: 100%;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
&__notes-textarea {
|
||||
background: #2a2a2a !important;
|
||||
border: 1px solid @color-border-bright !important;
|
||||
border-radius: @radius-input !important;
|
||||
box-shadow: none !important;
|
||||
outline: none !important;
|
||||
transition: border-color @transition-fast, background @transition-fast !important;
|
||||
|
||||
textarea {
|
||||
background: transparent !important;
|
||||
border: 0 !important;
|
||||
box-shadow: none !important;
|
||||
outline: none !important;
|
||||
color: @color-text !important;
|
||||
font-family: @font-ui !important;
|
||||
font-size: @text-base !important;
|
||||
}
|
||||
|
||||
&.semi-input-textarea-focus,
|
||||
&:focus,
|
||||
&:focus-within {
|
||||
border-color: @color-accent !important;
|
||||
box-shadow: none !important;
|
||||
outline: none !important;
|
||||
background: #2f2f2f !important;
|
||||
}
|
||||
}
|
||||
|
||||
&__watch-btn {
|
||||
color: @color-muted !important;
|
||||
border: 1px solid @color-border-bright !important;
|
||||
@@ -128,6 +171,13 @@
|
||||
padding: 1.5rem;
|
||||
}
|
||||
|
||||
&__details-item {
|
||||
cursor: help;
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
&__map-container {
|
||||
height: 400px;
|
||||
width: 100%;
|
||||
|
||||
@@ -6,11 +6,15 @@
|
||||
import ListingsOverview from '../../components/listings/ListingsOverview.jsx';
|
||||
import Headline from '../../components/headline/Headline.jsx';
|
||||
|
||||
export default function Listings() {
|
||||
/**
|
||||
* @param {{ mode?: 'all' | 'watchlist' }} props
|
||||
*/
|
||||
export default function Listings({ mode = 'all' }) {
|
||||
const title = mode === 'watchlist' ? 'Watchlist' : 'Listings';
|
||||
return (
|
||||
<>
|
||||
<Headline text="Listings" />
|
||||
<ListingsOverview />
|
||||
<Headline text={title} />
|
||||
<ListingsOverview mode={mode} />
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -37,7 +37,10 @@ export default function MapView() {
|
||||
const sp = useSearchParams();
|
||||
const [searchParams, setSearchParams] = sp;
|
||||
const listings = useSelector((state) => state.listingsData.mapListings);
|
||||
const homeAddress = useSelector((state) => state.userSettings.settings.home_address);
|
||||
const userSettings = useSelector((state) => state.userSettings.settings);
|
||||
const homeAddress = userSettings?.home_address;
|
||||
const listingDeletionPref = userSettings?.listing_deletion_preference;
|
||||
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
|
||||
|
||||
const jobs = useSelector((state) => state.jobsData.jobs);
|
||||
const [jobId, setJobId] = useSearchParamState(sp, 'job', null, parseString);
|
||||
@@ -52,10 +55,14 @@ export default function MapView() {
|
||||
|
||||
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
|
||||
const [listingToDelete, setListingToDelete] = useState(null);
|
||||
const deleteListingRef = useRef(null);
|
||||
|
||||
const confirmListingDeletion = async (hardDelete) => {
|
||||
const confirmListingDeletion = async (hardDelete, remember, id = listingToDelete) => {
|
||||
try {
|
||||
await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
|
||||
if (remember) {
|
||||
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
|
||||
}
|
||||
await xhrDelete('/api/listings/', { ids: [id], hardDelete });
|
||||
Toast.success('Listing successfully removed');
|
||||
fetchListings();
|
||||
} catch (error) {
|
||||
@@ -66,6 +73,15 @@ export default function MapView() {
|
||||
}
|
||||
};
|
||||
|
||||
deleteListingRef.current = (id) => {
|
||||
if (listingDeletionPref?.skipPrompt) {
|
||||
confirmListingDeletion(listingDeletionPref.hardDelete, false, id);
|
||||
return;
|
||||
}
|
||||
setListingToDelete(id);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
// Only reset to full range when no URL override is set
|
||||
if (urlPriceMax === null) {
|
||||
@@ -88,10 +104,7 @@ export default function MapView() {
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
window.deleteListing = (id) => {
|
||||
setListingToDelete(id);
|
||||
setDeleteModalVisible(true);
|
||||
};
|
||||
window.deleteListing = (id) => deleteListingRef.current(id);
|
||||
|
||||
window.viewDetails = (id) => {
|
||||
navigate(`/listings/listing/${id}`);
|
||||
@@ -472,6 +485,7 @@ export default function MapView() {
|
||||
|
||||
<ListingDeletionModal
|
||||
visible={deleteModalVisible}
|
||||
defaultDeleteType={defaultDeleteType}
|
||||
onConfirm={confirmListingDeletion}
|
||||
onCancel={() => {
|
||||
setDeleteModalVisible(false);
|
||||
|
||||
@@ -37,7 +37,7 @@ const Users = function Users() {
|
||||
await actions.jobsData.getJobs();
|
||||
await actions.user.getUsers();
|
||||
} catch (error) {
|
||||
Toast.error(error);
|
||||
Toast.error(error.error);
|
||||
setUserIdToBeRemoved(null);
|
||||
}
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user