Compare commits

...

45 Commits

Author SHA1 Message Date
orangecoding
9296bcdc86 fixing 'open in fredy' link 2026-06-03 08:12:30 +02:00
orangecoding
bbebc2a1a2 storing the date when a status was set 2026-06-02 21:09:35 +02:00
orangecoding
d2978c14db merged master 2026-06-02 20:41:43 +02:00
orangecoding
5ceac25aa6 fixing #319 & #318 2026-06-02 20:11:43 +02:00
orangecoding
34b68e1f52 fixing notification adapter 2026-06-02 19:53:06 +02:00
orangecoding
696ae451d3 fixing tooltip issues 2026-06-02 12:57:12 +02:00
orangecoding
317ef79336 adding ability to tag listings eg if you have applied to it / adding ability to add notes to a listing 2026-06-02 12:48:01 +02:00
Christian Kellner
6428e7ad78 Ghost commit for rebuilding due to ci interruption 2026-06-02 11:02:05 +02:00
orangecoding
2bcec04d55 next release version 2026-06-02 10:56:05 +02:00
orangecoding
ee2112a24d fixing tests harder 2026-06-02 10:55:16 +02:00
orangecoding
5a54448288 fixing tests 2026-06-02 10:49:06 +02:00
Ramin
f1b8709ab7 feat: remember listing delete preference (#314)
* feat: remember listing delete preference

Persist soft/hard choice and skip-confirm in user settings.
2026-06-02 10:23:45 +02:00
orangecoding
b56e13aa16 upgrading dependencies 2026-06-02 09:26:46 +02:00
Christian Kellner
a834abc31c fixing filtering of lists (#311)
* fixing listing filtering by applying the correct id
2026-06-02 09:24:45 +02:00
Ramin
573868eccb feat(ui): zoom map to saved area when editing a job (#313)
Fit the job edit map to existing polygon areas on init.
2026-06-02 08:54:56 +02:00
orangecoding
1a210d7c1c poi for proxies 2026-05-25 11:54:49 +02:00
orangecoding
996b841cfb adding ability to add proxies for cloak 2026-05-24 20:49:27 +02:00
orangecoding
b2e294e38c next release version 2026-05-21 21:46:13 +02:00
orangecoding
8afeaa05d9 fixing cloakbrowser connection issue 2026-05-21 21:45:57 +02:00
orangecoding
ec47137b89 upgrading dependencies 2026-05-21 21:40:35 +02:00
orangecoding
33161de087 upgrading dependencies 2026-05-19 09:15:12 +02:00
Stephan
acab23207e fix: kleinanzige listing might have a different structure when it was with no image created (#308) 2026-05-16 14:16:31 +02:00
orangecoding
2896d531e4 upgrading pois 2026-05-13 08:14:57 +02:00
orangecoding
0cbfa25062 upgrading pois 2026-05-12 19:28:58 +02:00
orangecoding
bcd3042026 fixing error messages not being shown properly in user table 2026-05-12 13:24:13 +02:00
orangecoding
0ce93acaf6 more demo fixes 2026-05-12 13:12:26 +02:00
orangecoding
cabef973a2 forbid backuo/restore in demo mode 2026-05-12 12:42:25 +02:00
orangecoding
3d0fa87d19 upgrading dependencies 2026-05-12 09:23:52 +02:00
orangecoding
8b012ef2f1 upgrading dependencies / new pois 2026-05-11 09:18:32 +02:00
orangecoding
6816b0aded next release version 2026-05-10 15:43:13 +02:00
Christian Kellner
ac02817d4e Switch browser engine from puppeteer-extra/stealth to CloakBrowser (#307)
* Switch browser engine from puppeteer-extra/stealth to CloakBrowser

- Replace puppeteer, puppeteer-extra, puppeteer-extra-plugin-stealth with
  cloakbrowser + puppeteer-core; CloakBrowser applies 49 source-level C++
  fingerprint patches that cannot be detected at the JS layer.
- Enable humanize:true in launchBrowser() for Bézier mouse curves, natural
  keyboard timing, and realistic scroll physics.
- Remove manual userDataDir management and ARM64 executablePath override;
  CloakBrowser ships its own binary for x86_64 and arm64.
- Proxy is now passed via CloakBrowser's native proxy option instead of
  --proxy-server Chrome flag.
- Dockerfile: add fonts-noto-color-emoji + fonts-freefont-ttf so canvas
  fingerprint hashes match real browsers (required for Kasada/Akamai);
  replace npx puppeteer browsers install with node ensureBinary() call;
  remove TARGETARCH ARG and ARM64 system-Chromium branch.
- Update test mock to reflect simplified browser object (no __fredy_* fields).

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Add --ignore-certificate-errors for CloakBrowser's custom Chromium

CloakBrowser ships its own Chromium binary with an independent CA bundle.
This flag prevents ERR_CERT_AUTHORITY_INVALID failures in environments with
SSL-inspecting proxies or non-standard root CAs (Docker CI, corporate networks).

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Harden CloakBrowser integration and fix kleinanzeigen detail test

- Remove all CDP overrides (applyBotPreventionToPage, applyLanguagePersistence,
  applyPostNavigationHumanSignals) that created detectable inconsistencies on top
  of CloakBrowser's C++ patches; pass locale to CloakBrowser launch instead
- Drop --lang arg (replaced by CloakBrowser locale flag)
- Extend immowelt puppeteerTimeout to 90 s to accommodate React SPA rendering
  latency under CloakBrowser's humanise delays
- Fix kleinanzeigen detail test: serve the offline fixture for the search URL
  so only individual detail pages are fetched live, avoiding rate limiting from
  a second fresh session hitting the same search endpoint

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Fix immowelt bot detection with two-phase navigation and fixture-backed detail test

Immowelt's CDN challenges cold browser sessions before React can render the
listing grid, causing the old waitForSelector approach to silently timeout.

- Add preNavigateUrl option to puppeteerExtractor: visits a warm-up page
  first so the site sees an established session before the search URL
- Add waitForNetworkIdle option: a second idle-wait phase after domcontentloaded
  that catches React's listing API round-trip (which fires long after the
  initial HTML is parsed); errors are swallowed so partial DOM is still used
- Switch immowelt config to waitForSelector=null + networkidle warm-up so
  page.content() is returned after the SPA has loaded its data
- Set immowelt preNavigateUrl to the homepage to warm the session
- In the detail enrichment test, spy on puppeteerExtractor to serve the
  offline fixture for the search URL; only individual listing detail pages
  are fetched live (they are far less aggressively protected)

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Ensure CloakBrowser binary is present before any live test runs

Add a Vitest globalSetup that calls ensureBinary() once in the main process
before workers start. Without this, running yarn test on a fresh checkout
(or after the binary cache is cleared) immediately fails every browser-based
test with "Failed to launch the browser process" before any useful output
appears. The setup is a no-op in offline mode and when the binary is already
cached.

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Ensure CloakBrowser binary at startup for non-Docker installs

Direct runs (yarn start:backend) on a fresh checkout have no binary and
only crash when the first scraping job fires. Calling ensureBinary() at
startup downloads it on first run and is instant when already cached.
In Docker it stays a no-op since the binary is pre-baked during docker build.

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Fix --no-zygote comment: ICU crash was corrupted .4 binary, not fd issue

The "Invalid file descriptor to ICU data received" crash seen in Sparkasse
tests was caused by a partially-extracted CloakBrowser .4 binary that
contained only the chrome executable but was missing icudtl.dat and other
resource files. The ensureBinary() function returned this incomplete
installation because latest_version_linux-x64 pointed to .4.

The --no-zygote flag is kept as a safeguard for container environments
with limited kernel namespaces, but the comment now accurately describes
its purpose rather than attributing it to a non-existent fd inheritance issue.

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Add ensureValidBinary() to detect and auto-heal corrupt CloakBrowser installs

CloakBrowser's ensureBinary() only checks that the chrome executable exists,
not that required resource files (icudtl.dat, resources.pak) are present.
A partial extraction — e.g. an interrupted update — can leave a directory
that passes ensureBinary()'s check but causes Chrome to crash immediately
with "Invalid file descriptor to ICU data received".

ensureValidBinary() wraps ensureBinary() with a completeness check:
- If the required resource files are missing it removes the corrupt directory
  and all latest_version* markers, then calls ensureBinary() again so it
  falls back to (or re-downloads) a complete build.
- It pins the validated path via CLOAKBROWSER_BINARY_PATH so CloakBrowser's
  own internal ensureBinary() call inside launch() always uses the same,
  verified binary.

Used in index.js (app startup) and test/globalSetup.js (before live tests).

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Fix sparkasse detail test: serve search URL from fixture to avoid rate-limiting

The second sparkasse test launched a fresh browser against the live search
endpoint right after the first test already did, leaving the IP in a suspicious
state that caused bot detection or rate-limiting to return empty results.
When getListings() returns nothing, execute() resolves to undefined and
expect(listings).toBeInstanceOf(Array) fails.

Apply the same hybrid fixture approach used by kleinanzeigen and immowelt:
intercept puppeteerExtractor calls whose pathname matches the search URL and
return the offline fixture, while letting individual detail page requests go
live (they are less aggressively rate-limited than the search endpoint).

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Fix sparkasse detail test: shared browser, direct fetchDetails call

Remove the fixture-backed spy — live tests must hit the real server.

Root problem: two cold browser sessions hitting sparkasse in quick succession
triggered bot detection, causing the second search request to return empty
results and execute() to resolve undefined.

Fix:
- One browser launched in beforeAll and reused across both tests, so both
  the search and detail requests come from the same warm session.
- The detail test calls provider.config.fetchDetails() directly on the
  listings returned by the first test instead of re-running the full pipeline.
  This avoids a redundant second scrape of the search page while still
  exercising the live detail endpoint.

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Eliminate fixture spies and double live requests in all provider detail tests

All five provider tests with a 'with provider_details enabled' describe block
were either (a) intercepting the search URL with an offline fixture to avoid
hitting the live server twice, or (b) re-running the full execute() pipeline
with a fresh browser, which triggered rate-limiting / bot detection on the
second cold request.

Pattern applied to all five:
- immowelt, kleinanzeigen, wgGesucht, immobilienDe: launch one browser in
  beforeAll/afterAll, pass it to the first test's Fredy constructor, and call
  provider.config.fetchDetails() directly in the second test using the listings
  and browser already in hand. One warm session, two live endpoints tested.
- immoscout: API-based (no browser), so no browser sharing needed. Second test
  calls provider.config.fetchDetails() directly on liveListings[0] from the
  first test instead of re-querying the search API.

Removed: all readFixture spies, getKnownListingHashesForJobAndProvider mocks,
and the puppeteerExtractorMod imports that were only needed for the spy.

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

* Fix ensureValidBinary for macOS: platform-aware completeness check

On macOS the CloakBrowser binary lives at:
  ~/.cloakbrowser/chromium-X.Y.Z/Chromium.app/Contents/MacOS/Chromium

path.dirname() gave Contents/MacOS/ — but icudtl.dat and resources.pak
are inside Contents/Frameworks/…, not next to the binary. So the old
code incorrectly flagged every macOS installation as corrupt, deleted only
the MacOS/ subdirectory (not the full versioned dir), then failed again.

Fixes:
- isBinaryComplete: on macOS check for Info.plist and Frameworks/ inside
  Chromium.app/Contents/ instead of looking for Linux resource files next
  to the binary. On Linux/Windows the existing check is unchanged.
- getVersionedDir: resolves the full chromium-X.Y.Z/ directory regardless
  of platform (4 levels up on macOS, 1 on Linux/Windows) so
  removeCorruptInstallation always deletes the entire versioned tree.
- missingDescription: reports the correct missing items per platform.

https://claude.ai/code/session_01WXzA3orbwE2hdk723c6MgH

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-05-10 15:42:31 +02:00
datenwurm
fe0a09fe1c fix: Sort direction toggle not working in listings overview (#306)
Co-authored-by: datenwurm <git@datenwurm.net>
2026-05-08 09:24:20 +02:00
orangecoding
2f00966f27 next release version 2026-05-07 19:12:17 +02:00
orangecoding
921057252d adding immoscout shape search 2026-05-07 19:11:47 +02:00
orangecoding
703c602527 table overview for jobs 2026-05-07 15:59:55 +02:00
orangecoding
0e29c9b9c6 next release version 2026-05-07 12:45:52 +02:00
datenwurm
f60c5859f9 feat: Add grid/table view toggle to listings overview (#305)
* feat: Add delete button to listing detail view

* feat: Add grid/table view toggle to listings overview

---------

Co-authored-by: datenwurm <git@datenwurm.net>
2026-05-07 12:12:49 +02:00
datenwurm
ee54cc495b feat: Add delete button to listing detail view (#304)
Co-authored-by: datenwurm <git@datenwurm.net>
2026-05-07 11:53:51 +02:00
orangecoding
96582ecff4 gmx example 2026-05-02 20:07:03 +02:00
orangecoding
3de82dfa41 fixing error message when passwords do not match / fixing placeholder image 2026-05-02 20:00:11 +02:00
orangecoding
d7ee4f6909 adding gitattributed§ 2026-05-01 20:12:58 +02:00
orangecoding
bf4bae9bf5 upgrading dependencies 2026-05-01 20:12:40 +02:00
orangecoding
3d10dc6042 moving from restana to fastify 2026-04-27 16:56:04 +02:00
orangecoding
fef6d06a9d next release version 2026-04-27 16:04:05 +02:00
orangecoding
951b69a67f downgrading restana 2026-04-27 16:03:45 +02:00
117 changed files with 6691 additions and 6259 deletions

1
.gitattributes vendored Normal file
View File

@@ -0,0 +1 @@
test/testFixtures/** linguist-vendored

View File

@@ -8,7 +8,7 @@ Fredy is a self-hosted real estate finder for Germany. It scrapes German real es
- Node.js >= 22, ESM-only (`"type": "module"`)
- Default port: 9998, default login: admin / admin
- SQLite via `better-sqlite3` (synchronous all DB ops are sync; only network I/O is async)
- SQLite via `better-sqlite3` (synchronous - all DB ops are sync; only network I/O is async)
## Commands
@@ -66,13 +66,13 @@ scheduler (every N minutes) or manual trigger via POST /api/jobs/:id/run
### Plugin systems
**Providers** (`lib/provider/*.js`) each module exports:
- `metaInformation` `{ id, name, baseUrl }`
- `config` `ProviderConfig` with `requiredFieldNames`, `crawlContainer`, `crawlFields`, `sortByDateParam`, `normalize()`, `filter()`, optional `getListings()`, `fetchDetails()`, `activeTester()`
- `init(sourceConfig, blacklist)` called before each job run; providers are **stateful modules** holding mutable `url` and `appliedBlackList` at module scope
**Providers** (`lib/provider/*.js`) - each module exports:
- `metaInformation` - `{ id, name, baseUrl }`
- `config` - `ProviderConfig` with `requiredFieldNames`, `crawlContainer`, `crawlFields`, `sortByDateParam`, `normalize()`, `filter()`, optional `getListings()`, `fetchDetails()`, `activeTester()`
- `init(sourceConfig, blacklist)` - called before each job run; providers are **stateful modules** holding mutable `url` and `appliedBlackList` at module scope
**Notification adapters** (`lib/notification/adapter/*.js`) each exports:
- `config` `{ id, name, description, fields }` (drives the UI form)
**Notification adapters** (`lib/notification/adapter/*.js`) - each exports:
- `config` - `{ id, name, description, fields }` (drives the UI form)
- `send({ serviceName, newListings, notificationConfig, jobKey, baseUrl })`
- Loaded dynamically at startup via `fs.readdirSync`
@@ -98,18 +98,18 @@ scheduler (every N minutes) or manual trigger via POST /api/jobs/:id/run
### MCP server
Two transports:
1. **stdio** (`lib/mcp/stdio.js`) for Claude Desktop/LM Studio; opens its own DB connection (main process need not be running)
2. **HTTP** (`/api/mcp`) authenticated via Bearer token (`mcp_token` column in `users` table)
1. **stdio** (`lib/mcp/stdio.js`) - for Claude Desktop/LM Studio; opens its own DB connection (main process need not be running)
2. **HTTP** (`/api/mcp`) - authenticated via Bearer token (`mcp_token` column in `users` table)
Tools: `list_jobs`, `get_job`, `list_listings`, `get_listing`, `get_current_date_time`. Responses are Markdown via `lib/mcp/mcpNormalizer.js`.
## Key Conventions
- **ESM only** `import`/`export` everywhere, no CommonJS
- **JSDoc typedefs** (no TypeScript) in `lib/types/` `listing.js`, `job.js`, `filter.js`, `providerConfig.js`
- **Copyright header** required on all `.js` files enforced by `lint-staged` pre-commit hook via `copyright.js`
- **ESM only** - `import`/`export` everywhere, no CommonJS
- **JSDoc typedefs** (no TypeScript) in `lib/types/` - `listing.js`, `job.js`, `filter.js`, `providerConfig.js`
- **Copyright header** required on all `.js` files - enforced by `lint-staged` pre-commit hook via `copyright.js`
- **`NoNewListingsWarning`** (`lib/errors.js`) is used as control flow to short-circuit the pipeline (not an error)
- **Test fixtures** in `test/testFixtures/` HTML/JSON snapshots per provider; `TEST_MODE=offline` mocks `puppeteerExtractor` and global `fetch` via `test/offlineFixtures.js`
- **Test fixtures** in `test/testFixtures/` - HTML/JSON snapshots per provider; `TEST_MODE=offline` mocks `puppeteerExtractor` and global `fetch` via `test/offlineFixtures.js`
- **`conf/config.json`** is the only runtime config file; created with defaults if missing
## Coding

View File

@@ -1,16 +1,15 @@
FROM node:22-slim
ARG TARGETARCH
# System deps for Chrome for Testing + build tools for native modules (better-sqlite3)
# On ARM64 we also install system Chromium (Chrome for Testing has no ARM64 binary)
# System deps for CloakBrowser + build tools for native modules (better-sqlite3)
# fonts-noto-color-emoji and fonts-freefont-ttf are required so canvas fingerprint
# hashes match real browsers; missing emoji fonts cause bot detection on Kasada/Akamai.
RUN apt-get update && apt-get install -y --no-install-recommends \
curl ca-certificates fonts-liberation libasound2 \
libatk-bridge2.0-0 libatk1.0-0 libcups2 libdbus-1-3 \
libdrm2 libgbm1 libgtk-3-0 libnspr4 libnss3 \
libx11-xcb1 libxcomposite1 libxdamage1 libxrandr2 xdg-utils \
fonts-noto-color-emoji fonts-freefont-ttf \
python3 make g++ \
&& if [ "$TARGETARCH" = "arm64" ]; then apt-get install -y --no-install-recommends chromium; fi \
&& rm -rf /var/lib/apt/lists/* \
&& mkdir -p /db /conf /fredy
@@ -26,8 +25,8 @@ RUN yarn config set network-timeout 600000 \
&& yarn --frozen-lockfile \
&& yarn cache clean
# on arm64 use the system Chromium installed above
RUN if [ "$TARGETARCH" != "arm64" ]; then npx puppeteer browsers install chrome; fi
# Pre-download the CloakBrowser stealth Chromium binary (supports x86_64 and arm64)
RUN node -e "import('cloakbrowser').then(({ensureBinary}) => ensureBinary())"
# Purge build tools now that native modules are compiled
RUN apt-get purge -y python3 make g++ \

View File

@@ -23,7 +23,7 @@
# Fredy 🏡 Your Self-Hosted Real Estate Finder for Germany
# Fredy 🏡 - Your Self-Hosted Real Estate Finder for Germany
Finding an apartment or house in Germany can be stressful and
time-consuming.\
@@ -167,6 +167,40 @@ For more information on how to set it up and use it, please refer to the [MCP Re
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
## 🛡️ Bot Detection & Proxies
Most browser-based providers (immowelt, immonet, kleinanzeigen, ...) are scraped through a hardened headless browser ([CloakBrowser](https://www.npmjs.com/package/cloakbrowser)). It makes the **browser fingerprint** indistinguishable from a real Chrome, which is enough when you run Fredy on a normal home connection.
On a **server / VPS the requests usually originate from a datacenter IP**, and providers behind anti-bot systems (e.g. AWS CloudFront/WAF) block those based on **IP reputation alone**, no matter how perfect the fingerprint is. The typical symptom: it works locally but you get `We have been detected as a bot :-/` on the server.
### The fix: a residential proxy
A **residential proxy** routes Fredy's browser through the internet connection of a real household, so the provider sees a "normal user" IP instead of a datacenter. For German portals, use a **German (DE) residential** (or mobile/4G) proxy. Plain VPNs and **datacenter proxies do not help** here, they share the same bad reputation as your server.
**Configure it** under **Settings → Execution → Proxy URL**. Supported formats:
```
http://user:pass@host:port
socks5://user:pass@host:port
```
Leave the field empty to disable. The proxy applies to all headless-browser providers and takes effect on the next job run (no restart needed). Immoscout uses a separate mobile API and is not affected.
### Where to get a residential proxy
Residential proxies are a paid service (usually billed per GB, Fredy's traffic is small). Well-known providers offering German residential IPs include:
| Provider | Notes |
|---|---|
| [IPRoyal](https://iproyal.com) | Pay-as-you-go, no monthly minimum, good for low volume |
| [Webshare](https://www.webshare.io) | Cheap entry tier, has a small free plan to test with |
| [Decodo (formerly Smartproxy)](https://decodo.com) | Easy setup, country/city targeting |
| [SOAX](https://soax.com) | Residential + mobile, fine-grained geo-targeting |
| [Bright Data](https://brightdata.com) | Largest pool, most features, higher complexity/price |
| [Oxylabs](https://oxylabs.io) | Enterprise-grade, larger plans |
This is not an endorsement, pick whatever fits your budget. For low-volume use like Fredy, a pay-as-you-go plan (e.g. IPRoyal) or a cheap entry tier (e.g. Webshare) is usually plenty. Make sure to select **Germany** as the proxy location and keep the search interval reasonable (the higher the interval, the less you look like a bot).
## Analytics
Fredy is completely free (and will always remain free). However, it would be a huge help if youd allow me to collect some analytical data.

View File

@@ -43,13 +43,13 @@ for i in $(seq 1 30); do
done
# Verify the DB is readable/writable via the API.
# /api/demo is unauthenticated and reads the settings table if SQLite is broken this returns an error.
# /api/demo is unauthenticated and reads the settings table - if SQLite is broken this returns an error.
echo "Testing DB via API (/api/demo)..."
DEMO_RESPONSE=$(docker exec fredy curl -sf http://localhost:9998/api/demo 2>&1)
if echo "$DEMO_RESPONSE" | grep -q "demoMode"; then
echo "DB is readable (got demoMode from /api/demo)"
else
echo "DB check failed unexpected response from /api/demo: $DEMO_RESPONSE"
echo "DB check failed - unexpected response from /api/demo: $DEMO_RESPONSE"
docker logs fredy
exit 1
fi

File diff suppressed because it is too large Load Diff

View File

@@ -11,6 +11,8 @@
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<title>Fredy || Real Estate Finder</title>
<link rel="icon" type="image/png" href="/ui/src/assets/heart.png" />
<link rel="apple-touch-icon" href="/ui/src/assets/heart.png" />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />

View File

@@ -15,6 +15,15 @@ import { initGeocodingCron } from './lib/services/crons/geocoding-cron.js';
import { getSettings } from './lib/services/storage/settingsStorage.js';
import SqliteConnection, { computeDbPath } from './lib/services/storage/SqliteConnection.js';
import { initJobExecutionService } from './lib/services/jobs/jobExecutionService.js';
import { ensureValidBinary } from './lib/services/ensureValidBinary.js';
// Ensure the CloakBrowser stealth Chromium binary is present and complete before
// jobs run. ensureValidBinary() also detects and auto-heals partial extractions
// (e.g. a newer version that was downloaded but only the chrome executable was
// written) so Chrome never crashes with "Invalid file descriptor to ICU data".
logger.info('Checking CloakBrowser binary...');
await ensureValidBinary();
logger.info('CloakBrowser binary ready.');
//in the config, we store the path of the sqlite file, thus we must check if it is available
const isConfigAccessible = await checkIfConfigIsAccessible();

View File

@@ -5,9 +5,10 @@
import { NoNewListingsWarning } from './errors.js';
import {
storeListings,
getKnownListingHashesForJobAndProvider,
deleteListingsById,
getKnownListingHashesForJobAndProvider,
storeListings,
updateListingDistance,
} from './services/storage/listingsStorage.js';
import { getJob } from './services/storage/jobStorage.js';
import * as notify from './notification/notify.js';
@@ -16,8 +17,7 @@ import urlModifier from './services/queryStringMutator.js';
import logger from './services/logger.js';
import { geocodeAddress } from './services/geocoding/geoCodingService.js';
import { distanceMeters } from './services/listings/distanceCalculator.js';
import { getUserSettings, getSettings } from './services/storage/settingsStorage.js';
import { updateListingDistance } from './services/storage/listingsStorage.js';
import { getSettings, getUserSettings } from './services/storage/settingsStorage.js';
import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
import { formatListing } from './utils/formatListing.js';
@@ -97,9 +97,9 @@ class FredyPipelineExecutioner {
}
/**
* Optionally enrich new listings with data from their detail pages.
* Optionally, enrich new listings with data from their detail pages.
* Only called when the provider config defines a `fetchDetails` function.
* Runs all fetches in parallel. Each individual fetch must handle its own errors
* Runs all fetches in parallel. Each fetch must handle its own errors
* and always resolve (never reject) to avoid aborting other listings.
*
* @param {Listing[]} newListings New listings to enrich.
@@ -132,7 +132,7 @@ class FredyPipelineExecutioner {
for (const listing of newListings) {
if (listing.address) {
const coords = await geocodeAddress(listing.address);
if (coords) {
if (coords && coords.lat !== -1 && coords.lng !== -1) {
listing.latitude = coords.lat;
listing.longitude = coords.lng;
}
@@ -227,7 +227,7 @@ class FredyPipelineExecutioner {
const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
return new Promise((resolve, reject) => {
extractor
.execute(url, this._providerConfig.waitForSelector)
.execute(url, this._providerConfig.waitForSelector, this._providerId)
.then(() => {
const listings = extractor.parseResponseText(
this._providerConfig.crawlContainer,
@@ -264,15 +264,15 @@ class FredyPipelineExecutioner {
const requiredKeys = this._providerConfig.requiredFieldNames;
const requireValues = ['id', 'link', 'title'];
const filteredListings = listings
// this should never filter some listings out, because the normalize function should always extract all fields.
.filter((item) => requiredKeys.every((key) => key in item))
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
.filter(this._providerConfig.filter)
// filter out listings that are missing required fields
.filter((item) => requireValues.every((key) => item[key] != null));
return filteredListings;
return (
listings
// this should never filter some listings out, because the normalize function should always extract all fields.
.filter((item) => requiredKeys.every((key) => key in item))
// TODO: move blacklist filter to this file, so it will handle for all providers in same way.
.filter(this._providerConfig.filter)
// filter out listings that are missing required fields
.filter((item) => requireValues.every((key) => item[key] != null))
);
}
/**

View File

@@ -7,4 +7,11 @@ export const TRACKING_POIS = {
DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
WELCOME_FINISHED: 'WELCOME_FINISHED',
WELCOME_SKIPPED: 'WELCOME_SKIPPED',
JOBS_TABLE_VIEW: 'JOBS_TABLE_VIEW',
LISTING_TABLE_VIEW: 'LISTING_TABLE_VIEW',
BASE_URL_SETTING: 'BASE_URL_SETTING',
SET_PROXY_SETTING: 'SET_PROXY_SETTING',
DETECTED_AS_BOT: 'DETECTED_AS_BOT',
NOTES_CREATE: 'NOTES_CREATE',
USING_LISTING_STATUS: 'USING_LISTING_STATUS',
};

View File

@@ -3,64 +3,100 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { notificationAdapterRouter } from './routes/notificationAdapterRouter.js';
import { authInterceptor, cookieSession, adminInterceptor } from './security.js';
import { generalSettingsRouter } from './routes/generalSettingsRoute.js';
import { providerRouter } from './routes/providerRouter.js';
import { versionRouter } from './routes/versionRouter.js';
import { loginRouter } from './routes/loginRoute.js';
import { userRouter } from './routes/userRoute.js';
import { userSettingsRouter } from './routes/userSettingsRoute.js';
import { jobRouter } from './routes/jobRouter.js';
import bodyParser from 'body-parser';
import restana from 'restana';
import files from 'serve-static';
import Fastify from 'fastify';
import fastifyHelmet from '@fastify/helmet';
import fastifyCookie from '@fastify/cookie';
import fastifySession from '@fastify/session';
import fastifyStatic from '@fastify/static';
import path from 'path';
import { getDirName } from '../utils.js';
import { demoRouter } from './routes/demoRouter.js';
import logger from '../services/logger.js';
import { listingsRouter } from './routes/listingsRouter.js';
import { getSettings, getOrCreateSessionSecret } from '../services/storage/settingsStorage.js';
import { dashboardRouter } from './routes/dashboardRouter.js';
import { backupRouter } from './routes/backupRouter.js';
import { trackingRouter } from './routes/trackingRoute.js';
import logger from '../services/logger.js';
import { authHook, adminHook } from './security.js';
import loginPlugin from './routes/loginRoute.js';
import demoPlugin from './routes/demoRouter.js';
import jobPlugin from './routes/jobRouter.js';
import versionPlugin from './routes/versionRouter.js';
import listingsPlugin from './routes/listingsRouter.js';
import dashboardPlugin from './routes/dashboardRouter.js';
import userSettingsPlugin from './routes/userSettingsRoute.js';
import trackingPlugin from './routes/trackingRoute.js';
import generalSettingsPlugin from './routes/generalSettingsRoute.js';
import backupPlugin from './routes/backupRouter.js';
import userPlugin from './routes/userRoute.js';
import notificationAdapterPlugin from './routes/notificationAdapterRouter.js';
import providerPlugin from './routes/providerRouter.js';
import { registerMcpRoutes } from '../mcp/mcpHttpRoute.js';
const service = restana();
const staticService = files(path.join(getDirName(), '../ui/public'));
const PORT = (await getSettings()).port || 9998;
const sessionSecret = await getOrCreateSessionSecret();
const SESSION_MAX_AGE = 2 * 60 * 60 * 1000;
service.use(bodyParser.json());
service.use(cookieSession(sessionSecret));
service.use(staticService);
service.use('/api/admin', authInterceptor());
service.use('/api/jobs', authInterceptor());
service.use('/api/version', authInterceptor());
service.use('/api/listings', authInterceptor());
service.use('/api/dashboard', authInterceptor());
service.use('/api/user/settings', authInterceptor());
service.use('/api/tracking', authInterceptor());
// /admin can only be accessed when user is having admin permissions
service.use('/api/admin', adminInterceptor());
service.use('/api/jobs/notificationAdapter', notificationAdapterRouter);
service.use('/api/admin/generalSettings', generalSettingsRouter);
service.use('/api/admin/backup', backupRouter);
service.use('/api/jobs/provider', providerRouter);
service.use('/api/admin/users', userRouter);
service.use('/api/user/settings', userSettingsRouter);
service.use('/api/version', versionRouter);
service.use('/api/jobs', jobRouter);
service.use('/api/login', loginRouter);
service.use('/api/listings', listingsRouter);
service.use('/api/dashboard', dashboardRouter);
service.use('/api/tracking', trackingRouter);
//this route is unsecured intentionally as it is being queried from the login page
service.use('/api/demo', demoRouter);
// MCP Streamable HTTP endpoint (secured via Bearer token, not cookie-session)
registerMcpRoutes(service);
service.start(PORT).then(() => {
logger.debug(`Started API service on port ${PORT}`);
const fastify = Fastify({
logger: false,
bodyLimit: 50 * 1024 * 1024, // 50 MB for backup uploads
});
// Security headers (CSP disabled to avoid breaking the SPA)
await fastify.register(fastifyHelmet, { contentSecurityPolicy: false });
// Cookie + session (in-memory store, signed cookie)
await fastify.register(fastifyCookie);
await fastify.register(fastifySession, {
secret: sessionSecret,
cookieName: 'fredy-admin-session',
cookie: {
maxAge: SESSION_MAX_AGE,
httpOnly: true,
secure: false,
sameSite: 'lax',
},
saveUninitialized: false,
});
// Serve the React SPA from ui/public/
await fastify.register(fastifyStatic, {
root: path.join(getDirName(), '../ui/public'),
wildcard: false,
});
// Public routes - no auth required
fastify.register(loginPlugin, { prefix: '/api/login' });
fastify.register(demoPlugin, { prefix: '/api/demo' });
// User-authenticated routes
fastify.register(async (app) => {
app.addHook('preHandler', authHook);
app.register(jobPlugin, { prefix: '/api/jobs' });
app.register(notificationAdapterPlugin, { prefix: '/api/jobs/notificationAdapter' });
app.register(providerPlugin, { prefix: '/api/jobs/provider' });
app.register(versionPlugin, { prefix: '/api/version' });
app.register(listingsPlugin, { prefix: '/api/listings' });
app.register(dashboardPlugin, { prefix: '/api/dashboard' });
app.register(userSettingsPlugin, { prefix: '/api/user/settings' });
app.register(trackingPlugin, { prefix: '/api/tracking' });
app.register(generalSettingsPlugin, { prefix: '/api/admin/generalSettings' });
});
// Admin-only routes
fastify.register(async (app) => {
app.addHook('preHandler', authHook);
app.addHook('preHandler', adminHook);
app.register(backupPlugin, { prefix: '/api/admin/backup' });
app.register(userPlugin, { prefix: '/api/admin/users' });
});
// MCP Streamable HTTP (Bearer token auth - no session)
registerMcpRoutes(fastify);
// SPA fallback - serve index.html for all non-API GET requests
fastify.setNotFoundHandler((request, reply) => {
if (!request.url.startsWith('/api/')) {
return reply.sendFile('index.html');
}
return reply.code(404).send({ error: 'Not found' });
});
await fastify.listen({ port: PORT, host: '0.0.0.0' });
logger.debug(`Started API service on port ${PORT}`);

View File

@@ -3,73 +3,61 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import {
buildBackupFileName,
createBackupZip,
precheckRestore,
restoreFromZip,
} from '../../services/storage/backupRestoreService.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin } from '../security.js';
const DEMO_MODE_ERROR = 'Backup and restore are not available in demo mode.';
/**
* Backup & Restore Admin Router
*
* Endpoints:
* - GET /api/admin/backup
* Returns the current database as a zip download. Content-Type: application/zip
* - POST /api/admin/backup/restore?dryRun=true
* Accepts a zip file (raw body). Returns a compatibility report, does not restore.
* - POST /api/admin/backup/restore?force=true|false
* Accepts a zip file (raw body). Restores the database; when incompatible and force=false, returns 400.
* @param {import('fastify').FastifyInstance} fastify
*/
const service = restana();
const backupRouter = service.newRouter();
export default async function backupPlugin(fastify) {
// Parse raw binary uploads as Buffer
fastify.addContentTypeParser(
['application/zip', 'application/octet-stream'],
{ parseAs: 'buffer' },
(req, body, done) => done(null, body),
);
backupRouter.get('/', async (req, res) => {
const zipBuffer = await createBackupZip();
const fileName = await buildBackupFileName();
res.setHeader('Content-Type', 'application/zip');
res.setHeader('Content-Disposition', `attachment; filename="${fileName}"`);
res.send(zipBuffer);
});
fastify.get('/', async (request, reply) => {
const settings = await getSettings();
if (settings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: DEMO_MODE_ERROR });
}
const zipBuffer = await createBackupZip();
const fileName = await buildBackupFileName();
reply.header('Content-Type', 'application/zip');
reply.header('Content-Disposition', `attachment; filename="${fileName}"`);
return reply.send(zipBuffer);
});
/**
* Read the full request body as a Buffer. Used for raw zip uploads.
* @param {import('http').IncomingMessage} req
* @returns {Promise<Buffer>}
*/
function readBody(req) {
return new Promise((resolve, reject) => {
const chunks = [];
req.on('data', (c) => chunks.push(c));
req.on('end', () => resolve(Buffer.concat(chunks)));
req.on('error', (e) => reject(e));
fastify.post('/restore', async (request, reply) => {
const settings = await getSettings();
if (settings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: DEMO_MODE_ERROR });
}
const { dryRun = 'false', force = 'false' } = request.query || {};
const doDryRun = String(dryRun) === 'true';
const doForce = String(force) === 'true';
const body = request.body; // Buffer from addContentTypeParser
if (doDryRun) {
return precheckRestore(body);
}
try {
return restoreFromZip(body, { force: doForce });
} catch (e) {
return reply.code(400).send({
message: e?.message || 'Restore failed',
details: e?.payload || null,
});
}
});
}
// Upload endpoint. Accepts raw zip (Content-Type: application/zip or application/octet-stream)
// Query parameters:
// - dryRun=true => only validate and return compatibility info
// - force=true => proceed even if incompatible
backupRouter.post('/restore', async (req, res) => {
const { dryRun = 'false', force = 'false' } = req.query || {};
const doDryRun = String(dryRun) === 'true';
const doForce = String(force) === 'true';
const body = await readBody(req);
if (doDryRun) {
res.body = await precheckRestore(body);
return res.send();
}
try {
res.body = await restoreFromZip(body, { force: doForce });
return res.send();
} catch (e) {
res.statusCode = 400;
res.body = { message: e?.message || 'Restore failed', details: e?.payload || null };
return res.send();
}
});
export { backupRouter };

View File

@@ -3,23 +3,14 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import * as jobStorage from '../../services/storage/jobStorage.js';
import * as userStorage from '../../services/storage/userStorage.js';
import { getListingsKpisForJobIds, getProviderDistributionForJobIds } from '../../services/storage/listingsStorage.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin } from '../security.js';
const service = restana();
export const dashboardRouter = service.newRouter();
function isAdmin(req) {
const user = req.session?.currentUser ? userStorage.getUser(req.session.currentUser) : null;
return !!user?.isAdmin;
}
function getAccessibleJobs(req) {
const currentUser = req.session.currentUser;
const admin = isAdmin(req);
function getAccessibleJobs(request) {
const currentUser = request.session.currentUser;
const admin = isAdmin(request);
return jobStorage
.getJobs()
.filter((job) => admin || job.userId === currentUser || job.shared_with_user.includes(currentUser));
@@ -29,43 +20,45 @@ function cap(val) {
return String(val).charAt(0).toUpperCase() + String(val).slice(1);
}
dashboardRouter.get('/', async (req, res) => {
const jobs = getAccessibleJobs(req);
const settings = await getSettings();
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function dashboardPlugin(fastify) {
fastify.get('/', async (request) => {
const jobs = getAccessibleJobs(request);
const settings = await getSettings();
// KPIs
const totalJobs = jobs.length;
const totalListings = jobs.reduce((sum, j) => sum + (j.numberOfFoundListings || 0), 0);
const jobIds = jobs.map((j) => j.id);
const { numberOfActiveListings, medianPriceOfListings } = getListingsKpisForJobIds(jobIds);
// Build Pie data in a simple shape the frontend can consume directly
// Shape: { labels: string[], values: number[] } with values as percentages
const providerPieRaw = getProviderDistributionForJobIds(jobIds);
const providerPie = Array.isArray(providerPieRaw)
? {
labels: providerPieRaw.map((p) => cap(p.type)),
values: providerPieRaw.map((p) => Number(p.value) || 0),
}
: providerPieRaw && typeof providerPieRaw === 'object'
const totalJobs = jobs.length;
const totalListings = jobs.reduce((sum, j) => sum + (j.numberOfFoundListings || 0), 0);
const jobIds = jobs.map((j) => j.id);
const { numberOfActiveListings, medianPriceOfListings } = getListingsKpisForJobIds(jobIds);
const providerPieRaw = getProviderDistributionForJobIds(jobIds);
const providerPie = Array.isArray(providerPieRaw)
? {
labels: Array.isArray(providerPieRaw.labels) ? providerPieRaw.labels : [],
values: Array.isArray(providerPieRaw.values) ? providerPieRaw.values : [],
labels: providerPieRaw.map((p) => cap(p.type)),
values: providerPieRaw.map((p) => Number(p.value) || 0),
}
: { labels: [], values: [] };
: providerPieRaw && typeof providerPieRaw === 'object'
? {
labels: Array.isArray(providerPieRaw.labels) ? providerPieRaw.labels : [],
values: Array.isArray(providerPieRaw.values) ? providerPieRaw.values : [],
}
: { labels: [], values: [] };
res.body = {
general: {
interval: settings.interval,
lastRun: settings.lastRun || null,
nextRun: settings.lastRun == null ? 0 : settings.lastRun + settings.interval * 60000,
},
kpis: {
totalJobs,
totalListings,
numberOfActiveListings,
medianPriceOfListings,
},
pie: providerPie,
};
res.send();
});
return {
general: {
interval: settings.interval,
lastRun: settings.lastRun || null,
nextRun: settings.lastRun == null ? 0 : settings.lastRun + settings.interval * 60000,
},
kpis: {
totalJobs,
totalListings,
numberOfActiveListings,
medianPriceOfListings,
},
pie: providerPie,
};
});
}

View File

@@ -3,15 +3,14 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import { getSettings } from '../../services/storage/settingsStorage.js';
const service = restana();
const demoRouter = service.newRouter();
demoRouter.get('/', async (req, res) => {
const settings = await getSettings();
res.body = Object.assign({}, { demoMode: settings.demoMode });
res.send();
});
export { demoRouter };
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function demoPlugin(fastify) {
fastify.get('/', async () => {
const settings = await getSettings();
return { demoMode: settings.demoMode };
});
}

View File

@@ -3,43 +3,54 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import { getDirName } from '../../utils.js';
import fs from 'fs';
import { ensureDemoUserExists } from '../../services/storage/userStorage.js';
import logger from '../../services/logger.js';
import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin } from '../security.js';
const service = restana();
const generalSettingsRouter = service.newRouter();
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
generalSettingsRouter.get('/', async (req, res) => {
res.body = Object.assign({}, await getSettings());
res.send();
});
generalSettingsRouter.post('/', async (req, res) => {
const { sqlitepath, ...appSettings } = req.body || {};
if (typeof appSettings.baseUrl === 'string') {
appSettings.baseUrl = appSettings.baseUrl.trim().replace(/\/$/, '');
}
const localSettings = await getSettings();
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function generalSettingsPlugin(fastify) {
fastify.get('/', async () => {
return Object.assign({}, await getSettings());
});
if (localSettings.demoMode && !isAdmin(req)) {
res.send(new Error('In demo mode, it is not allowed to change these settings.'));
return;
}
try {
if (typeof sqlitepath !== 'undefined') {
fs.writeFileSync(`${getDirName()}/../conf/config.json`, JSON.stringify({ sqlitepath }));
fastify.post('/', async (request, reply) => {
const { sqlitepath, ...appSettings } = request.body || {};
if (typeof appSettings.baseUrl === 'string') {
appSettings.baseUrl = appSettings.baseUrl.trim().replace(/\/$/, '');
}
upsertSettings(appSettings);
ensureDemoUserExists();
} catch (err) {
logger.error(err);
res.send(new Error('Error while trying to write settings.'));
return;
}
res.send();
});
export { generalSettingsRouter };
const localSettings = await getSettings();
if (!isAdmin(request)) {
const reason = localSettings.demoMode
? 'In demo mode, it is not allowed to change these settings.'
: 'Only admins can change these settings.';
return reply.code(403).send({ error: reason });
}
try {
if (typeof sqlitepath !== 'undefined') {
fs.writeFileSync(`${getDirName()}/../conf/config.json`, JSON.stringify({ sqlitepath }));
}
upsertSettings(appSettings);
ensureDemoUserExists();
if (appSettings.baseUrl != null) {
await trackPoi(TRACKING_POIS.BASE_URL_SETTING);
}
if (appSettings.proxyUrl != null) {
await trackPoi(TRACKING_POIS.SET_PROXY_SETTING);
}
} catch (err) {
logger.error(err);
return reply.code(500).send({ error: 'Error while trying to write settings.' });
}
return reply.send();
});
}

View File

@@ -3,7 +3,6 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import * as jobStorage from '../../services/storage/jobStorage.js';
import * as userStorage from '../../services/storage/userStorage.js';
import { isAdmin } from '../security.js';
@@ -13,257 +12,234 @@ import { isRunning as isJobRunning } from '../../services/jobs/run-state.js';
import { addClient as addSseClient, removeClient } from '../../services/sse/sse-broker.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
const service = restana();
const jobRouter = service.newRouter();
const DEMO_JOB_NAME = 'Demo-Job';
function doesJobBelongsToUser(job, req) {
const userId = req.session.currentUser;
if (userId == null) {
return false;
}
function doesJobBelongsToUser(job, request) {
const userId = request.session.currentUser;
if (userId == null) return false;
const user = userStorage.getUser(userId);
if (user == null) {
return false;
}
if (user == null) return false;
return user.isAdmin || job.userId === user.id;
}
jobRouter.get('/', async (req, res) => {
const isUserAdmin = isAdmin(req);
//show only the jobs which belongs to the user (or all of the user is an admin)
res.body = jobStorage
.getJobs()
.filter(
(job) =>
isUserAdmin || job.userId === req.session.currentUser || job.shared_with_user.includes(req.session.currentUser),
)
.map((job) => {
return {
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function jobPlugin(fastify) {
fastify.get('/', async (request) => {
const isUserAdmin = isAdmin(request);
return jobStorage
.getJobs()
.filter(
(job) =>
isUserAdmin ||
job.userId === request.session.currentUser ||
job.shared_with_user.includes(request.session.currentUser),
)
.map((job) => ({
...job,
running: isJobRunning(job.id),
isOnlyShared:
!isUserAdmin &&
job.userId !== req.session.currentUser &&
job.shared_with_user.includes(req.session.currentUser),
};
});
res.send();
});
jobRouter.get('/data', async (req, res) => {
const { page, pageSize = 50, activityFilter, sortfield = null, sortdir = 'asc', freeTextFilter } = req.query || {};
// normalize booleans
const toBool = (v) => {
if (v === true || v === 'true' || v === 1 || v === '1') return true;
if (v === false || v === 'false' || v === 0 || v === '0') return false;
return null;
};
const normalizedActivity = toBool(activityFilter);
const queryResult = jobStorage.queryJobs({
page: page ? parseInt(page, 10) : 1,
pageSize: pageSize ? parseInt(pageSize, 10) : 50,
freeTextFilter: freeTextFilter || null,
activityFilter: normalizedActivity,
sortField: sortfield || null,
sortDir: sortdir === 'desc' ? 'desc' : 'asc',
userId: req.session.currentUser,
isAdmin: isAdmin(req),
job.userId !== request.session.currentUser &&
job.shared_with_user.includes(request.session.currentUser),
}));
});
const isUserAdmin = isAdmin(req);
fastify.get('/data', async (request) => {
const {
page,
pageSize = 50,
activityFilter,
sortfield = null,
sortdir = 'asc',
freeTextFilter,
} = request.query || {};
// Map result to include runtime status
queryResult.result = queryResult.result.map((job) => {
return {
const toBool = (v) => {
if (v === true || v === 'true' || v === 1 || v === '1') return true;
if (v === false || v === 'false' || v === 0 || v === '0') return false;
return null;
};
const normalizedActivity = toBool(activityFilter);
const queryResult = jobStorage.queryJobs({
page: page ? parseInt(page, 10) : 1,
pageSize: pageSize ? parseInt(pageSize, 10) : 50,
freeTextFilter: freeTextFilter || null,
activityFilter: normalizedActivity,
sortField: sortfield || null,
sortDir: sortdir === 'desc' ? 'desc' : 'asc',
userId: request.session.currentUser,
isAdmin: isAdmin(request),
});
const isUserAdmin = isAdmin(request);
queryResult.result = queryResult.result.map((job) => ({
...job,
running: isJobRunning(job.id),
isOnlyShared:
!isUserAdmin &&
job.userId !== req.session.currentUser &&
job.shared_with_user.includes(req.session.currentUser),
};
job.userId !== request.session.currentUser &&
job.shared_with_user.includes(request.session.currentUser),
}));
return queryResult;
});
res.body = queryResult;
res.send();
});
// Server-Sent Events for real-time job status updates
fastify.get('/events', async (request, reply) => {
const userId = request.session?.currentUser;
if (userId == null) {
return reply.code(401).send({ message: 'Unauthorized' });
}
reply.hijack();
const raw = reply.raw;
raw.setHeader('Content-Type', 'text/event-stream');
raw.setHeader('Cache-Control', 'no-cache');
raw.setHeader('Connection', 'keep-alive');
// Server-Sent Events for job status updates
jobRouter.get('/events', async (req, res) => {
const userId = req.session.currentUser;
if (userId == null) {
res.send({ message: 'Unauthorized' }, 401);
return;
}
// SSE headers
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
try {
// Initial comment to establish stream
res.write(': connected\n\n');
addSseClient(userId, res);
// Cleanup on close/aborted
const onClose = () => removeClient(userId, res);
// restana exposes original req/res; use both close and finish
req.on('close', onClose);
req.on('aborted', onClose);
res.on('close', onClose);
} catch (e) {
logger.error('Error establishing SSE connection', e);
try {
res.end();
} catch {
//noop
raw.write(': connected\n\n');
addSseClient(userId, raw);
const onClose = () => removeClient(userId, raw);
request.raw.on('close', onClose);
} catch (e) {
logger.error('Error establishing SSE connection', e);
try {
raw.end();
} catch {
/* noop */
}
}
}
});
});
jobRouter.post('/startAll', async (req, res) => {
try {
const userId = req.session.currentUser;
// Emit only the userId; handler will decide based on admin/ownership
bus.emit('jobs:runAll', { userId });
res.send({ message: 'Run all accepted' }, 202);
} catch (err) {
logger.error('Failed to trigger startAll', err);
res.send({ message: 'Unexpected error' }, 500);
}
});
// Trigger a single job run
jobRouter.post('/:jobId/run', async (req, res) => {
const { jobId } = req.params;
try {
const job = jobStorage.getJob(jobId);
if (!job) {
res.send({ message: 'Job not found' }, 404);
return;
fastify.post('/startAll', async (request, reply) => {
try {
const userId = request.session.currentUser;
bus.emit('jobs:runAll', { userId });
return reply.code(202).send({ message: 'Run all accepted' });
} catch (err) {
logger.error('Failed to trigger startAll', err);
return reply.code(500).send({ message: 'Unexpected error' });
}
if (!doesJobBelongsToUser(job, req)) {
res.send({ message: 'You are trying to run a job that is not associated to your user' }, 403);
return;
}
if (isJobRunning(jobId)) {
res.send({ message: 'Job is already running' }, 409);
return;
}
// fire and forget; actual execution handled by index.js listener
bus.emit('jobs:runOne', { jobId });
res.send({ message: 'Job run accepted' }, 202);
} catch (error) {
logger.error(error);
res.send({ message: 'Unexpected error triggering job' }, 500);
}
});
});
jobRouter.post('/', async (req, res) => {
const {
provider,
notificationAdapter,
name,
blacklist = [],
jobId,
enabled,
shareWithUsers = [],
spatialFilter = null,
specFilter = null,
} = req.body;
const settings = await getSettings();
try {
let jobFromDb = jobStorage.getJob(jobId);
if (jobFromDb && !doesJobBelongsToUser(jobFromDb, req)) {
res.send(new Error('You are trying to change a job that is not associated to your user.'));
return;
fastify.post('/:jobId/run', async (request, reply) => {
const { jobId } = request.params;
try {
const job = jobStorage.getJob(jobId);
if (!job) {
return reply.code(404).send({ message: 'Job not found' });
}
if (!doesJobBelongsToUser(job, request)) {
return reply.code(403).send({ message: 'You are trying to run a job that is not associated to your user' });
}
if (isJobRunning(jobId)) {
return reply.code(409).send({ message: 'Job is already running' });
}
bus.emit('jobs:runOne', { jobId });
return reply.code(202).send({ message: 'Job run accepted' });
} catch (error) {
logger.error(error);
return reply.code(500).send({ message: 'Unexpected error triggering job' });
}
});
if (settings.demoMode && !isAdmin(req) && jobFromDb && jobFromDb.name === DEMO_JOB_NAME) {
res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
return;
}
jobStorage.upsertJob({
userId: req.session.currentUser,
jobId,
enabled,
name,
blacklist,
fastify.post('/', async (request, reply) => {
const {
provider,
notificationAdapter,
shareWithUsers,
spatialFilter,
specFilter,
});
} catch (error) {
res.send(new Error(error));
logger.error(error);
}
res.send();
});
name,
blacklist = [],
jobId,
enabled,
shareWithUsers = [],
spatialFilter = null,
specFilter = null,
} = request.body;
const settings = await getSettings();
try {
const jobFromDb = jobStorage.getJob(jobId);
jobRouter.delete('', async (req, res) => {
const { jobId } = req.body;
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (settings.demoMode && !isAdmin(req) && job.name === DEMO_JOB_NAME) {
res.send(new Error('Sorry, but you cannot remove the Demo Job ;)'));
return;
}
if (jobFromDb && !doesJobBelongsToUser(jobFromDb, request)) {
return reply.code(403).send({ error: 'You are trying to change a job that is not associated to your user.' });
}
if (!doesJobBelongsToUser(job, req)) {
res.send(new Error('You are trying to remove a job that is not associated to your user'));
} else {
jobStorage.removeJob(jobId);
}
} catch (error) {
res.send(new Error(error));
logger.error(error);
}
res.send();
});
jobRouter.put('/:jobId/status', async (req, res) => {
const { status } = req.body;
const { jobId } = req.params;
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (settings.demoMode && !isAdmin(request) && jobFromDb && jobFromDb.name === DEMO_JOB_NAME) {
return reply.code(403).send({ error: 'Sorry, but you cannot change the Status of our Demo Job ;)' });
}
if (settings.demoMode && !isAdmin(req) && job.name === DEMO_JOB_NAME) {
res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
return;
}
if (!doesJobBelongsToUser(job, req)) {
res.send(new Error('You are trying change a job that is not associated to your user'));
} else {
jobStorage.setJobStatus({
jobStorage.upsertJob({
userId: request.session.currentUser,
jobId,
status,
enabled,
name,
blacklist,
provider,
notificationAdapter,
shareWithUsers,
spatialFilter,
specFilter,
});
} catch (error) {
logger.error(error);
return reply.code(500).send({ error: error.message });
}
} catch (error) {
res.send(new Error(error));
logger.error(error);
}
res.send();
});
return reply.send();
});
jobRouter.get('/shareableUserList', async (req, res) => {
const currentUser = req.session.currentUser;
const users = userStorage.getUsers(false);
res.body = users
.filter((user) => !user.isAdmin && user.id !== currentUser)
.map((user) => ({
id: user.id,
name: user.username,
}));
res.send();
});
export { jobRouter };
fastify.delete('/', async (request, reply) => {
const { jobId } = request.body;
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (settings.demoMode && !isAdmin(request) && job.name === DEMO_JOB_NAME) {
return reply.code(403).send({ error: 'Sorry, but you cannot remove the Demo Job ;)' });
}
if (!doesJobBelongsToUser(job, request)) {
return reply.code(403).send({ error: 'You are trying to remove a job that is not associated to your user' });
}
jobStorage.removeJob(jobId);
} catch (error) {
logger.error(error);
return reply.code(500).send({ error: error.message });
}
return reply.send();
});
fastify.put('/:jobId/status', async (request, reply) => {
const { status } = request.body;
const { jobId } = request.params;
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (settings.demoMode && !isAdmin(request) && job.name === DEMO_JOB_NAME) {
return reply.code(403).send({ error: 'Sorry, but you cannot change the Status of our Demo Job ;)' });
}
if (!doesJobBelongsToUser(job, request)) {
return reply.code(403).send({ error: 'You are trying change a job that is not associated to your user' });
}
jobStorage.setJobStatus({ jobId, status });
} catch (error) {
logger.error(error);
return reply.code(500).send({ error: error.message });
}
return reply.send();
});
fastify.get('/shareableUserList', async (request) => {
const currentUser = request.session.currentUser;
const users = userStorage.getUsers(false);
return users
.filter((user) => !user.isAdmin && user.id !== currentUser)
.map((user) => ({
id: user.id,
name: user.username,
}));
});
}

View File

@@ -3,7 +3,6 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import * as listingStorage from '../../services/storage/listingsStorage.js';
import * as watchListStorage from '../../services/storage/watchListStorage.js';
import { isAdmin as isAdminFn } from '../security.js';
@@ -11,129 +10,173 @@ import logger from '../../services/logger.js';
import { nullOrEmpty } from '../../utils.js';
import { getJobs } from '../../services/storage/jobStorage.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
const service = restana();
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function listingsPlugin(fastify) {
fastify.get('/table', async (request) => {
const {
page,
pageSize = 50,
activityFilter,
jobNameFilter,
providerFilter,
watchListFilter,
statusFilter,
sortfield = null,
sortdir = 'asc',
freeTextFilter,
} = request.query || {};
const listingsRouter = service.newRouter();
const toBool = (v) => {
if (v === true || v === 'true' || v === 1 || v === '1') return true;
if (v === false || v === 'false' || v === 0 || v === '0') return false;
return null;
};
const normalizedActivity = toBool(activityFilter);
const normalizedWatch = toBool(watchListFilter);
const allowedStatuses = ['applied', 'rejected', 'accepted', 'none'];
const normalizedStatus =
typeof statusFilter === 'string' && allowedStatuses.includes(statusFilter.toLowerCase())
? statusFilter.toLowerCase()
: undefined;
listingsRouter.get('/table', async (req, res) => {
const {
page,
pageSize = 50,
activityFilter,
jobNameFilter,
providerFilter,
watchListFilter,
sortfield = null,
sortdir = 'asc',
freeTextFilter,
} = req.query || {};
let jobFilter = null;
let jobIdFilter = null;
const jobs = getJobs();
if (!nullOrEmpty(jobNameFilter)) {
const job = jobs.find((j) => j.id === jobNameFilter);
jobFilter = job != null ? job.name : null;
jobIdFilter = job != null ? job.id : null;
}
// normalize booleans (accept true, 'true', 1, '1' for true; false, 'false', 0, '0' for false)
const toBool = (v) => {
if (v === true || v === 'true' || v === 1 || v === '1') return true;
if (v === false || v === 'false' || v === 0 || v === '0') return false;
return null;
};
const normalizedActivity = toBool(activityFilter);
const normalizedWatch = toBool(watchListFilter);
let jobFilter = null;
let jobIdFilter = null;
const jobs = getJobs();
if (!nullOrEmpty(jobNameFilter)) {
const job = jobs.find((j) => j.id === jobNameFilter);
jobFilter = job != null ? job.name : null;
jobIdFilter = job != null ? job.id : null;
}
res.body = listingStorage.queryListings({
page: page ? parseInt(page, 10) : 1,
pageSize: pageSize ? parseInt(pageSize, 10) : 50,
freeTextFilter: freeTextFilter || null,
activityFilter: normalizedActivity,
jobNameFilter: jobFilter,
jobIdFilter: jobIdFilter,
providerFilter,
watchListFilter: normalizedWatch,
sortField: sortfield || null,
sortDir: sortdir === 'desc' ? 'desc' : 'asc',
userId: req.session.currentUser,
isAdmin: isAdminFn(req),
return listingStorage.queryListings({
page: page ? parseInt(page, 10) : 1,
pageSize: pageSize ? parseInt(pageSize, 10) : 50,
freeTextFilter: freeTextFilter || null,
activityFilter: normalizedActivity,
jobNameFilter: jobFilter,
jobIdFilter: jobIdFilter,
providerFilter,
watchListFilter: normalizedWatch,
statusFilter: normalizedStatus,
sortField: sortfield || null,
sortDir: sortdir === 'desc' ? 'desc' : 'asc',
userId: request.session.currentUser,
isAdmin: isAdminFn(request),
});
});
res.send();
});
listingsRouter.get('/map', async (req, res) => {
const { jobId } = req.query || {};
res.body = listingStorage.getListingsForMap({
jobId: nullOrEmpty(jobId) ? null : jobId,
userId: req.session.currentUser,
isAdmin: isAdminFn(req),
fastify.get('/map', async (request) => {
const { jobId } = request.query || {};
return listingStorage.getListingsForMap({
jobId: nullOrEmpty(jobId) ? null : jobId,
userId: request.session.currentUser,
isAdmin: isAdminFn(request),
});
});
res.send();
});
listingsRouter.get('/:listingId', async (req, res) => {
const { listingId } = req.params;
const listing = listingStorage.getListingById(listingId, req.session.currentUser, isAdminFn(req));
if (!listing) {
res.statusCode = 404;
res.body = { message: 'Listing not found' };
return res.send();
}
res.body = listing;
res.send();
});
fastify.get('/:listingId', async (request, reply) => {
const { listingId } = request.params;
const listing = listingStorage.getListingById(listingId, request.session.currentUser, isAdminFn(request));
if (!listing) {
return reply.code(404).send({ message: 'Listing not found' });
}
return listing;
});
// Toggle watch state for the current user on a listing
listingsRouter.post('/watch', async (req, res) => {
try {
const { listingId } = req.body || {};
const userId = req.session?.currentUser;
fastify.post('/watch', async (request, reply) => {
try {
const { listingId } = request.body || {};
const userId = request.session?.currentUser;
if (!listingId || !userId) {
return reply.code(400).send({ message: 'listingId or user not provided' });
}
watchListStorage.toggleWatch(listingId, userId);
} catch (error) {
logger.error(error);
return reply.code(500).send({ message: 'Failed to toggle watch' });
}
return reply.send();
});
fastify.post('/:listingId/notes', async (request, reply) => {
const { listingId } = request.params || {};
const { notes } = request.body || {};
const userId = request.session?.currentUser;
if (!listingId || !userId) {
res.statusCode = 400;
res.body = { message: 'listingId or user not provided' };
return res.send();
return reply.code(400).send({ message: 'listingId or user not provided' });
}
watchListStorage.toggleWatch(listingId, userId);
} catch (error) {
logger.error(error);
res.statusCode = 500;
res.body = { message: 'Failed to toggle watch' };
}
res.send();
});
listingsRouter.delete('/job', async (req, res) => {
const { jobId, hardDelete = false } = req.body;
const settings = await getSettings();
try {
if (settings.demoMode && !isAdminFn(req)) {
res.send(new Error('Sorry, but you cannot remove listings in demo mode ;)'));
return;
try {
const changes = listingStorage.setListingNotes(listingId, typeof notes === 'string' ? notes : null);
if (changes === 0) {
return reply.code(404).send({ message: 'Listing not found' });
}
} catch (error) {
logger.error(error);
return reply.code(500).send({ message: 'Failed to update listing notes' });
}
listingStorage.deleteListingsByJobId(jobId, hardDelete);
} catch (error) {
res.send(new Error(error));
logger.error(error);
}
res.send();
});
await trackPoi(TRACKING_POIS.NOTES_CREATE);
return reply.send();
});
listingsRouter.delete('/', async (req, res) => {
const { ids, hardDelete = false } = req.body;
try {
if (Array.isArray(ids) && ids.length > 0) {
listingStorage.deleteListingsById(ids, hardDelete);
fastify.post('/:listingId/status', async (request, reply) => {
const { listingId } = request.params || {};
const { status } = request.body || {};
const userId = request.session?.currentUser;
if (!listingId || !userId) {
return reply.code(400).send({ message: 'listingId or user not provided' });
}
} catch (error) {
res.send(new Error(error));
logger.error(error);
}
res.send();
});
const allowed = ['applied', 'rejected', 'accepted'];
const normalized = status == null ? null : String(status).toLowerCase();
if (normalized != null && !allowed.includes(normalized)) {
return reply.code(400).send({ message: `Invalid status: ${status}` });
}
try {
const changes = listingStorage.setListingStatus(listingId, normalized);
await trackPoi(TRACKING_POIS.USING_LISTING_STATUS);
if (changes === 0) {
return reply.code(404).send({ message: 'Listing not found' });
}
if (normalized != null) {
watchListStorage.ensureWatch(listingId, userId);
}
} catch (error) {
logger.error(error);
return reply.code(500).send({ message: 'Failed to update listing status' });
}
return reply.send();
});
export { listingsRouter };
fastify.delete('/job', async (request, reply) => {
const { jobId, hardDelete = false } = request.body;
const settings = await getSettings();
try {
if (settings.demoMode && !isAdminFn(request)) {
return reply.code(403).send({ error: 'Sorry, but you cannot remove listings in demo mode ;)' });
}
listingStorage.deleteListingsByJobId(jobId, hardDelete);
} catch (error) {
logger.error(error);
return reply.code(500).send({ error: error.message });
}
return reply.send();
});
fastify.delete('/', async (request, reply) => {
const { ids, hardDelete = false } = request.body;
try {
if (Array.isArray(ids) && ids.length > 0) {
listingStorage.deleteListingsById(ids, hardDelete);
}
} catch (error) {
logger.error(error);
return reply.code(500).send({ error: error.message });
}
return reply.send();
});
}

View File

@@ -3,7 +3,6 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import * as userStorage from '../../services/storage/userStorage.js';
import * as hasher from '../../services/security/hash.js';
import { trackDemoAccessed } from '../../services/tracking/Tracker.js';
@@ -11,12 +10,12 @@ import logger from '../../services/logger.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
const MAX_LOGIN_ATTEMPTS = 10;
const LOGIN_WINDOW_MS = 15 * 60 * 1000; // 15 minutes
const loginAttempts = new Map(); // ip -> { count, firstAttempt }
const LOGIN_WINDOW_MS = 15 * 60 * 1000;
const loginAttempts = new Map();
function getClientIp(req) {
const forwarded = req.headers['x-forwarded-for'];
return (forwarded ? forwarded.split(',')[0] : req.socket?.remoteAddress) || 'unknown';
function getClientIp(request) {
const forwarded = request.headers['x-forwarded-for'];
return (forwarded ? forwarded.split(',')[0] : request.socket?.remoteAddress) || 'unknown';
}
function isRateLimited(ip) {
@@ -30,53 +29,51 @@ function isRateLimited(ip) {
return record.count > MAX_LOGIN_ATTEMPTS;
}
const service = restana();
const loginRouter = service.newRouter();
loginRouter.get('/user', async (req, res) => {
const currentUserId = req.session.currentUser;
const currentUser = currentUserId == null ? null : userStorage.getUser(currentUserId);
if (currentUser == null) {
res.body = {};
} else {
res.body = {
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function loginPlugin(fastify) {
fastify.get('/user', async (request) => {
const currentUserId = request.session?.currentUser;
const currentUser = currentUserId == null ? null : userStorage.getUser(currentUserId);
if (currentUser == null) {
return {};
}
return {
userId: currentUser.id,
isAdmin: currentUser.isAdmin,
};
}
res.send();
});
loginRouter.post('/', async (req, res) => {
const ip = getClientIp(req);
if (isRateLimited(ip)) {
logger.error(`Login rate limit exceeded for IP ${ip}`);
res.send(429);
return;
}
const settings = await getSettings();
const { username, password } = req.body;
const user = userStorage.getUsers(true).find((user) => user.username === username);
if (user == null) {
res.send(401);
return;
}
if (user.password === hasher.hash(password)) {
if (settings.demoMode) {
await trackDemoAccessed();
}
});
req.session.currentUser = user.id;
req.session.createdAt = Date.now();
loginAttempts.delete(ip);
userStorage.setLastLoginToNow({ userId: user.id });
res.send(200);
return;
} else {
logger.error(`User ${username} tried to login, but password was wrong.`);
}
res.send(401);
});
loginRouter.post('/logout', async (req, res) => {
req.session = null;
res.send(200);
});
export { loginRouter };
fastify.post('/', async (request, reply) => {
const ip = getClientIp(request);
if (isRateLimited(ip)) {
logger.error(`Login rate limit exceeded for IP ${ip}`);
return reply.code(429).send();
}
const settings = await getSettings();
const { username, password } = request.body;
const user = userStorage.getUsers(true).find((u) => u.username === username);
if (user == null) {
return reply.code(401).send();
}
if (user.password === hasher.hash(password)) {
if (settings.demoMode) {
await trackDemoAccessed();
}
request.session.currentUser = user.id;
request.session.createdAt = Date.now();
loginAttempts.delete(ip);
userStorage.setLastLoginToNow({ userId: user.id });
return reply.code(200).send();
} else {
logger.error(`User ${username} tried to login, but password was wrong.`);
}
return reply.code(401).send();
});
fastify.post('/logout', async (request, reply) => {
await request.session.destroy();
return reply.code(200).send();
});
}

View File

@@ -4,62 +4,64 @@
*/
import fs from 'fs';
import restana from 'restana';
import logger from '../../services/logger.js';
const service = restana();
const notificationAdapterRouter = service.newRouter();
const notificationAdapterList = fs.readdirSync('./lib//notification/adapter').filter((file) => file.endsWith('.js'));
const notificationAdapter = await Promise.all(
notificationAdapterList.map(async (pro) => {
return await import(`../../notification/adapter/${pro}`);
}),
);
notificationAdapterRouter.post('/try', async (req, res) => {
const { id, fields } = req.body;
const adapter = notificationAdapter.find((adapter) => adapter.config.id === id);
if (adapter == null) {
res.send(404);
}
const notificationConfig = [];
const notificationObject = {};
Object.keys(fields).forEach((key) => {
notificationObject[key] = fields[key].value;
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function notificationAdapterPlugin(fastify) {
fastify.get('/', async () => {
return notificationAdapter.map((adapter) => adapter.config).filter(Boolean);
});
notificationConfig.push({
fields: { ...notificationObject },
enabled: true,
id,
});
try {
await adapter.send({
serviceName: 'TestCall',
newListings: [
{
address: 'Heidestrasse 17, 51147 Köln',
description: exampleDescription,
id: '1',
imageUrl: 'https://placehold.co/600x400/png',
price: '1.000 €',
size: '76 m²',
title: 'Stilvolle gepflegte 3-Raum-Wohnung mit gehobener Innenausstattung',
url: 'https://www.orange-coding.net',
},
],
notificationConfig,
jobKey: 'TestJob',
fastify.post('/try', async (request, reply) => {
const { id, fields } = request.body;
const adapter = notificationAdapter.find((adapter) => adapter.config.id === id);
if (adapter == null) {
return reply.code(404).send();
}
const notificationConfig = [];
const notificationObject = {};
Object.keys(fields).forEach((key) => {
notificationObject[key] = fields[key].value;
});
res.send();
} catch (Exception) {
logger.error('Error during notification adapter test:', Exception);
res.send(new Error(Exception));
}
});
notificationAdapterRouter.get('/', async (req, res) => {
res.body = notificationAdapter.map((adapter) => adapter.config);
res.send();
});
export { notificationAdapterRouter };
notificationConfig.push({
fields: { ...notificationObject },
enabled: true,
id,
});
try {
await adapter.send({
serviceName: 'TestCall',
newListings: [
{
address: 'Heidestrasse 17, 51147 Köln',
description: exampleDescription,
id: '1',
imageUrl: 'https://placehold.co/600x400/png',
price: '1.000 €',
size: '76 m²',
title: 'Stilvolle gepflegte 3-Raum-Wohnung mit gehobener Innenausstattung',
url: 'https://www.orange-coding.net',
},
],
notificationConfig,
jobKey: 'TestJob',
});
return reply.send();
} catch (Exception) {
logger.error('Error during notification adapter test:', Exception);
return reply.code(500).send({ error: String(Exception) });
}
});
}
const exampleDescription = `
Wohnungstyp: Etagenwohnung
@@ -94,7 +96,7 @@ Die Wohnung ist ideal für Paare oder kleine Familien geeignet.
Ausstattung:
- neuer hochwertiger Bodenbelag (Holzoptik) in allen Räumen außer Bad/Küche
- sonniger Balkon (Süd)
- Tiefgaragenstellplatz
- Tiefgaragenstellplatz
- Kellerabteil
- gepflegtes Mehrfamilienhaus
@@ -104,7 +106,7 @@ Vermietung direkt vom Eigentümer - provisionsfrei!
Lage:
• Park: 1 Minute zu Fuß
• S-Bahn Station: 2 Minuten zu Fuß
• S-Bahn Station: 2 Minuten zu Fuß
• Supermärkte, Restaurants, täglicher Bedarf in der Nähe
• Gute Anbindung Richtung Großstadt und Flughafen
`;

View File

@@ -4,17 +4,15 @@
*/
import fs from 'fs';
import restana from 'restana';
const service = restana();
const providerRouter = service.newRouter();
const providerList = fs.readdirSync('./lib/provider').filter((file) => file.endsWith('.js'));
const provider = await Promise.all(
providerList.map(async (pro) => {
return await import(`../../provider/${pro}`);
}),
);
providerRouter.get('/', async (req, res) => {
res.body = provider.map((p) => p.metaInformation);
res.send();
});
export { providerRouter };
const providers = await Promise.all(providerList.map(async (pro) => import(`../../provider/${pro}`)));
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function providerPlugin(fastify) {
fastify.get('/', async () => {
return providers.map((p) => p.metaInformation);
});
}

View File

@@ -3,35 +3,29 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
import logger from '../../services/logger.js';
const service = restana();
const trackingRouter = service.newRouter();
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function trackingPlugin(fastify) {
fastify.get('/trackingPois', async () => {
return TRACKING_POIS;
});
trackingRouter.get('/trackingPois', async (req, res) => {
res.body = TRACKING_POIS;
res.send();
});
trackingRouter.post('/poi', async (req, res) => {
const { poi } = req.body;
if (!poi) {
res.statusCode = 400;
res.send({ error: 'Feature name is required' });
return;
}
try {
await trackPoi(poi);
res.send({ success: true });
} catch (error) {
logger.error('Error tracking feature', error);
res.statusCode = 500;
res.send({ error: error.message });
}
});
export { trackingRouter };
fastify.post('/poi', async (request, reply) => {
const { poi } = request.body;
if (!poi) {
return reply.code(400).send({ error: 'Feature name is required' });
}
try {
await trackPoi(poi);
return { success: true };
} catch (error) {
logger.error('Error tracking feature', error);
return reply.code(500).send({ error: error.message });
}
});
}

View File

@@ -3,82 +3,73 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import * as userStorage from '../../services/storage/userStorage.js';
import * as jobStorage from '../../services/storage/jobStorage.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin as isAdminUser } from '../security.js';
const service = restana();
const userRouter = service.newRouter();
function checkIfAnyAdminAfterRemovingUser(userIdToBeRemoved, allUser) {
return allUser.filter((user) => user.id !== userIdToBeRemoved && user.isAdmin).length > 0;
}
function checkIfUserToBeRemovedIsLoggedIn(userIdToBeRemoved, req) {
return req.session.currentUser === userIdToBeRemoved;
function checkIfUserToBeRemovedIsLoggedIn(userIdToBeRemoved, request) {
return request.session.currentUser === userIdToBeRemoved;
}
const nullOrEmpty = (str) => str == null || str.length === 0;
userRouter.get('/', async (req, res) => {
res.body = userStorage.getUsers(false);
res.send();
});
userRouter.get('/:userId', async (req, res) => {
const { userId } = req.params;
res.body = userStorage.getUser(userId);
res.send();
});
userRouter.delete('/', async (req, res) => {
const settings = await getSettings();
if (settings.demoMode && !isAdminUser(req)) {
res.send(new Error('In demo mode, it is not allowed to remove user.'));
return;
}
const { userId } = req.body;
const allUser = userStorage.getUsers(false);
if (!checkIfAnyAdminAfterRemovingUser(userId, allUser)) {
res.send(new Error('You are trying to remove the last admin user. This is prohibited.'));
return;
}
if (checkIfUserToBeRemovedIsLoggedIn(userId, req)) {
res.send(new Error('You are trying to remove yourself. This is prohibited.'));
return;
}
//TODO: Remove also analytics
jobStorage.removeJobsByUserId(userId);
userStorage.removeUser(userId);
res.send();
});
userRouter.post('/', async (req, res) => {
const settings = await getSettings();
if (settings.demoMode && !isAdminUser(req)) {
res.send(new Error('In demo mode, it is not allowed to change or add user.'));
return;
}
const { username, password, password2, isAdmin, userId } = req.body;
if (password !== password2) {
res.send(new Error('Passwords does not match'));
return;
}
if (nullOrEmpty(username) || nullOrEmpty(password) || nullOrEmpty(password2)) {
res.send(new Error('Username and password are mandatory.'));
return;
}
const allUser = userStorage.getUsers(false);
if (!isAdmin && !checkIfAnyAdminAfterRemovingUser(userId, allUser)) {
res.send(
new Error('You cannot change the admin flag for this user as otherwise, there is no other user in the system'),
);
return;
}
userStorage.upsertUser({
userId,
username,
password,
isAdmin,
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function userPlugin(fastify) {
fastify.get('/', async () => {
return userStorage.getUsers(false);
});
res.send();
});
export { userRouter };
fastify.get('/:userId', async (request) => {
const { userId } = request.params;
return userStorage.getUser(userId);
});
fastify.delete('/', async (request, reply) => {
const settings = await getSettings();
if (settings.demoMode && !isAdminUser(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to remove user.' });
}
const { userId } = request.body;
const allUser = userStorage.getUsers(false);
if (!checkIfAnyAdminAfterRemovingUser(userId, allUser)) {
return reply.code(400).send({ error: 'You are trying to remove the last admin user. This is prohibited.' });
}
if (checkIfUserToBeRemovedIsLoggedIn(userId, request)) {
return reply.code(400).send({ error: 'You are trying to remove yourself. This is prohibited.' });
}
jobStorage.removeJobsByUserId(userId);
userStorage.removeUser(userId);
return reply.send();
});
fastify.post('/', async (request, reply) => {
const settings = await getSettings();
if (settings.demoMode && !isAdminUser(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change or add user.' });
}
const { username, password, password2, isAdmin, userId } = request.body;
if (password !== password2) {
return reply.code(400).send({ error: 'Passwords do not match.' });
}
if (nullOrEmpty(username) || nullOrEmpty(password) || nullOrEmpty(password2)) {
return reply.code(400).send({ error: 'Username and password are mandatory.' });
}
const allUser = userStorage.getUsers(false);
if (!isAdmin && !checkIfAnyAdminAfterRemovingUser(userId, allUser)) {
return reply.code(400).send({
error: 'You cannot change the admin flag for this user as otherwise, there is no other user in the system',
});
}
userStorage.upsertUser({ userId, username, password, isAdmin });
return reply.send();
});
}

View File

@@ -3,7 +3,6 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import SqliteConnection from '../../services/storage/SqliteConnection.js';
import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
import { isAdmin } from '../security.js';
@@ -16,113 +15,164 @@ import { TRACKING_POIS } from '../../TRACKING_POIS.js';
import logger from '../../services/logger.js';
import { runGeoCordTask } from '../../services/crons/geocoding-cron.js';
const service = restana();
const userSettingsRouter = service.newRouter();
userSettingsRouter.get('/', async (req, res) => {
const userId = req.session.currentUser;
const rows = SqliteConnection.query('SELECT name, value FROM settings WHERE user_id = @userId', { userId });
const settings = {};
for (const r of rows) {
settings[r.name] = fromJson(r.value, null);
}
res.body = settings;
res.send();
});
userSettingsRouter.get('/autocomplete', async (req, res) => {
const { q } = req.query;
try {
const results = await autocompleteAddress(q);
res.body = results;
res.send();
} catch (error) {
res.statusCode = 500;
res.send({ error: error.message });
}
});
userSettingsRouter.post('/home-address', async (req, res) => {
const userId = req.session.currentUser;
const { home_address } = req.body;
const settings = await getSettings();
if (settings.demoMode && !isAdmin(req)) {
res.send(new Error('In demo mode, it is not allowed to change the home address.'));
return;
}
try {
if (home_address) {
await trackPoi(TRACKING_POIS.DISTANCE_ADDRESS_ENTERED);
const coords = await geocodeAddress(home_address);
if (coords && coords.lat !== -1) {
upsertSettings({ home_address: { address: home_address, coords } }, userId);
resetGeocoordinatesAndDistanceForUser(userId);
//we do NOT wait for this to finish, as we don't want to block the response
runGeoCordTask();
res.send({ success: true, coords });
} else {
res.statusCode = 400;
res.send({ error: 'Could not geocode address' });
}
} else {
upsertSettings({ home_address: null }, userId);
res.send({ success: true });
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function userSettingsPlugin(fastify) {
fastify.get('/', async (request) => {
const userId = request.session.currentUser;
const rows = SqliteConnection.query('SELECT name, value FROM settings WHERE user_id = @userId', { userId });
const settings = {};
for (const r of rows) {
settings[r.name] = fromJson(r.value, null);
}
} catch (error) {
logger.error('Error updating home address settings', error);
res.statusCode = 500;
res.send({ error: error.message });
}
});
return settings;
});
userSettingsRouter.post('/news-hash', async (req, res) => {
const userId = req.session.currentUser;
const { news_hash } = req.body;
fastify.get('/autocomplete', async (request, reply) => {
const { q } = request.query;
try {
const results = await autocompleteAddress(q);
return results;
} catch (error) {
return reply.code(500).send({ error: error.message });
}
});
const globalSettings = await getSettings();
if (globalSettings.demoMode && !isAdmin(req)) {
res.statusCode = 403;
res.send({ error: 'In demo mode, it is not allowed to change settings.' });
return;
}
fastify.post('/home-address', async (request, reply) => {
const userId = request.session.currentUser;
const { home_address } = request.body;
const settings = await getSettings();
try {
upsertSettings({ news_hash }, userId);
res.send({ success: true });
} catch (error) {
logger.error('Error updating news hash', error);
res.statusCode = 500;
res.send({ error: error.message });
}
});
if (settings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change the home address.' });
}
userSettingsRouter.post('/provider-details', async (req, res) => {
const userId = req.session.currentUser;
const { provider_details } = req.body;
try {
if (home_address) {
await trackPoi(TRACKING_POIS.DISTANCE_ADDRESS_ENTERED);
const coords = await geocodeAddress(home_address);
if (coords && coords.lat !== -1) {
upsertSettings({ home_address: { address: home_address, coords } }, userId);
resetGeocoordinatesAndDistanceForUser(userId);
runGeoCordTask();
return { success: true, coords };
} else {
return reply.code(400).send({ error: 'Could not geocode address' });
}
} else {
upsertSettings({ home_address: null }, userId);
return { success: true };
}
} catch (error) {
logger.error('Error updating home address settings', error);
return reply.code(500).send({ error: error.message });
}
});
const globalSettings = await getSettings();
if (globalSettings.demoMode && !isAdmin(req)) {
res.statusCode = 403;
res.send({ error: 'In demo mode, it is not allowed to change settings.' });
return;
}
fastify.post('/news-hash', async (request, reply) => {
const userId = request.session.currentUser;
const { news_hash } = request.body;
if (!Array.isArray(provider_details)) {
res.statusCode = 400;
res.send({ error: 'provider_details must be an array of provider ids.' });
return;
}
const globalSettings = await getSettings();
if (globalSettings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
}
try {
upsertSettings({ provider_details }, userId);
res.send({ success: true });
} catch (error) {
logger.error('Error updating provider details setting', error);
res.statusCode = 500;
res.send({ error: error.message });
}
});
try {
upsertSettings({ news_hash }, userId);
return { success: true };
} catch (error) {
logger.error('Error updating news hash', error);
return reply.code(500).send({ error: error.message });
}
});
export { userSettingsRouter };
fastify.post('/provider-details', async (request, reply) => {
const userId = request.session.currentUser;
const { provider_details } = request.body;
const globalSettings = await getSettings();
if (globalSettings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
}
if (!Array.isArray(provider_details)) {
return reply.code(400).send({ error: 'provider_details must be an array of provider ids.' });
}
try {
upsertSettings({ provider_details }, userId);
return { success: true };
} catch (error) {
logger.error('Error updating provider details setting', error);
return reply.code(500).send({ error: error.message });
}
});
fastify.post('/listings-view-mode', async (request, reply) => {
const userId = request.session.currentUser;
const { listings_view_mode } = request.body;
if (listings_view_mode !== 'grid' && listings_view_mode !== 'table') {
return reply.code(400).send({ error: 'listings_view_mode must be "grid" or "table".' });
}
if (listings_view_mode === 'table') {
await trackPoi(TRACKING_POIS.LISTING_TABLE_VIEW);
}
try {
upsertSettings({ listings_view_mode }, userId);
return { success: true };
} catch (error) {
logger.error('Error updating listings view mode setting', error);
return reply.code(500).send({ error: error.message });
}
});
fastify.post('/jobs-view-mode', async (request, reply) => {
const userId = request.session.currentUser;
const { jobs_view_mode } = request.body;
if (jobs_view_mode !== 'grid' && jobs_view_mode !== 'table') {
return reply.code(400).send({ error: 'jobs_view_mode must be "grid" or "table".' });
}
if (jobs_view_mode === 'table') {
await trackPoi(TRACKING_POIS.JOBS_TABLE_VIEW);
}
try {
upsertSettings({ jobs_view_mode }, userId);
return { success: true };
} catch (error) {
logger.error('Error updating jobs view mode setting', error);
return reply.code(500).send({ error: error.message });
}
});
fastify.post('/listing-deletion-preference', async (request, reply) => {
const userId = request.session.currentUser;
const { listing_deletion_preference } = request.body;
const globalSettings = await getSettings();
if (globalSettings.demoMode && !isAdmin(request)) {
return reply.code(403).send({ error: 'In demo mode, it is not allowed to change settings.' });
}
if (listing_deletion_preference == null) {
return reply.code(400).send({ error: 'listing_deletion_preference is required.' });
}
const { skipPrompt, hardDelete } = listing_deletion_preference;
try {
upsertSettings({ listing_deletion_preference: { skipPrompt, hardDelete } }, userId);
return { success: true };
} catch (error) {
logger.error('Error updating listing deletion preference', error);
return reply.code(500).send({ error: error.message });
}
});
}

View File

@@ -3,27 +3,10 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import fetch from 'node-fetch';
import { getPackageVersion } from '../../utils.js';
import semver from 'semver';
const service = restana();
const versionRouter = service.newRouter();
versionRouter.get('/', async (req, res) => {
const versionPayload = await getCurrentVersionFromGithub();
const localFredyVersion = await getPackageVersion();
res.body =
versionPayload == null
? {
newVersion: false,
localFredyVersion,
}
: versionPayload;
res.send();
});
async function getCurrentVersionFromGithub() {
const raw = await fetch('https://api.github.com/repos/orangecoding/fredy/releases/latest');
const data = await raw.json();
@@ -40,4 +23,13 @@ async function getCurrentVersionFromGithub() {
};
}
export { versionRouter };
/**
* @param {import('fastify').FastifyInstance} fastify
*/
export default async function versionPlugin(fastify) {
fastify.get('/', async () => {
const versionPayload = await getCurrentVersionFromGithub();
const localFredyVersion = await getPackageVersion();
return versionPayload ?? { newVersion: false, localFredyVersion };
});
}

View File

@@ -4,53 +4,50 @@
*/
import * as userStorage from '../services/storage/userStorage.js';
import cookieSession from 'cookie-session';
const SESSION_MAX_AGE = 2 * 60 * 60 * 1000; // 2 hours
const unauthorized = (res) => {
return res.send(401);
};
const isUnauthorized = (req) => {
if (req.session.currentUser == null) return true;
if (Date.now() - req.session.createdAt > SESSION_MAX_AGE) {
req.session = null;
return true;
}
/**
* Returns true when the request has no valid, non-expired session.
* @param {import('fastify').FastifyRequest} request
* @returns {boolean}
*/
export function isUnauthorized(request) {
if (!request.session?.currentUser) return true;
if (Date.now() - (request.session.createdAt || 0) > SESSION_MAX_AGE) return true;
return false;
};
const isAdmin = (req) => {
if (!isUnauthorized(req)) {
const user = userStorage.getUser(req.session.currentUser);
return user != null && user.isAdmin;
}
/**
* Returns true when the session belongs to an admin user.
* @param {import('fastify').FastifyRequest} request
* @returns {boolean}
*/
export function isAdmin(request) {
if (isUnauthorized(request)) return false;
const user = userStorage.getUser(request.session.currentUser);
return user != null && user.isAdmin;
}
/**
* Fastify preHandler hook - rejects unauthenticated requests with 401.
* @param {import('fastify').FastifyRequest} request
* @param {import('fastify').FastifyReply} reply
*/
export async function authHook(request, reply) {
if (isUnauthorized(request)) {
reply.code(401).send();
}
return false;
};
const authInterceptor = () => {
return (req, res, next) => {
if (isUnauthorized(req)) {
return unauthorized(res);
} else {
next();
}
};
};
const adminInterceptor = () => {
return (req, res, next) => {
if (!isAdmin(req)) {
return unauthorized(res);
} else {
next();
}
};
};
const cookieSession$0 = (secret) => {
return cookieSession({
name: 'fredy-admin-session',
keys: [secret],
maxAge: SESSION_MAX_AGE,
});
};
export { cookieSession$0 as cookieSession };
export { adminInterceptor };
export { authInterceptor };
export { isUnauthorized };
export { isAdmin };
}
/**
* Fastify preHandler hook - rejects non-admin requests with 401.
* Apply after authHook.
* @param {import('fastify').FastifyRequest} request
* @param {import('fastify').FastifyReply} reply
*/
export async function adminHook(request, reply) {
if (!isAdmin(request)) {
reply.code(401).send();
}
}

View File

@@ -133,7 +133,7 @@ The LLM will automatically call the appropriate Fredy MCP tools and present the
#### Setup
1. Open **Claude Desktop**
2. Go to **Settings → Developer → Edit Config** this opens the `claude_desktop_config.json` file
2. Go to **Settings → Developer → Edit Config** - this opens the `claude_desktop_config.json` file
3. Add the `fredy` server to the `mcpServers` object:
```json
@@ -158,7 +158,7 @@ The LLM will automatically call the appropriate Fredy MCP tools and present the
> - nvm: `/Users/<you>/.nvm/versions/node/<version>/bin/node`
4. Save the file and **restart Claude Desktop**
5. You should see a hammer icon (🔨) in the chat input click it to confirm the Fredy tools are listed
5. You should see a hammer icon (🔨) in the chat input - click it to confirm the Fredy tools are listed
#### Usage
@@ -170,7 +170,7 @@ Once connected, simply ask Claude about your real estate data:
Claude will automatically call the appropriate Fredy MCP tools.
> **Note:** Fredy's main web process does not need to be running the stdio transport opens its own database connection directly. But the SQLite database file must exist and migrations must have been applied.
> **Note:** Fredy's main web process does not need to be running - the stdio transport opens its own database connection directly. But the SQLite database file must exist and migrations must have been applied.
---
@@ -252,7 +252,7 @@ Example list response:
```
**Tool:** list_listings | **Status:** OK
Found **85** listing(s). Showing page 1 of 2 (50 on this page). More pages available use page=2 to continue.
Found **85** listing(s). Showing page 1 of 2 (50 on this page). More pages available - use page=2 to continue.
| ID | Title | Address | Price | Size | Provider | Active | Created | Job |
|----|-------|---------|-------|------|----------|--------|---------|-----|

View File

@@ -49,7 +49,7 @@ export function createMcpServer() {
'list_listings to search listings (supports time filters like createdAfter/createdBefore), ' +
'and get_listing for full details of a single listing. ' +
'Responses are formatted as markdown with a summary, data (tables for lists, key-value for details), and pagination info. ' +
'Always present results to the user as soon as you have them do NOT call the tool again unless you need additional pages or different data.',
'Always present results to the user as soon as you have them - do NOT call the tool again unless you need additional pages or different data.',
},
);
@@ -155,6 +155,12 @@ export function createMcpServer() {
),
sortField: z.string().optional().describe('Sort by: created_at, price, size, provider, title, is_active'),
sortDir: z.string().optional().describe('Sort direction: asc or desc'),
status: z
.enum(['applied', 'rejected', 'accepted', 'none'])
.optional()
.describe(
'Filter by user-set status. "applied", "rejected", or "accepted" return only listings with that status; "none" returns only listings without a status set.',
),
},
async (
{
@@ -170,6 +176,7 @@ export function createMcpServer() {
maxPrice,
sortField,
sortDir,
status,
},
extra,
) => {
@@ -192,6 +199,7 @@ export function createMcpServer() {
maxPrice: maxPrice ?? null,
sortField: sortField ?? null,
sortDir: sortDir ?? 'desc',
statusFilter: status,
userId: user.id,
isAdmin: user.isAdmin,
});

View File

@@ -3,10 +3,6 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';
import { createMcpServer } from './mcpAdapter.js';
import { authenticateRequest } from './mcpAuthentication.js';
@@ -15,16 +11,13 @@ import crypto from 'crypto';
/**
* Active transports keyed by session id.
* Each session gets its own McpServer + StreamableHTTPServerTransport pair.
* @type {Map<string, { server: McpServer, transport: StreamableHTTPServerTransport }>}
*/
const sessions = new Map();
/**
* Get or create a session for the given session id with authentication.
* @param {string|undefined} sessionId
* @param {{ userId: string }} auth
* @returns {{ server: McpServer, transport: StreamableHTTPServerTransport }}
*/
function getOrCreateSession(sessionId, auth) {
if (sessionId && sessions.has(sessionId)) {
@@ -54,77 +47,67 @@ function getOrCreateSession(sessionId, auth) {
}
/**
* Register MCP Streamable HTTP routes on a restana service.
* Register MCP Streamable HTTP routes on a fastify instance.
*
* Mounts handlers at /api/mcp to handle the MCP Streamable HTTP protocol:
* - POST /api/mcp JSON-RPC messages (initialize, tool calls, etc.)
* - GET /api/mcp SSE stream for server-initiated notifications
* - DELETE /api/mcp session termination
* POST /api/mcp JSON-RPC messages
* GET /api/mcp SSE stream for server-initiated notifications
* DELETE /api/mcp session termination
*
* All endpoints require a valid Bearer token in the Authorization header.
*
* @param {import('restana').Service} service - The restana service instance.
* @param {import('fastify').FastifyInstance} fastify
*/
export function registerMcpRoutes(service) {
// POST main JSON-RPC endpoint
service.post('/api/mcp', async (req, res) => {
const auth = authenticateRequest(req);
export function registerMcpRoutes(fastify) {
fastify.post('/api/mcp', async (request, reply) => {
const auth = authenticateRequest(request.raw);
if (!auth) {
res.statusCode = 401;
return res.send({ error: 'Unauthorized. Provide a valid Bearer token.' });
return reply.code(401).send({ error: 'Unauthorized. Provide a valid Bearer token.' });
}
const sessionId = req.headers['mcp-session-id'];
const sessionId = request.raw.headers['mcp-session-id'];
const { server, transport } = getOrCreateSession(sessionId, auth);
// Connect server to transport if not already connected
if (!transport.onmessage) {
await server.connect(transport);
}
// Inject authInfo so tools can access the authenticated user
req.auth = { userId: auth.userId };
request.raw.auth = { userId: auth.userId };
await transport.handleRequest(req, res, req.body);
reply.hijack();
await transport.handleRequest(request.raw, reply.raw, request.body);
});
// GET SSE stream for server-initiated messages
service.get('/api/mcp', async (req, res) => {
const auth = authenticateRequest(req);
fastify.get('/api/mcp', async (request, reply) => {
const auth = authenticateRequest(request.raw);
if (!auth) {
res.statusCode = 401;
return res.send({ error: 'Unauthorized. Provide a valid Bearer token.' });
return reply.code(401).send({ error: 'Unauthorized. Provide a valid Bearer token.' });
}
const sessionId = req.headers['mcp-session-id'];
const sessionId = request.raw.headers['mcp-session-id'];
if (!sessionId || !sessions.has(sessionId)) {
res.statusCode = 400;
return res.send({ error: 'Invalid or missing session. Send an initialize request first.' });
return reply.code(400).send({ error: 'Invalid or missing session. Send an initialize request first.' });
}
const { transport } = sessions.get(sessionId);
await transport.handleRequest(req, res);
reply.hijack();
await transport.handleRequest(request.raw, reply.raw);
});
// DELETE terminate session
service.delete('/api/mcp', async (req, res) => {
const auth = authenticateRequest(req);
fastify.delete('/api/mcp', async (request, reply) => {
const auth = authenticateRequest(request.raw);
if (!auth) {
res.statusCode = 401;
return res.send({ error: 'Unauthorized. Provide a valid Bearer token.' });
return reply.code(401).send({ error: 'Unauthorized. Provide a valid Bearer token.' });
}
const sessionId = req.headers['mcp-session-id'];
const sessionId = request.raw.headers['mcp-session-id'];
if (!sessionId || !sessions.has(sessionId)) {
res.statusCode = 404;
return res.send({ error: 'Session not found.' });
return reply.code(404).send({ error: 'Session not found.' });
}
const { transport } = sessions.get(sessionId);
await transport.close();
sessions.delete(sessionId);
res.statusCode = 200;
res.send({ ok: true });
return { ok: true };
});
logger.debug('MCP Streamable HTTP endpoint registered at /api/mcp');

View File

@@ -70,7 +70,7 @@ export function normalizeListJobs(queryResult, { page, pageSize }) {
let md = `**Tool:** list_jobs | **Status:** OK\n\n`;
md += `Found **${queryResult.totalNumber}** job(s). Showing page ${page} of ${maxPage} (${jobs.length} on this page).`;
if (hasMore) md += ` More pages available use page=${page + 1} to continue.`;
if (hasMore) md += ` More pages available - use page=${page + 1} to continue.`;
md += '\n\n';
if (jobs.length > 0) {
@@ -120,14 +120,14 @@ export function normalizeListListings(queryResult, { page, pageSize }) {
let md = `**Tool:** list_listings | **Status:** OK\n\n`;
md += `Found **${queryResult.totalNumber}** listing(s). Showing page ${page} of ${maxPage} (${listings.length} on this page).`;
if (hasMore) md += ` More pages available use page=${page + 1} to continue.`;
if (hasMore) md += ` More pages available - use page=${page + 1} to continue.`;
md += '\n\n';
if (listings.length > 0) {
md += `| ID | Title | Address | Price | Size | Provider | Active | Created | Job |\n`;
md += `|----|-------|---------|-------|------|----------|--------|---------|-----|\n`;
md += `| ID | Title | Address | Price | Size | Provider | Active | Status | Created | Job |\n`;
md += `|----|-------|---------|-------|------|----------|--------|--------|---------|-----|\n`;
for (const l of listings) {
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
md += `| ${cell(l.id)} | ${cell(l.title)} | ${cell(l.address)} | ${cell(l.price)} | ${cell(l.size)} | ${cell(l.provider)} | ${l.is_active ? 'yes' : 'no'} | ${cell(l.status?.status)} | ${formatDate(l.created_at)} | ${cell(l.job_name)} |\n`;
}
md += `\nUse **get_listing** with an ID for full details (description, link, image).\n`;
} else {
@@ -156,6 +156,10 @@ export function normalizeGetListing(listing) {
md += `- **Link:** ${listing.link || ''}\n`;
md += `- **Image:** ${listing.image_url || ''}\n`;
md += `- **Active:** ${listing.is_active ? 'yes' : 'no'}\n`;
md += `- **Status:** ${listing.status?.status || ''}\n`;
if (listing.status?.setAt) {
md += `- **Status set at:** ${formatDate(listing.status.setAt)}\n`;
}
md += `- **Created:** ${formatDate(listing.created_at)}\n`;
md += `- **Job:** ${listing.job_name || ''}\n`;
if (listing.latitude != null && listing.longitude != null) {

View File

@@ -13,7 +13,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
const jobName = job == null ? jobKey : job.name;
const promises = newListings.map((newListing) => {
const title = `${jobName} at ${serviceName}: ${newListing.title}`;
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
const message = `Address: ${newListing.address}\nSize: ${newListing.size}\nPrice: ${newListing.price}\nLink: ${newListing.link}${fredyLine}`;
return fetch(server, {
method: 'POST',

View File

@@ -7,7 +7,7 @@ import { markdown2Html } from '../../services/markdown.js';
export const send = ({ serviceName, newListings, jobKey, baseUrl }) => {
/* eslint-disable no-console */
const fredyLinks = baseUrl ? newListings.map((l) => `${baseUrl}/listings/listing/${l.id}`).join(', ') : null;
const fredyLinks = baseUrl ? newListings.map((l) => `${baseUrl}/#/listings/listing/${l.id}`).join(', ') : null;
return [
Promise.resolve(
console.info(

View File

@@ -83,7 +83,7 @@ const buildEmbed = (jobKey, listing, baseUrl) => {
if (baseUrl && listing.id) {
fields.push({
name: 'Open in Fredy',
value: `[Open in Fredy](${baseUrl}/listings/listing/${listing.id})`,
value: `[Open in Fredy](${baseUrl}/#/listings/listing/${listing.id})`,
inline: false,
});
}

View File

@@ -13,7 +13,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
let message = `### *${jobName}* (${serviceName}) found **${newListings.length}** new listings:\n\n`;
message += `| Title | Address | Size | Price |${baseUrl ? ' Open in Fredy |' : ''}\n|:----|:----|:----|:----|${baseUrl ? ':----|\n' : '\n'}`;
message += newListings.map((o) => {
const fredyCell = baseUrl && o.id ? ` [Open in Fredy](${baseUrl}/listings/listing/${o.id}) |` : '';
const fredyCell = baseUrl && o.id ? ` [Open in Fredy](${baseUrl}/#/listings/listing/${o.id}) |` : '';
return (
`| [${o.title}](${o.link}) | ` +
[o.address, o.size.replace(/2m/g, '$m^2$'), o.price].join(' | ') +

View File

@@ -14,7 +14,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey, bas
const jobName = job == null ? jobKey : job.name;
const promises = newListings.map((newListing) => {
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
const message = `
Address: ${newListing.address}
Size: ${newListing.size == null ? 'N/A' : newListing.size.replace(/2m/g, '$m^2$')}

View File

@@ -15,7 +15,8 @@ export const send = async ({ serviceName, newListings, notificationConfig, jobKe
const results = await Promise.all(
newListings.map(async (newListing) => {
const title = `${jobName} at ${serviceName}: ${newListing.title}`;
const fredyLine = baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${newListing.id}` : '';
const fredyLine =
baseUrl && newListing.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${newListing.id}` : '';
const message = `Address: ${newListing.address}\nSize: ${newListing.size}\nPrice: ${newListing.price}\nLink: ${newListing.link}${fredyLine}`;
const form = new FormData();

View File

@@ -39,7 +39,7 @@ const buildBlocks = (serviceName, jobKey, p, baseUrl) => {
if (baseUrl && p.id) {
blocks.push({
type: 'section',
text: { type: 'mrkdwn', text: `<${baseUrl}/listings/listing/${p.id}|Open in Fredy>` },
text: { type: 'mrkdwn', text: `<${baseUrl}/#/listings/listing/${p.id}|Open in Fredy>` },
});
}

View File

@@ -39,7 +39,7 @@ const buildBlocks = (serviceName, jobKey, p, baseUrl) => {
if (baseUrl && p.id) {
blocks.push({
type: 'section',
text: { type: 'mrkdwn', text: `<${baseUrl}/listings/listing/${p.id}|Open in Fredy>` },
text: { type: 'mrkdwn', text: `<${baseUrl}/#/listings/listing/${p.id}|Open in Fredy>` },
});
}

View File

@@ -17,6 +17,7 @@ Multiple recipients:
Common SMTP settings:
- **Gmail** `smtp.gmail.com`, port 587, secure: false
- **Outlook** `smtp.office365.com`, port 587, secure: false
- **Yahoo** `smtp.mail.yahoo.com`, port 465, secure: true
- **Gmail** - `smtp.gmail.com`, port 587, secure: false
- **Outlook** - `smtp.office365.com`, port 587, secure: false
- **Yahoo** - `smtp.mail.yahoo.com`, port 465, secure: true
- **Gmx** - `mail.gmx.net`, port 587, secure: true

View File

@@ -9,6 +9,7 @@ import fetch from 'node-fetch';
import pThrottle from 'p-throttle';
import { normalizeImageUrl } from '../../utils.js';
import logger from '../../services/logger.js';
import { shouldUseMultipart, buildPhotoFormData } from './telegramPhotoUploader.js';
const RATE_LIMIT_INTERVAL = 1000;
const chatThrottleMap = new Map();
@@ -84,7 +85,7 @@ function buildCaption(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLink =
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/#/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
return `<i>${escapeHtml(jobName)}</i> (${escapeHtml(serviceName)})\n<a href='${escapeHtml(
o.link || '',
)}'><b>${escapeHtml(title)}</b></a>\n${escapeHtml(meta)}${fredyLink}`.slice(0, 1024);
@@ -101,7 +102,7 @@ function buildText(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLink =
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
baseUrl && o.id ? `\n<a href='${escapeHtml(`${baseUrl}/#/listings/listing/${o.id}`)}'>Open in Fredy</a>` : '';
return (
`<i>${escapeHtml(jobName)}</i> (${escapeHtml(serviceName)})\n` +
`<a href='${escapeHtml(o.link || '')}'><b>${escapeHtml(title)}</b></a>\n` +
@@ -120,7 +121,7 @@ function buildText(jobName, serviceName, o, baseUrl) {
function buildCaptionPlain(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${o.id}` : '';
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${o.id}` : '';
return `${jobName} (${serviceName})\n${title}\n${meta}\n\n${o.link || ''}${fredyLine}`.slice(0, 4096);
}
@@ -134,7 +135,7 @@ function buildCaptionPlain(jobName, serviceName, o, baseUrl) {
function buildTextPlain(jobName, serviceName, o, baseUrl) {
const title = shorten((o.title || '').replace(/\*/g, ''), 90);
const meta = [o.address, o.price, o.size].filter(Boolean).join(' | ');
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/listings/listing/${o.id}` : '';
const fredyLine = baseUrl && o.id ? `\nOpen in Fredy: ${baseUrl}/#/listings/listing/${o.id}` : '';
return `${jobName} (${serviceName})\n${title}\n${o.link || ''}\n${meta}${fredyLine}`;
}
@@ -177,11 +178,13 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
const jobName = job == null ? jobKey : job.name;
const throttledCall = getThrottled(chatId, async function (endpoint, body) {
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, {
method: 'post',
body: JSON.stringify(body),
headers: { 'Content-Type': 'application/json' },
});
// FormData (multipart) vs JSON. node-fetch sets its own multipart boundary
// header, so we must NOT supply Content-Type ourselves in that case.
const isFormData = body instanceof FormData;
const opts = isFormData
? { method: 'post', body }
: { method: 'post', body: JSON.stringify(body), headers: { 'Content-Type': 'application/json' } };
const res = await fetch(`https://api.telegram.org/bot${token}/${endpoint}`, opts);
if (!res.ok) {
const errorBody = await res.text();
@@ -208,16 +211,28 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
});
}
return await throttledCall('sendPhoto', {
chat_id: chatId,
photo: img,
caption: plainText
? buildCaptionPlain(jobName, serviceName, o, baseUrl)
: buildCaption(jobName, serviceName, o, baseUrl),
...(plainText ? {} : { parse_mode: 'HTML' }),
...(message_thread_id ? { message_thread_id } : {}),
}).catch(async (e) => {
logger.error(`Error sending photo to Telegram and use a fallback: ${e.message}`);
const caption = plainText
? buildCaptionPlain(jobName, serviceName, o, baseUrl)
: buildCaption(jobName, serviceName, o, baseUrl);
const parseMode = plainText ? undefined : 'HTML';
// .webp URLs (Immowelt/Cloudimage) fail Telegram's URL-based sendPhoto with
// "failed to get HTTP URL content". Upload the bytes via multipart instead;
// the rendered chat message is identical.
const photoCall = shouldUseMultipart(img)
? buildPhotoFormData({ chatId, imageUrl: img, caption, parseMode, messageThreadId: message_thread_id }).then(
(fd) => throttledCall('sendPhoto', fd),
)
: throttledCall('sendPhoto', {
chat_id: chatId,
photo: img,
caption,
...(parseMode ? { parse_mode: parseMode } : {}),
...(message_thread_id ? { message_thread_id } : {}),
});
return await photoCall.catch(async (e) => {
logger.warn(`Error sending photo to Telegram and use a fallback: ${e.message}`);
return await throttledCall('sendMessage', textPayload).catch((e) => {
logger.error(`Error sending message to Telegram: ${e.message}`);
throw e;

View File

@@ -0,0 +1,106 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
/**
* Helpers for sending photos to Telegram via `multipart/form-data` instead of
* the HTTP-URL path. Used when the URL is one that Telegram's URL-fetcher will
* reject - notably `.webp` images from Cloudimage (mms.immowelt.de), which
* Telegram refuses with "Bad Request: failed to get HTTP URL content".
*
* The HTTP-URL path is faster and is still the default in telegram.js; this
* module is the fallback for URLs whose extension makes Telegram fail.
*/
/** Telegram's sendPhoto limit when uploading bytes via multipart/form-data. */
const TELEGRAM_MULTIPART_MAX_BYTES = 10 * 1024 * 1024;
/** Accept header used when re-fetching the image ourselves.
* Deliberately excludes `image/webp` so CDNs that content-negotiate
* (like Cloudimage on mms.immowelt.de) transcode WEBP to JPEG. */
const NON_WEBP_ACCEPT = 'image/jpeg,image/png,image/*;q=0.8';
/**
* Returns true if the URL's path ends in a `.webp` extension. Such URLs need
* multipart upload because Telegram identifies media types from the URL path
* and rejects `.webp` in sendPhoto via HTTP URL.
*
* Conservative: returns false for null/empty/non-string input, malformed URLs,
* and non-https schemes.
*
* @param {string|null|undefined} url
* @returns {boolean}
*/
export function shouldUseMultipart(url) {
if (typeof url !== 'string' || url.length === 0) return false;
let parsed;
try {
parsed = new URL(url);
} catch {
return false;
}
if (parsed.protocol !== 'https:') return false;
return /\.webp$/i.test(parsed.pathname);
}
/**
* Fetch an image from `imageUrl` and build a `FormData` body suitable for
* POSTing to `https://api.telegram.org/bot<token>/sendPhoto`.
*
* - Sends an `Accept` header that excludes `image/webp` so origin/CDN servers
* that content-negotiate return JPEG bytes.
* - Rejects images larger than Telegram's 10 MB multipart limit, both
* advertised via `Content-Length` and (defensively) after download.
* - The `photo` field is named with a `.jpg` extension because Telegram
* identifies file type from the filename.
*
* Throws if the image fetch fails, the size limit is exceeded, or the URL is
* unreachable. The caller is responsible for catching and falling back.
*
* @param {Object} args
* @param {string|number} args.chatId
* @param {string} args.imageUrl
* @param {string} args.caption
* @param {string} [args.parseMode] - Telegram parse_mode, e.g. 'HTML'.
* @param {number} [args.messageThreadId] - Telegram supergroup topic id.
* @returns {Promise<FormData>}
*/
export async function buildPhotoFormData({ chatId, imageUrl, caption, parseMode, messageThreadId }) {
const res = await fetch(imageUrl, {
method: 'GET',
headers: { Accept: NON_WEBP_ACCEPT },
});
if (!res.ok) {
throw new Error(`Failed to fetch image for multipart upload (${res.status}): ${imageUrl}`);
}
const advertised = Number(res.headers.get('content-length'));
if (Number.isFinite(advertised) && advertised > TELEGRAM_MULTIPART_MAX_BYTES) {
throw new Error(
`Image exceeds Telegram multipart size limit (advertised ${advertised} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
);
}
const buf = await res.arrayBuffer();
if (buf.byteLength > TELEGRAM_MULTIPART_MAX_BYTES) {
throw new Error(
`Image exceeds Telegram multipart size limit (downloaded ${buf.byteLength} bytes, max ${TELEGRAM_MULTIPART_MAX_BYTES}): ${imageUrl}`,
);
}
// Telegram identifies the media type from the filename extension. We always
// upload as .jpg because the Accept header forces JPEG bytes from CDNs that
// honor it; for the rare CDN that ignores Accept and still returns WEBP, the
// .jpg filename is a small lie but Telegram's image pipeline accepts it.
const blob = new Blob([buf], { type: 'image/jpeg' });
const fd = new FormData();
fd.append('chat_id', String(chatId));
fd.append('caption', caption);
if (parseMode) fd.append('parse_mode', parseMode);
if (messageThreadId != null) fd.append('message_thread_id', String(messageThreadId));
fd.append('photo', blob, 'photo.jpg');
return fd;
}

View File

@@ -26,7 +26,7 @@ function parseId(shortenedLink) {
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, null, { browser });
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immobilienDe_details' });
if (!html) return listing;
const $ = cheerio.load(html);

View File

@@ -16,7 +16,7 @@ let appliedBlackList = [];
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, null, { browser });
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'immowelt_details' });
if (!html) return listing;
const $ = cheerio.load(html);
@@ -87,7 +87,19 @@ const config = {
crawlContainer:
'div[data-testid="serp-core-scrollablelistview-testid"]:not(div[data-testid="serp-enlargementlist-testid"] div[data-testid="serp-card-testid"]) div[data-testid="serp-core-classified-card-testid"]',
sortByDateParam: 'order=DateDesc',
waitForSelector: 'div[data-testid="serp-gridcontainer-testid"]',
// waitForSelector is null: extract the full page via page.content() so the
// Cheerio crawler can search anywhere in the rendered document.
// preNavigateUrl visits the homepage first to establish a trusted session
// before hitting the search URL; this prevents CDN-level bot challenges that
// fire on cold sessions. waitForNetworkIdle (phase 2) then catches React's
// listing API round-trip that fires well after domcontentloaded.
waitForSelector: null,
puppeteerOptions: {
puppeteerTimeout: 60_000,
preNavigateUrl: 'https://www.immowelt.de/',
waitForNetworkIdle: true,
waitForNetworkIdleTimeout: 60_000,
},
crawlFields: {
id: 'a@href',
price: 'div[data-testid="cardmfe-price-testid"] | removeNewline | trim',

View File

@@ -128,7 +128,7 @@ async function enrichListingFromDetails(listing, browser) {
if (!absoluteLink) return listing;
try {
const html = await puppeteerExtractor(absoluteLink, null, { browser });
const html = await puppeteerExtractor(absoluteLink, null, { browser, name: 'kleinanzeigen_details' });
if (!html) return { ...listing, link: absoluteLink };
const { detailAddress, detailDescription } = extractDetailFromHtml(html);
@@ -196,8 +196,8 @@ const config = {
id: '.aditem@data-adid',
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
tags: '.aditem-main--middle--tags | removeNewline | trim',
title: '.aditem-main .text-module-begin a | removeNewline | trim',
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
title: '.aditem-main .text-module-begin | removeNewline | trim',
link: '.aditem@data-href',
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
address: '.aditem-main--top--left | trim | removeNewline',
image: 'img@src',

View File

@@ -16,7 +16,7 @@ let appliedBlackList = [];
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, 'body', { browser });
const html = await puppeteerExtractor(listing.link, 'body', { browser, name: 'sparkasse_details' });
const $ = cheerio.load(html);
const nextDataRaw = $('#__NEXT_DATA__').text;

View File

@@ -16,7 +16,7 @@ let appliedBlackList = [];
async function fetchDetails(listing, browser) {
try {
const html = await puppeteerExtractor(listing.link, null, { browser });
const html = await puppeteerExtractor(listing.link, null, { browser, name: 'wgGesucht_details' });
if (!html) return listing;
const $ = cheerio.load(html);

View File

@@ -0,0 +1,147 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { ensureBinary } from 'cloakbrowser';
import fs from 'fs';
import path from 'path';
import os from 'os';
/**
* Resource files required on Linux/Windows — they must live next to the chrome binary.
* macOS packages these inside the .app bundle's Frameworks directory so a different
* check is used there (see isBinaryComplete).
*/
const LINUX_WIN_REQUIRED_FILES = ['icudtl.dat', 'resources.pak'];
/**
* Return the top-level versioned installation directory for any platform.
*
* - Linux/Windows: binaryPath is ~/.cloakbrowser/chromium-X.Y.Z/chrome
* → dirname ~/.cloakbrowser/chromium-X.Y.Z/
* - macOS: binaryPath is ~/.cloakbrowser/chromium-X.Y.Z/Chromium.app/Contents/MacOS/Chromium
* → 4 levels up ~/.cloakbrowser/chromium-X.Y.Z/
*
* @param {string} binaryPath
* @returns {string}
*/
function getVersionedDir(binaryPath) {
if (process.platform === 'darwin') {
return path.resolve(path.dirname(binaryPath), '../../..');
}
return path.dirname(binaryPath);
}
/**
* Return true when the binary at binaryPath belongs to a complete installation.
*
* On macOS the binary lives inside an .app bundle:
* Chromium.app/Contents/MacOS/Chromium
* Resource files (icudtl.dat etc.) are deep inside
* Chromium.app/Contents/Frameworks/…
* so checking for them next to the binary is wrong. Instead we verify the two
* structural markers that are only present after a full extraction: Info.plist
* and the Frameworks directory inside Contents/.
*
* On Linux/Windows the binary and all resource files are siblings in the same
* directory.
*
* @param {string} binaryPath
* @returns {boolean}
*/
function isBinaryComplete(binaryPath) {
if (process.platform === 'darwin') {
const contentsDir = path.resolve(path.dirname(binaryPath), '..');
return fs.existsSync(path.join(contentsDir, 'Info.plist')) && fs.existsSync(path.join(contentsDir, 'Frameworks'));
}
const dir = path.dirname(binaryPath);
return LINUX_WIN_REQUIRED_FILES.every((f) => fs.existsSync(path.join(dir, f)));
}
/**
* Return a human-readable description of which required files/dirs are missing.
*
* @param {string} binaryPath
* @returns {string}
*/
function missingDescription(binaryPath) {
if (process.platform === 'darwin') {
const contentsDir = path.resolve(path.dirname(binaryPath), '..');
return ['Info.plist', 'Frameworks'].filter((f) => !fs.existsSync(path.join(contentsDir, f))).join(', ');
}
const dir = path.dirname(binaryPath);
return LINUX_WIN_REQUIRED_FILES.filter((f) => !fs.existsSync(path.join(dir, f))).join(', ');
}
/**
* Remove a corrupt binary installation and all `latest_version*` markers from
* the CloakBrowser cache so the next `ensureBinary()` call falls back to the
* package-bundled version.
*
* Removes the full versioned directory (e.g. chromium-X.Y.Z/) on all platforms,
* not just the subdirectory that contains the binary.
*
* @param {string} binaryPath - Path to the (corrupt) chrome/Chromium binary.
*/
function removeCorruptInstallation(binaryPath) {
const versionedDir = getVersionedDir(binaryPath);
const cacheDir = process.env.CLOAKBROWSER_CACHE_DIR || path.join(os.homedir(), '.cloakbrowser');
fs.rmSync(versionedDir, { recursive: true, force: true });
try {
for (const entry of fs.readdirSync(cacheDir)) {
if (entry.startsWith('latest_version')) {
fs.rmSync(path.join(cacheDir, entry), { force: true });
}
}
} catch {
// Cache dir may not exist if versionedDir was the only entry — ignore.
}
}
/**
* Ensure the CloakBrowser stealth Chromium binary is present **and** complete.
*
* `cloakbrowser`'s own `ensureBinary()` only checks that the chrome/Chromium
* file exists. An incomplete extraction (e.g. interrupted download, disk full)
* can leave a directory that contains the executable but is missing essential
* resource files. Chrome then crashes immediately on launch.
*
* This wrapper validates the path returned by `ensureBinary()`. If the
* installation is incomplete it removes the corrupt directory, clears the
* version marker files, and calls `ensureBinary()` again so it falls back to
* (or re-downloads) a complete build.
*
* The validated path is also pinned via `CLOAKBROWSER_BINARY_PATH` so that
* CloakBrowser's own internal `ensureBinary()` call inside `launch()` always
* picks up the same, verified binary.
*
* @returns {Promise<string>} Absolute path to the validated binary.
* @throws {Error} When even the fallback binary is incomplete.
*/
export async function ensureValidBinary() {
const binaryPath = await ensureBinary();
if (isBinaryComplete(binaryPath)) {
process.env.CLOAKBROWSER_BINARY_PATH = binaryPath;
return binaryPath;
}
console.warn(
`[fredy] CloakBrowser installation at ${getVersionedDir(binaryPath)} is missing: ${missingDescription(binaryPath)}. Removing and retrying.`,
);
removeCorruptInstallation(binaryPath);
const fallbackPath = await ensureBinary();
if (!isBinaryComplete(fallbackPath)) {
throw new Error(
`CloakBrowser binary at ${getVersionedDir(fallbackPath)} is still missing required files after re-download: ${missingDescription(fallbackPath)}`,
);
}
process.env.CLOAKBROWSER_BINARY_PATH = fallbackPath;
return fallbackPath;
}

View File

@@ -94,7 +94,7 @@ export async function applyBotPreventionToPage(page, cfg) {
// webdriver
Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
// chrome runtime expose loadTimes, csi and app like real Chrome
// chrome runtime - expose loadTimes, csi and app like real Chrome
// @ts-ignore
window.chrome = {
runtime: {},
@@ -129,7 +129,7 @@ export async function applyBotPreventionToPage(page, cfg) {
get: () => (window.localStorage.getItem('__LANGS__') || 'de-DE,de').split(','),
});
// plugins mimic real Chrome's built-in PDF plugins
// plugins - mimic real Chrome's built-in PDF plugins
const makePlugin = (name, filename, description, mimeType, mimeTypeSuffix) => {
const mimeObj = { type: mimeType, suffixes: mimeTypeSuffix, description, enabledPlugin: null };
const plugin = { name, filename, description, length: 1, 0: mimeObj };
@@ -274,14 +274,14 @@ export async function applyBotPreventionToPage(page, cfg) {
//noop
}
// document.hasFocus headless returns false; real active tabs return true
// document.hasFocus - headless returns false; real active tabs return true
try {
document.hasFocus = () => true;
} catch {
//noop
}
// screen color depth normalise in case headless reports 0
// screen color depth - normalise in case headless reports 0
try {
Object.defineProperty(screen, 'colorDepth', { get: () => 24 });
Object.defineProperty(screen, 'pixelDepth', { get: () => 24 });

View File

@@ -29,11 +29,12 @@ export default class Extractor {
* your response will never contain what you are really looking for
* @param url
* @param waitForSelector
* @param jobKey
*/
execute = async (url, waitForSelector = null) => {
execute = async (url, waitForSelector = null, jobKey = null) => {
this.responseText = null;
try {
this.responseText = await puppeteerExtractor(url, waitForSelector, this.options);
this.responseText = await puppeteerExtractor(url, waitForSelector, { ...this.options, name: jobKey });
if (this.responseText != null) {
loadParser(this.responseText);
}

View File

@@ -3,121 +3,133 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import puppeteer from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
import { debug, botDetected } from './utils.js';
import {
getPreLaunchConfig,
applyBotPreventionToPage,
applyLanguagePersistence,
applyPostNavigationHumanSignals,
} from './botPrevention.js';
import { launch } from 'cloakbrowser/puppeteer';
import { botDetected, debug } from './utils.js';
import { getPreLaunchConfig } from './botPrevention.js';
import logger from '../logger.js';
import fs from 'fs';
import os from 'os';
import path from 'path';
puppeteer.use(StealthPlugin());
import { trackPoi } from '../tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
/**
* Launch a CloakBrowser/Puppeteer browser instance with stealth and humanizer enabled.
*
* CloakBrowser applies 49 C++ source-level patches (canvas, WebGL, audio, WebRTC,
* navigator.*, automation signals) that are indistinguishable from a real browser.
* All fingerprinting and human-behaviour simulation is handled natively; no CDP
* overrides (setUserAgent, setExtraHTTPHeaders, evaluateOnNewDocument) are applied
* here because they would create detectable inconsistencies on top of the C++ patches.
*
* @param {string} url - Initial URL (used to derive locale/timezone hints).
* @param {object} [options]
* @param {boolean} [options.puppeteerHeadless]
* @param {number} [options.puppeteerTimeout]
* @param {string} [options.proxyUrl]
* @param {string} [options.timezone]
* @param {string} [options.acceptLanguage]
* @param {object} [options.viewport]
* @returns {Promise<import('puppeteer-core').Browser>}
*/
export async function launchBrowser(url, options) {
const preCfg = getPreLaunchConfig(url, options || {});
const launchArgs = [
// Docker requires --no-sandbox; CloakBrowser handles all stealth args internally.
// --ignore-certificate-errors is needed because CloakBrowser ships its own Chromium
// binary with an independent CA bundle that may not trust proxies or interceptors
// present in the host environment.
const args = [
'--no-sandbox',
'--disable-gpu',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-crash-reporter',
'--no-first-run',
'--no-default-browser-check',
preCfg.langArg,
'--ignore-certificate-errors',
// Disables the zygote process model. Required in some container environments
// (e.g. limited kernel namespaces) where the zygote cannot acquire the
// locks it needs and exits with "Invalid file descriptor to ICU data received".
'--no-zygote',
preCfg.windowSizeArg,
...preCfg.extraArgs,
];
if (options?.proxyUrl) {
launchArgs.push(`--proxy-server=${options.proxyUrl}`);
}
let userDataDir;
let removeUserDataDir = false;
if (options && options.userDataDir) {
userDataDir = options.userDataDir;
} else {
const prefix = path.join(os.tmpdir(), 'puppeteer-fredy-');
userDataDir = fs.mkdtempSync(prefix);
removeUserDataDir = true;
}
// On ARM64 Docker, Chrome for Testing has no native binary — use system Chromium instead.
const executablePath =
options?.executablePath ||
(process.arch === 'arm64' && process.env.IS_DOCKER === 'true' ? '/usr/bin/chromium' : undefined);
const browser = await puppeteer.launch({
return await launch({
headless: options?.puppeteerHeadless ?? true,
args: launchArgs,
timeout: options?.puppeteerTimeout || 45_000,
userDataDir,
executablePath,
humanize: true,
args,
// locale sets Accept-Language headers and JS navigator.language consistently
locale: preCfg.langForFlag,
...(options?.proxyUrl ? { proxy: options.proxyUrl } : {}),
...(preCfg.timezone ? { timezone: preCfg.timezone } : {}),
});
browser.__fredy_userDataDir = userDataDir;
browser.__fredy_removeUserDataDir = removeUserDataDir;
return browser;
}
/**
* Close a browser instance returned by {@link launchBrowser}.
*
* @param {import('puppeteer-core').Browser | null} browser
*/
export async function closeBrowser(browser) {
if (!browser) return;
const userDataDir = browser.__fredy_userDataDir;
const removeUserDataDir = browser.__fredy_removeUserDataDir;
try {
await browser.close();
} catch {
// ignore
}
if (removeUserDataDir && userDataDir) {
try {
await fs.promises.rm(userDataDir, { recursive: true, force: true });
} catch {
// ignore
}
}
}
/**
* Open a page in a (possibly reused) browser, navigate to `url`, and return the HTML source.
* Returns `null` when a bot-detection page is encountered or on timeout.
*
* @param {string} url
* @param {string | null} waitForSelector
* @param {object} [options]
* @returns {Promise<string | null>}
*/
export default async function execute(url, waitForSelector, options) {
let browser = options?.browser;
let isExternalBrowser = !!browser;
let page;
let result;
try {
debug(`Sending request to ${url} using Puppeteer.`);
debug(`Sending request to ${url} using CloakBrowser.`);
if (!isExternalBrowser) {
browser = await launchBrowser(url, options);
}
page = await browser.newPage();
const preCfg = getPreLaunchConfig(url, options || {});
await applyBotPreventionToPage(page, preCfg);
// Provide languages value before navigation
await applyLanguagePersistence(page, preCfg);
// Optional cookies
if (Array.isArray(options?.cookies) && options.cookies.length > 0) {
await page.setCookie(...options.cookies);
}
// Navigation
// Warm-up navigation: visit a trusted page first so the site sees an
// established session before the actual target URL. Silently ignored on
// failure so it never blocks the main request.
if (options?.preNavigateUrl) {
try {
await page.goto(options.preNavigateUrl, { waitUntil: 'domcontentloaded', timeout: 30_000 });
await new Promise((r) => setTimeout(r, 1500 + Math.random() * 2000));
} catch {
// ignore
}
}
const response = await page.goto(url, {
waitUntil: options?.waitUntil || 'domcontentloaded',
timeout: options?.puppeteerTimeout || 60000,
});
// Optionally wait and add subtle human-like interactions
await applyPostNavigationHumanSignals(page, preCfg);
// Optional second idle wait: useful for React SPAs that trigger API calls
// after domcontentloaded. Times out silently so we use whatever is rendered.
if (options?.waitForNetworkIdle) {
try {
await page.waitForNetworkIdle({ timeout: options?.waitForNetworkIdleTimeout ?? 60_000 });
} catch {
// ignore — we proceed with whatever the DOM contains at this point
}
}
let pageSource;
// if we're extracting data from a SPA, we must wait for the selector
if (waitForSelector != null) {
const selectorTimeout = options?.puppeteerSelectorTimeout ?? options?.puppeteerTimeout ?? 30_000;
await page.waitForSelector(waitForSelector, { timeout: selectorTimeout });
@@ -133,15 +145,22 @@ export default async function execute(url, waitForSelector, options) {
if (botDetected(pageSource, statusCode)) {
logger.warn('We have been detected as a bot :-/ Tried url: => ', url);
if (options != null && options.name != null) {
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT + '_' + options.name);
} else {
await trackPoi(TRACKING_POIS.DETECTED_AS_BOT);
}
result = null;
} else {
result = pageSource || (await page.content());
}
} catch (error) {
if (error?.name?.includes('Timeout')) {
logger.debug('Error executing with puppeteer executor', error);
logger.debug('Error executing with CloakBrowser executor', error);
} else {
logger.warn('Error executing with puppeteer executor', error);
logger.warn('Error executing with CloakBrowser executor', error);
}
result = null;
} finally {

View File

@@ -141,6 +141,43 @@ const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
'barrierefreie-wohnung-mieten': { equipment: ['handicappedaccessible'] },
};
// SEO-optimized rental paths used by the ImmoScout web UI when the user
// configures a maximum warmrent. Example: "wohnung-bis-800-euro-warm" means
// "apartment for rent up to 800 EUR warmrent". The web UI generates these
// paths instead of explicit `price` / `pricetype` query parameters.
// Note: only the warmrent variant uses an SEO slug; max coldrent searches
// use the regular "wohnung-mieten" path with explicit `price` and
// `pricetype=rentpermonth` query params, which the existing translator
// already handles.
const SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE = {
wohnung: 'apartmentrent',
haus: 'houserent',
};
const SEO_MAX_WARMRENT_PATH_PATTERN = /^(?<type>wohnung|haus)-bis-(?<price>\d+)-euro-warm$/;
/**
* Parses SEO-optimized ImmoScout web paths that encode a maximum warmrent, such
* as "wohnung-bis-800-euro-warm". Returns the corresponding mobile API real
* estate type and the implicit price/pricetype parameters, or null if the path
* does not match the known SEO max-warmrent pattern.
*
* @param {string} realTypeKey The last segment of the URL path.
* @returns {{ realType: string, additionalParams: Record<string, string> } | null}
*/
function parseSeoMaxWarmrentPath(realTypeKey) {
const match = realTypeKey.match(SEO_MAX_WARMRENT_PATH_PATTERN);
if (!match) return null;
const { type, price } = match.groups;
return {
realType: SEO_RENT_TYPE_TO_REAL_ESTATE_TYPE[type],
additionalParams: {
price: `-${price}`,
pricetype: 'calculatedtotalrent',
},
};
}
export function convertWebToMobile(webUrl) {
let url;
try {
@@ -164,14 +201,17 @@ export function convertWebToMobile(webUrl) {
additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey];
realType = REAL_ESTATE_TYPE['wohnung-mieten'];
} else {
throw new Error(`Real estate type not found: ${realTypeKey}`);
// Test for SEO max-warmrent path, e.g. "wohnung-bis-800-euro-warm"
const seoMaxWarmrent = parseSeoMaxWarmrentPath(realTypeKey);
if (seoMaxWarmrent) {
realType = seoMaxWarmrent.realType;
additionalParamsFromWebPath = seoMaxWarmrent.additionalParams;
} else {
throw new Error(`Real estate type not found: ${realTypeKey}`);
}
}
}
if (segments.includes('shape')) {
throw new Error('Shape is currently not supported using Immoscout');
}
const { query: rawParams } = queryString.parseUrl(webUrl, { arrayFormat: 'comma' });
const webParams = Object.fromEntries(
Object.entries(rawParams).filter(([key]) => key !== 'enteredFrom' && PARAM_NAME_MAP[key]),
@@ -179,18 +219,31 @@ export function convertWebToMobile(webUrl) {
const geocodes = `/${segments.slice(2, segments.length - 1).join('/')}`;
const isRadius = segments.includes('radius');
const isShape = segments.includes('shape');
const mobileParams = {
searchType: isRadius ? 'radius' : 'region',
searchType: isRadius ? 'radius' : isShape ? 'shape' : 'region',
realestatetype: realType,
...(isRadius ? {} : { geocodes }),
...(isRadius || isShape ? {} : { geocodes }),
...additionalParamsFromWebPath,
};
if (isShape && !webParams.shape) {
throw new Error('Shape search URL is missing the required "shape" query parameter');
}
if (isShape && webParams.shape) {
const browserShape = webParams.shape;
const normalized = browserShape.replace(/\.\./g, '==').replace(/\./g, '=');
const polyline = Buffer.from(normalized, 'base64').toString('utf-8');
mobileParams.shape = polyline;
}
if (webParams.geocoordinates) {
mobileParams.geocoordinates = webParams.geocoordinates;
}
for (const [key, val] of Object.entries(webParams)) {
if (key === 'shape') continue;
if (key === 'equipment') {
const items = [].concat(val).flatMap((v) => `${v}`.split(','));
const currentEquipmentParams = mobileParams[PARAM_NAME_MAP[key]];

View File

@@ -14,6 +14,7 @@ import * as similarityCache from '../similarity-check/similarityCache.js';
import { isRunning, markFinished, markRunning } from './run-state.js';
import { sendToUsers } from '../sse/sse-broker.js';
import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
import { getSettings } from '../storage/settingsStorage.js';
/**
* Initializes the job execution service.
@@ -160,6 +161,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
}
let browser;
try {
// Read the proxy live (not from the startup snapshot) so changing it in the
// UI takes effect on the next run without a backend restart. An empty value
// disables the proxy. Routing the headless browser through a (German
// residential) proxy avoids datacenter-IP based bot detection on the
// Puppeteer-based providers (immowelt, immonet, kleinanzeigen, ...).
const liveSettings = await getSettings();
const proxyUrl = typeof liveSettings?.proxyUrl === 'string' ? liveSettings.proxyUrl.trim() : '';
const jobProviders = job.provider.filter(
(p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
);
@@ -168,14 +177,14 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
if (browser && !browser.isConnected()) {
if (browser && !browser.connected) {
logger.debug('Browser is disconnected, nullifying to launch a new one.');
await puppeteerExtractor.closeBrowser(browser);
browser = null;
}
if (!browser && matchedProvider.config.getListings == null) {
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, proxyUrl ? { proxyUrl } : {});
}
await new FredyPipelineExecutioner(matchedProvider.config, job, prov.id, similarityCache, browser).execute();

View File

@@ -3,10 +3,27 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { nullOrEmpty } from '../../utils.js';
import { nullOrEmpty, fromJson } from '../../utils.js';
import SqliteConnection from './SqliteConnection.js';
import { nanoid } from 'nanoid';
/**
* Parse the JSON `status` column of a listing row in place.
*
* The DB stores status as a JSON payload `{ status, setAt }` (or NULL).
* Consumers expect an object/null, so we normalize before returning.
*
* @param {Object|null|undefined} row - A raw row from the listings table.
* @returns {Object|null|undefined} The same row with `status` parsed.
*/
const parseListingStatus = (row) => {
if (row == null) return row;
if (typeof row.status === 'string') {
row.status = fromJson(row.status, null);
}
return row;
};
/**
* Return a list of known listing hashes for a given job and provider.
* Useful to de-duplicate before inserting new listings.
@@ -214,6 +231,8 @@ export const storeListings = (jobId, providerId, listings) => {
longitude: item.longitude || null,
};
stmt.run(params);
// Propagate the DB primary key back so downstream pipeline steps use the correct id
item.id = params.id;
}
});
@@ -242,6 +261,7 @@ export const storeListings = (jobId, providerId, listings) => {
* @param {object} [params.jobNameFilter]
* @param {object} [params.providerFilter]
* @param {object} [params.watchListFilter]
* @param {('applied'|'rejected'|'accepted'|'none')} [params.statusFilter] - Filter by listing status. 'none' matches NULL.
* @param {string|null} [params.sortField=null] - One of: 'created_at','price','size','provider','title'.
* @param {('asc'|'desc')} [params.sortDir='asc']
* @param {number} [params.createdAfter] - Only include listings created at or after this unix timestamp (ms).
@@ -258,6 +278,7 @@ export const queryListings = ({
jobIdFilter,
providerFilter,
watchListFilter,
statusFilter,
freeTextFilter,
sortField = null,
sortDir = 'asc',
@@ -315,6 +336,18 @@ export const queryListings = ({
} else if (watchListFilter === false) {
whereParts.push('(wl.id IS NULL)');
}
// statusFilter: 'applied'|'rejected'|'accepted' -> equality on JSON status field; 'none' -> NULL.
// The status column is a JSON payload `{ status, setAt }`, so we extract the inner
// status string for comparison instead of matching the raw text.
if (statusFilter === 'none') {
whereParts.push('(l.status IS NULL)');
} else if (
typeof statusFilter === 'string' &&
['applied', 'rejected', 'accepted'].includes(statusFilter.toLowerCase())
) {
params.statusValue = statusFilter.toLowerCase();
whereParts.push(`(json_extract(l.status, '$.status') = @statusValue)`);
}
// Time range filters (unix timestamps in milliseconds)
if (Number.isFinite(createdAfter) && createdAfter > 0) {
params.createdAfter = createdAfter;
@@ -389,7 +422,7 @@ export const queryListings = ({
params,
);
return { totalNumber, page: safePage, result: rows };
return { totalNumber, page: safePage, result: rows.map(parseListingStatus) };
};
/**
@@ -417,9 +450,10 @@ export const deleteListingsByJobId = (jobId, hardDelete = false) => {
};
/**
* Delete listings by a list of listing IDs.
* Delete listings by a list of listing IDs (the nanoid primary key stored in the `id` column).
* Used by API routes that receive row IDs from the client.
*
* @param {string[]} ids - Array of listing IDs to delete.
* @param {string[]} ids - Array of DB row IDs to delete.
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
* @returns {any} The result from SqliteConnection.execute.
*/
@@ -623,7 +657,7 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
if (!isAdmin) {
whereScoping = `AND (j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`;
}
return (
return parseListingStatus(
SqliteConnection.query(
`SELECT l.*, j.name AS job_name, CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END AS isWatched
FROM listings l
@@ -631,10 +665,57 @@ export const getListingById = (id, userId = null, isAdmin = false) => {
LEFT JOIN watch_list wl ON wl.listing_id = l.id AND wl.user_id = @userId
WHERE l.id = @id AND l.manually_deleted = 0 ${whereScoping}`,
params,
)[0] || null
)[0] || null,
);
};
/**
* Set or clear the notes attached to a single listing.
*
* Empty strings are normalized to NULL so the DB doesn't keep meaningless
* whitespace and queries can filter "has notes" with a simple IS NOT NULL.
*
* @param {string} id - The listing ID.
* @param {string|null} notes - The note text to store, or null/empty to clear.
* @returns {number} Number of rows affected (0 if listing not found).
*/
export const setListingNotes = (id, notes) => {
if (!id) return 0;
const trimmed = typeof notes === 'string' ? notes.trim() : null;
const value = trimmed && trimmed.length > 0 ? trimmed : null;
const res = SqliteConnection.execute(`UPDATE listings SET notes = @notes WHERE id = @id`, {
id,
notes: value,
});
return res?.changes ?? 0;
};
/**
* Set or clear the status of a single listing.
*
* The status column stores a JSON payload `{ status, setAt }` so consumers
* can show both the user's decision and when it was made. Passing `null`
* clears the column.
*
* @param {string} id - The listing ID.
* @param {('applied'|'rejected'|'accepted'|null)} status - New status, or null to clear.
* @returns {number} Number of rows affected (0 if listing not found).
*/
export const setListingStatus = (id, status) => {
if (!id) return 0;
const allowed = ['applied', 'rejected', 'accepted'];
const normalized = status == null ? null : String(status).toLowerCase();
if (normalized != null && !allowed.includes(normalized)) {
throw new Error(`Invalid listing status: ${status}`);
}
const payload = normalized == null ? null : JSON.stringify({ status: normalized, setAt: Date.now() });
const res = SqliteConnection.execute(`UPDATE listings SET status = @status WHERE id = @id`, {
id,
status: payload,
});
return res?.changes ?? 0;
};
/**
* Resets geocoordinates and distance for all listings related to a user.
*

View File

@@ -0,0 +1,11 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
export function up(db) {
db.exec(`
ALTER TABLE listings ADD COLUMN status JSON;
CREATE INDEX IF NOT EXISTS idx_listings_status ON listings (json_extract(status, '$.status'));
`);
}

View File

@@ -0,0 +1,10 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
export function up(db) {
db.exec(`
ALTER TABLE listings ADD COLUMN notes TEXT;
`);
}

View File

@@ -47,6 +47,25 @@ export const deleteWatch = (listingId, userId) => {
return { deleted: Boolean(res?.changes) };
};
/**
* Ensure a watch entry exists. Does not toggle; safe to call when row may already exist.
* Used by the status endpoint to auto-watch a listing when a status is set.
* @param {string} listingId
* @param {string} userId
* @returns {{watched:boolean}}
*/
export const ensureWatch = (listingId, userId) => {
if (!listingId || !userId) return { watched: false };
const { created } = createWatch(listingId, userId);
if (created) return { watched: true };
const exists =
SqliteConnection.query(
`SELECT 1 AS ok FROM watch_list WHERE listing_id = @listing_id AND user_id = @user_id LIMIT 1`,
{ listing_id: listingId, user_id: userId },
).length > 0;
return { watched: exists };
};
/**
* Toggle a watch entry. If exists -> delete, otherwise create.
* @param {string} listingId

View File

@@ -1,6 +1,6 @@
{
"name": "fredy",
"version": "21.0.1",
"version": "22.3.1",
"description": "[F]ind [R]eal [E]states [d]amn eas[y].",
"scripts": {
"prepare": "husky",
@@ -62,65 +62,65 @@
"Firefox ESR"
],
"dependencies": {
"@douyinfe/semi-icons": "^2.95.1",
"@douyinfe/semi-ui": "2.95.1",
"@douyinfe/semi-ui-19": "^2.95.1",
"@douyinfe/semi-icons": "^2.99.3",
"@douyinfe/semi-ui": "2.99.3",
"@douyinfe/semi-ui-19": "^2.99.3",
"@fastify/cookie": "^11.0.2",
"@fastify/helmet": "^13.0.2",
"@fastify/session": "^11.1.1",
"@fastify/static": "^9.1.3",
"@mapbox/mapbox-gl-draw": "^1.5.1",
"@modelcontextprotocol/sdk": "^1.29.0",
"@sendgrid/mail": "8.1.6",
"@turf/boolean-point-in-polygon": "^7.3.5",
"@vitejs/plugin-react": "6.0.1",
"@vitejs/plugin-react": "6.0.2",
"adm-zip": "^0.5.17",
"better-sqlite3": "^12.9.0",
"body-parser": "2.2.2",
"better-sqlite3": "^12.10.0",
"chart.js": "^4.5.1",
"cheerio": "^1.2.0",
"cookie-session": "2.1.1",
"cloakbrowser": "^0.3.31",
"fastify": "^5.8.5",
"handlebars": "4.7.9",
"maplibre-gl": "^5.24.0",
"nanoid": "5.1.9",
"nanoid": "5.1.11",
"node-cron": "^4.2.1",
"node-fetch": "3.3.2",
"node-mailjet": "6.0.11",
"nodemailer": "^8.0.6",
"nodemailer": "^8.0.10",
"p-throttle": "^8.1.0",
"package-up": "^5.0.0",
"puppeteer": "^24.42.0",
"puppeteer-extra": "^3.3.6",
"puppeteer-extra-plugin-stealth": "^2.11.2",
"query-string": "9.3.1",
"react": "19.2.5",
"puppeteer-core": "^25.1.0",
"query-string": "9.4.0",
"react": "19.2.7",
"react-chartjs-2": "^5.3.1",
"react-dom": "19.2.5",
"react-dom": "19.2.7",
"react-range-slider-input": "^3.3.5",
"react-router": "7.14.2",
"react-router-dom": "7.14.2",
"resend": "^6.12.2",
"restana": "6.0.0",
"semver": "^7.7.4",
"serve-static": "2.2.1",
"react-router": "7.16.0",
"react-router-dom": "7.16.0",
"resend": "^6.12.4",
"semver": "^7.8.1",
"slack": "11.0.2",
"vite": "8.0.10",
"vite": "8.0.16",
"x-var": "^3.0.1",
"zustand": "^5.0.12"
"zustand": "^5.0.14"
},
"devDependencies": {
"@babel/core": "7.29.0",
"@babel/eslint-parser": "7.28.6",
"@babel/preset-env": "7.29.2",
"@babel/preset-react": "7.28.5",
"@babel/core": "7.29.7",
"@babel/eslint-parser": "7.29.7",
"@babel/preset-env": "7.29.7",
"@babel/preset-react": "7.29.7",
"@eslint/js": "^10.0.1",
"chalk": "^5.6.2",
"eslint": "10.2.1",
"eslint": "10.4.1",
"eslint-config-prettier": "10.1.8",
"eslint-plugin-react": "7.37.5",
"globals": "^17.5.0",
"globals": "^17.6.0",
"history": "5.3.0",
"husky": "9.1.7",
"less": "4.6.4",
"lint-staged": "16.4.0",
"lint-staged": "17.0.7",
"nodemon": "^3.1.14",
"prettier": "3.8.3",
"vitest": "^4.1.5"
"vitest": "^4.1.8"
}
}

18
test/globalSetup.js Normal file
View File

@@ -0,0 +1,18 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { ensureValidBinary } from '../lib/services/ensureValidBinary.js';
/**
* Vitest global setup — runs once in the main process before any workers start.
* Downloads and validates the CloakBrowser stealth Chromium binary.
* ensureValidBinary() also removes and re-downloads partial/corrupt installations
* so tests never fail with "Invalid file descriptor to ICU data received".
* Skipped in offline mode because the browser is fully mocked there.
*/
export async function setup() {
if (process.env.TEST_MODE === 'offline') return;
await ensureValidBinary();
}

View File

@@ -32,4 +32,7 @@ export const deletedIds = [];
export const deleteListingsById = (ids) => {
deletedIds.push(...ids);
};
export const deleteListingsByHash = (hashes) => {
deletedIds.push(...hashes);
};
/* eslint-enable no-unused-vars */

View File

@@ -0,0 +1,360 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
// Mock external deps BEFORE importing the module under test.
vi.mock('node-fetch', () => ({ default: vi.fn() }));
vi.mock('../../lib/services/storage/jobStorage.js', () => ({
getJob: (jobKey) => ({ id: jobKey, name: jobKey }),
}));
vi.mock('../../lib/services/markdown.js', () => ({
markdown2Html: () => '',
}));
// Helpers to build mock fetch responses.
function jsonOk(body = { ok: true }) {
return {
ok: true,
status: 200,
text: async () => JSON.stringify(body),
};
}
function jsonErr(status, body) {
return {
ok: false,
status,
text: async () => JSON.stringify(body),
};
}
function imageOk(bytes = new Uint8Array([0xff, 0xd8, 0xff])) {
return {
ok: true,
status: 200,
headers: {
get: (h) => {
const k = h.toLowerCase();
if (k === 'content-type') return 'image/jpeg';
if (k === 'content-length') return String(bytes.byteLength);
return null;
},
},
arrayBuffer: async () => bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength),
};
}
// Globals are mocked too so buildPhotoFormData (which uses global fetch) can be
// intercepted by the same single mock.
let mockNodeFetch;
let mockGlobalFetch;
let send;
beforeEach(async () => {
// Reset modules to get a fresh import with our mocks applied.
vi.resetModules();
const nodeFetchMod = await import('node-fetch');
mockNodeFetch = nodeFetchMod.default;
mockNodeFetch.mockReset();
mockGlobalFetch = vi.fn();
vi.stubGlobal('fetch', mockGlobalFetch);
({ send } = await import('../../lib/notification/adapter/telegram.js'));
});
afterEach(() => {
vi.unstubAllGlobals();
vi.useRealTimers();
});
const baseConfig = {
id: 'telegram',
fields: { token: 'TKN', chatId: '999' },
};
describe('telegram send() - HTTP URL path (default for .jpg / .png)', () => {
it('POSTs JSON to sendPhoto for a .jpg image URL', async () => {
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 'Listing',
link: 'https://example.com/a',
address: 'Addr',
price: '500€',
size: '50m²',
image: 'https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
const [url, opts] = mockNodeFetch.mock.calls[0];
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
expect(opts.method).toBe('post');
expect(opts.headers?.['Content-Type']).toBe('application/json');
const body = JSON.parse(opts.body);
expect(body.chat_id).toBe('999');
expect(body.photo).toBe('https://mms.immowelt.de/x/y/z/w/abc.jpg?ci_seal=hash&w=525&h=394');
expect(body.parse_mode).toBe('HTML');
});
it('does NOT pre-fetch the image when using HTTP URL path', async () => {
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/x.jpg',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
// global fetch (used by buildPhotoFormData) must not be called
expect(mockGlobalFetch).not.toHaveBeenCalled();
});
it('falls back to sendMessage when sendPhoto fails', async () => {
mockNodeFetch
.mockResolvedValueOnce(jsonErr(400, { ok: false, description: 'boom' }))
.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/x.jpg',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendPhoto');
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
});
});
describe('telegram send() - multipart path (.webp URLs)', () => {
it('pre-fetches the image then POSTs FormData to sendPhoto for a .webp URL', async () => {
// 1st: GET image via global fetch
mockGlobalFetch.mockResolvedValueOnce(imageOk());
// 2nd: POST sendPhoto via node-fetch
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 'Listing',
link: 'https://example.com/a',
address: 'Addr',
price: '500€',
size: '50m²',
image: 'https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
// image was fetched
expect(mockGlobalFetch).toHaveBeenCalledTimes(1);
expect(mockGlobalFetch.mock.calls[0][0]).toBe('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394');
// sendPhoto called via node-fetch with FormData
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
const [url, opts] = mockNodeFetch.mock.calls[0];
expect(url).toBe('https://api.telegram.org/botTKN/sendPhoto');
expect(opts.method).toBe('post');
expect(opts.body).toBeInstanceOf(FormData);
// No explicit Content-Type header - fetch sets multipart boundary itself
expect(opts.headers).toBeUndefined();
expect(opts.body.get('chat_id')).toBe('999');
expect(opts.body.get('parse_mode')).toBe('HTML');
const photo = opts.body.get('photo');
expect(photo).toBeTruthy();
expect(photo.size).toBeGreaterThan(0);
});
it('falls back to sendMessage when the image pre-fetch fails for a .webp URL', async () => {
// image fetch fails (404 from CDN)
mockGlobalFetch.mockResolvedValueOnce({
ok: false,
status: 404,
headers: { get: () => null },
arrayBuffer: async () => new ArrayBuffer(0),
});
// then sendMessage succeeds via node-fetch
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/gone.webp',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
});
it('falls back to sendMessage when multipart sendPhoto returns a Telegram error', async () => {
mockGlobalFetch.mockResolvedValueOnce(imageOk());
mockNodeFetch
.mockResolvedValueOnce(jsonErr(400, { description: 'broke' })) // multipart sendPhoto
.mockResolvedValueOnce(jsonOk()); // sendMessage fallback
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/x.webp',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
expect(mockNodeFetch.mock.calls[1][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
});
});
describe('telegram send() - mixed batch (regression-safety)', () => {
it('handles a batch with both .jpg and .webp - jpg uses URL, webp uses multipart', async () => {
// .webp image fetch
mockGlobalFetch.mockResolvedValueOnce(imageOk());
// both sendPhoto calls succeed
mockNodeFetch
.mockResolvedValueOnce(jsonOk()) // could be either listing first
.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'jpg-listing',
title: 'a',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/a.jpg',
},
{
id: 'webp-listing',
title: 'b',
link: 'l',
address: 'a',
price: '',
size: '',
image: 'https://example.com/b.webp',
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockGlobalFetch).toHaveBeenCalledTimes(1); // only webp pre-fetches
expect(mockNodeFetch).toHaveBeenCalledTimes(2);
// Verify one call had FormData and one had JSON body
const bodies = mockNodeFetch.mock.calls.map((c) => c[1].body);
const hasFormData = bodies.some((b) => b instanceof FormData);
const hasJson = bodies.some((b) => typeof b === 'string' && b.startsWith('{'));
expect(hasFormData).toBe(true);
expect(hasJson).toBe(true);
});
it('uses sendMessage (not sendPhoto) when image is null', async () => {
mockNodeFetch.mockResolvedValueOnce(jsonOk());
await send({
serviceName: 'immowelt',
newListings: [
{
id: 'a',
title: 't',
link: 'l',
address: 'a',
price: '',
size: '',
image: null,
},
],
notificationConfig: [baseConfig],
jobKey: 'Berlin',
});
expect(mockNodeFetch).toHaveBeenCalledTimes(1);
expect(mockNodeFetch.mock.calls[0][0]).toBe('https://api.telegram.org/botTKN/sendMessage');
expect(mockGlobalFetch).not.toHaveBeenCalled();
});
});
describe('telegram send() - config validation', () => {
it('throws when telegram adapter config is missing', () => {
expect(() =>
send({
serviceName: 's',
newListings: [],
notificationConfig: [],
jobKey: 'k',
}),
).toThrow(/configuration missing/);
});
it('throws when token or chatId is missing', () => {
expect(() =>
send({
serviceName: 's',
newListings: [],
notificationConfig: [{ id: 'telegram', fields: { token: '' } }],
jobKey: 'k',
}),
).toThrow(/token.*chatId/);
});
});

View File

@@ -0,0 +1,287 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { shouldUseMultipart, buildPhotoFormData } from '../../lib/notification/adapter/telegramPhotoUploader.js';
describe('shouldUseMultipart', () => {
it('returns true for .webp URL with query string', () => {
expect(shouldUseMultipart('https://mms.immowelt.de/1/1/6/5/abc.webp?ci_seal=hash&w=525&h=394')).toBe(true);
});
it('returns true for .webp URL without query string', () => {
expect(shouldUseMultipart('https://example.com/photo.webp')).toBe(true);
});
it('returns true for uppercase .WEBP extension', () => {
expect(shouldUseMultipart('https://example.com/IMG.WEBP?x=1')).toBe(true);
});
it('returns false for .jpg URL with query string', () => {
expect(shouldUseMultipart('https://mms.immowelt.de/a/b/c/d/xyz.jpg?ci_seal=hash&w=525&h=394')).toBe(false);
});
it('returns false for .jpeg URL', () => {
expect(shouldUseMultipart('https://example.com/photo.jpeg')).toBe(false);
});
it('returns false for .png URL with query string', () => {
expect(shouldUseMultipart('https://example.com/photo.png?w=100')).toBe(false);
});
it('returns false for .gif URL', () => {
expect(shouldUseMultipart('https://example.com/photo.gif')).toBe(false);
});
it('returns false for null', () => {
expect(shouldUseMultipart(null)).toBe(false);
});
it('returns false for undefined', () => {
expect(shouldUseMultipart(undefined)).toBe(false);
});
it('returns false for empty string', () => {
expect(shouldUseMultipart('')).toBe(false);
});
it('returns false for malformed URL', () => {
expect(shouldUseMultipart('not a url')).toBe(false);
});
it('returns false for URL where webp is in the query but not the path', () => {
expect(shouldUseMultipart('https://example.com/photo.jpg?format=webp')).toBe(false);
});
it('returns false for URL with no extension at all', () => {
expect(shouldUseMultipart('https://example.com/photo')).toBe(false);
});
it('returns false for non-https schemes', () => {
// file/data/ftp URLs should not be relevant; safer to skip multipart
expect(shouldUseMultipart('http://example.com/photo.webp')).toBe(false);
});
});
describe('buildPhotoFormData', () => {
let mockFetch;
beforeEach(() => {
mockFetch = vi.fn();
vi.stubGlobal('fetch', mockFetch);
});
afterEach(() => {
vi.unstubAllGlobals();
});
function makeImageResponse({ contentType = 'image/jpeg', bytes = new Uint8Array([0xff, 0xd8, 0xff]) } = {}) {
const buf = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
return {
ok: true,
status: 200,
headers: {
get: (h) =>
h.toLowerCase() === 'content-type'
? contentType
: h.toLowerCase() === 'content-length'
? String(bytes.byteLength)
: null,
},
arrayBuffer: async () => buf,
};
}
it('fetches image with Accept header that excludes webp so the CDN transcodes to JPEG', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
await buildPhotoFormData({
chatId: '123',
imageUrl: 'https://example.com/photo.webp',
caption: 'hi',
parseMode: 'HTML',
});
expect(mockFetch).toHaveBeenCalledTimes(1);
const [url, opts] = mockFetch.mock.calls[0];
expect(url).toBe('https://example.com/photo.webp');
expect(opts?.headers?.Accept || opts?.headers?.accept).toMatch(/image\/jpeg/);
expect(opts?.headers?.Accept || opts?.headers?.accept).not.toMatch(/image\/webp/);
});
it('returns FormData containing chat_id, caption, parse_mode, and photo fields', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '12345',
imageUrl: 'https://example.com/abc.webp',
caption: 'My caption',
parseMode: 'HTML',
});
expect(fd).toBeInstanceOf(FormData);
expect(fd.get('chat_id')).toBe('12345');
expect(fd.get('caption')).toBe('My caption');
expect(fd.get('parse_mode')).toBe('HTML');
const photo = fd.get('photo');
expect(photo).toBeTruthy();
// File-like (Blob); has a name and a size
expect(typeof photo.name).toBe('string');
expect(photo.size).toBeGreaterThan(0);
});
it('uses a .jpg filename (Telegram uses URL/filename extension for type detection)', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
parseMode: 'HTML',
});
const photo = fd.get('photo');
expect(photo.name).toMatch(/\.jpg$/i);
});
it('includes message_thread_id when provided', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
parseMode: 'HTML',
messageThreadId: 42,
});
expect(fd.get('message_thread_id')).toBe('42');
});
it('omits message_thread_id when not provided', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
parseMode: 'HTML',
});
expect(fd.get('message_thread_id')).toBeNull();
});
it('omits parse_mode when not provided (plain text mode)', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/source.webp',
caption: 'c',
});
expect(fd.get('parse_mode')).toBeNull();
});
it('throws when the image fetch returns non-200', async () => {
mockFetch.mockResolvedValueOnce({
ok: false,
status: 404,
headers: { get: () => null },
arrayBuffer: async () => new ArrayBuffer(0),
});
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/gone.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/404/);
});
it('throws when the image fetch throws (network error)', async () => {
mockFetch.mockRejectedValueOnce(new Error('ECONNREFUSED'));
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/x.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/ECONNREFUSED/);
});
it('throws when the image exceeds 10 MB (Telegram multipart limit)', async () => {
// 11 MB
const big = new Uint8Array(11 * 1024 * 1024);
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes: big }));
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/huge.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/size|large|10/i);
});
it('rejects early when content-length header advertises > 10 MB (avoids download)', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
status: 200,
headers: {
get: (h) => {
const k = h.toLowerCase();
if (k === 'content-type') return 'image/jpeg';
if (k === 'content-length') return String(50 * 1024 * 1024);
return null;
},
},
arrayBuffer: async () => {
throw new Error('should not be called - size check should reject first');
},
});
await expect(
buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/huge.webp',
caption: 'c',
parseMode: 'HTML',
}),
).rejects.toThrow(/size|large|10/i);
});
it('accepts exactly 10 MB images (boundary)', async () => {
const bytes = new Uint8Array(10 * 1024 * 1024);
mockFetch.mockResolvedValueOnce(makeImageResponse({ bytes }));
const fd = await buildPhotoFormData({
chatId: '1',
imageUrl: 'https://example.com/exact.webp',
caption: 'c',
parseMode: 'HTML',
});
expect(fd.get('photo').size).toBe(10 * 1024 * 1024);
});
it('coerces non-string chatId (number) to string in form data', async () => {
mockFetch.mockResolvedValueOnce(makeImageResponse());
const fd = await buildPhotoFormData({
chatId: 999,
imageUrl: 'https://example.com/x.webp',
caption: 'c',
parseMode: 'HTML',
});
expect(fd.get('chat_id')).toBe('999');
});
});

View File

@@ -38,6 +38,20 @@ async function tryReadFile(filepath) {
}
}
function withRealEstateType(data, realEstateType) {
if (!realEstateType?.length || !Array.isArray(data?.resultListItems)) {
return data;
}
const cloned = typeof structuredClone === 'function' ? structuredClone(data) : JSON.parse(JSON.stringify(data));
for (const item of cloned.resultListItems) {
if (item?.type === 'EXPOSE_RESULT' && item?.item) {
item.item.realEstateType = realEstateType;
}
}
return cloned;
}
/**
* Returns fixture HTML for the given URL by mapping hostname → provider name,
* then distinguishing list vs detail pages by comparing the URL path against
@@ -83,7 +97,10 @@ export function buildFetchMock() {
const raw = await tryReadFile(path.join(FIXTURES_DIR, 'immoscout_list.json'));
listData = raw ? JSON.parse(raw) : { resultListItems: [] };
}
return { ok: true, status: 200, json: () => Promise.resolve(listData) };
const requestedType = new URL(urlStr).searchParams.get('realestatetype');
const responseData = withRealEstateType(listData, requestedType);
return { ok: true, status: 200, json: () => Promise.resolve(responseData) };
}
if (urlStr.includes('api.mobile.immobilienscout24.de/expose/')) {

View File

@@ -6,83 +6,89 @@
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
import { get } from '../mocks/mockNotification.js';
import { providerConfig, mockFredy } from '../utils.js';
import { expect, vi } from 'vitest';
import { expect } from 'vitest';
import * as provider from '../../lib/provider/immobilienDe.js';
import * as mockStore from '../mocks/mockStore.js';
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
// One browser shared across the whole suite so both requests (search + detail)
// come from the same warm session, avoiding double cold-start bot detection.
const TEST_TIMEOUT = 120_000;
describe('#immobilien.de testsuite()', () => {
provider.init(providerConfig.immobilienDe, [], []);
it('should test immobilien.de provider', async () => {
const mockedJob = {
id: 'test1',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
const Fredy = await mockFredy();
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
const listing = await fredy.execute();
let browser;
let liveListings;
if (listing == null || listing.length === 0) {
throw new Error('Listings is empty!');
}
beforeAll(async () => {
browser = await launchBrowser(providerConfig.immobilienDe.url);
}, TEST_TIMEOUT);
expect(listing).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('immobilienDe');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.price).toBeTypeOf('string');
expect(notify.size).toBeTypeOf('string');
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
expect(notify.price).toContain('€');
expect(notify.size).toContain('m²');
expect(notify.title).not.toBe('');
expect(notify.link).toContain('https://www.immobilien.de');
expect(notify.address).not.toBe('');
});
afterAll(async () => {
await closeBrowser(browser);
});
it(
'should test immobilien.de provider',
async () => {
const mockedJob = {
id: 'test1',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
const Fredy = await mockFredy();
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
liveListings = await fredy.execute();
if (liveListings == null || liveListings.length === 0) {
throw new Error('Listings is empty!');
}
expect(liveListings).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('immobilienDe');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.price).toBeTypeOf('string');
expect(notify.size).toBeTypeOf('string');
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
expect(notify.price).toContain('€');
expect(notify.size).toContain('m²');
expect(notify.title).not.toBe('');
expect(notify.link).toContain('https://www.immobilien.de');
expect(notify.address).not.toBe('');
});
},
TEST_TIMEOUT,
);
describe('with provider_details enabled', () => {
beforeEach(() => {
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
});
it(
'should enrich listings with details',
async () => {
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
afterEach(() => {
vi.restoreAllMocks();
});
// Call fetchDetails directly on the first live listing — no need to
// re-scrape the search page. The shared browser keeps the session warm.
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
it('should enrich listings with details', async () => {
const Fredy = await mockFredy();
provider.init(providerConfig.immobilienDe, [], []);
const mockedJob = { id: 'test1', notificationAdapter: null, specFilter: null, spatialFilter: null };
const fredy = new Fredy(
provider.config,
mockedJob,
provider.metaInformation.id,
{ checkAndAddEntry: () => false },
undefined,
);
const listings = await fredy.execute();
if (listings == null) return;
expect(listings).toBeInstanceOf(Array);
listings.forEach((listing) => {
expect(listing.link).toContain('https://www.immobilien.de');
expect(listing.address).toBeTypeOf('string');
expect(listing.address).not.toBe('');
if (enriched == null) return;
expect(enriched.link).toContain('https://www.immobilien.de');
expect(enriched.address).toBeTypeOf('string');
expect(enriched.address).not.toBe('');
// description may be null if selectors don't match yet — falls back gracefully
if (listing.description != null) {
expect(listing.description).toBeTypeOf('string');
if (enriched.description != null) {
expect(enriched.description).toBeTypeOf('string');
}
});
});
},
TEST_TIMEOUT,
);
});
});

View File

@@ -3,85 +3,85 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { expect, vi } from 'vitest';
import { expect } from 'vitest';
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
import { mockFredy, providerConfig } from '../utils.js';
import { get } from '../mocks/mockNotification.js';
import * as provider from '../../lib/provider/immoscout.js';
import * as mockStore from '../mocks/mockStore.js';
// immoscout uses the mobile REST API (fetch-based, no browser). Both tests share
// the same module-level listings so the API is only queried once, avoiding
// duplicate requests that could trigger rate-limiting.
const TEST_TIMEOUT = 120_000;
describe('#immoscout provider testsuite()', () => {
provider.init(providerConfig.immoscout, [], []);
it('should test immoscout provider', async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: '',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
return await new Promise((resolve, reject) => {
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
fredy.execute().then((listings) => {
if (listings == null || listings.length === 0) {
reject('Listings is empty!');
return;
}
let liveListings;
expect(listings).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
it(
'should test immoscout provider',
async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: '',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
// check if there is at least one valid notification
const hasValidNotification = notificationObj.payload.some((notify) => {
return (
typeof notify.id === 'string' &&
typeof notify.price === 'string' &&
notify.price.includes('€') &&
typeof notify.size === 'string' &&
notify.size.includes('m²') &&
typeof notify.title === 'string' &&
notify.title !== '' &&
typeof notify.link === 'string' &&
notify.link.includes('https://www.immobilienscout24.de/') &&
typeof notify.address === 'string'
);
return await new Promise((resolve, reject) => {
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
fredy.execute().then((listings) => {
if (listings == null || listings.length === 0) {
reject('Listings is empty!');
return;
}
liveListings = listings;
expect(listings).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
// check if there is at least one valid notification
const hasValidNotification = notificationObj.payload.some((notify) => {
return (
typeof notify.id === 'string' &&
typeof notify.price === 'string' &&
notify.price.includes('€') &&
typeof notify.size === 'string' &&
notify.size.includes('m²') &&
typeof notify.title === 'string' &&
notify.title !== '' &&
typeof notify.link === 'string' &&
notify.link.includes('https://www.immobilienscout24.de/') &&
typeof notify.address === 'string'
);
});
expect(hasValidNotification).toBe(true);
resolve();
});
expect(hasValidNotification).toBe(true);
resolve();
});
});
});
},
TEST_TIMEOUT,
);
describe('with provider_details enabled', () => {
beforeEach(() => {
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
});
it(
'should enrich listings with details',
async () => {
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
afterEach(() => {
vi.restoreAllMocks();
});
// Call fetchDetails directly on the first live listing — no need to
// re-query the search API. immoscout uses fetch (no browser).
const enriched = await provider.config.fetchDetails(liveListings[0]);
it('should enrich listings with details', async () => {
const Fredy = await mockFredy();
provider.init(providerConfig.immoscout, [], []);
const mockedJob = { id: '', notificationAdapter: null, specFilter: null, spatialFilter: null };
const fredy = new Fredy(
provider.config,
mockedJob,
provider.metaInformation.id,
{ checkAndAddEntry: () => false },
undefined,
);
const listings = await fredy.execute();
expect(listings).toBeInstanceOf(Array);
listings.forEach((listing) => {
expect(listing.description).toBeTypeOf('string');
expect(listing.description).not.toBe('');
});
});
expect(enriched).toBeTruthy();
expect(enriched.description).toBeTypeOf('string');
expect(enriched.description).not.toBe('');
},
TEST_TIMEOUT,
);
});
});

View File

@@ -6,87 +6,95 @@
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
import { get } from '../mocks/mockNotification.js';
import { mockFredy, providerConfig } from '../utils.js';
import { expect, vi } from 'vitest';
import { expect } from 'vitest';
import * as provider from '../../lib/provider/immowelt.js';
import * as mockStore from '../mocks/mockStore.js';
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
// One browser shared across the whole suite so both requests (search + detail)
// come from the same warm session. Immowelt's CDN challenges cold sessions
// aggressively; a shared warm browser prevents the second request from being
// blocked as a bot hit.
const TEST_TIMEOUT = 180_000;
describe('#immowelt testsuite()', () => {
it('should test immowelt provider', async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'immowelt',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
provider.init(providerConfig.immowelt, [], []);
let browser;
let liveListings;
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
beforeAll(async () => {
browser = await launchBrowser(providerConfig.immowelt.url);
}, TEST_TIMEOUT);
const listing = await fredy.execute();
if (listing == null || listing.length === 0) {
throw new Error('Listings is empty!');
}
expect(listing).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('immowelt');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
if (notify.price != null) {
expect(notify.price).toBeTypeOf('string');
expect(notify.price).toContain('€');
}
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
expect(notify.size).toBeTypeOf('string');
expect(notify.size).toContain('m²');
}
expect(notify.title).not.toBe('');
expect(notify.link).toContain('https://www.immowelt.de');
expect(notify.address).not.toBe('');
});
afterAll(async () => {
await closeBrowser(browser);
});
it(
'should test immowelt provider',
async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'immowelt',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
provider.init(providerConfig.immowelt, [], []);
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
liveListings = await fredy.execute();
if (liveListings == null || liveListings.length === 0) {
throw new Error('Listings is empty!');
}
expect(liveListings).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('immowelt');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
if (notify.price != null) {
expect(notify.price).toBeTypeOf('string');
expect(notify.price).toContain('€');
}
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
if (notify.size != null && notify.size.trim().toLowerCase() !== 'k.a.') {
expect(notify.size).toBeTypeOf('string');
expect(notify.size).toContain('m²');
}
expect(notify.title).not.toBe('');
expect(notify.link).toContain('https://www.immowelt.de');
expect(notify.address).not.toBe('');
});
},
TEST_TIMEOUT,
);
describe('with provider_details enabled', () => {
beforeEach(() => {
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
});
it(
'should enrich listings with details',
async () => {
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
afterEach(() => {
vi.restoreAllMocks();
});
// Call fetchDetails directly on the first live listing — no need to
// re-scrape the search page. The shared browser keeps the session warm.
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
it('should enrich listings with details', async () => {
const Fredy = await mockFredy();
provider.init(providerConfig.immowelt, [], []);
const mockedJob = { id: 'immowelt', notificationAdapter: null, specFilter: null, spatialFilter: null };
const fredy = new Fredy(
provider.config,
mockedJob,
provider.metaInformation.id,
{ checkAndAddEntry: () => false },
undefined,
);
const listings = await fredy.execute();
expect(listings).toBeInstanceOf(Array);
listings.forEach((listing) => {
expect(listing.link).toContain('https://www.immowelt.de');
expect(listing.address).toBeTypeOf('string');
expect(listing.address).not.toBe('');
expect(enriched).toBeTruthy();
expect(enriched.link).toContain('https://www.immowelt.de');
expect(enriched.address).toBeTypeOf('string');
expect(enriched.address).not.toBe('');
// description is enriched from the detail page; falls back gracefully if blocked
if (listing.description != null) {
expect(listing.description).toBeTypeOf('string');
if (enriched.description != null) {
expect(enriched.description).toBeTypeOf('string');
}
});
});
},
TEST_TIMEOUT,
);
});
});

View File

@@ -6,80 +6,88 @@
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
import { get } from '../mocks/mockNotification.js';
import { mockFredy, providerConfig } from '../utils.js';
import { expect, vi } from 'vitest';
import { expect } from 'vitest';
import * as provider from '../../lib/provider/kleinanzeigen.js';
import * as mockStore from '../mocks/mockStore.js';
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
// One browser shared across the whole suite so both requests (search + detail)
// come from the same warm session. Kleinanzeigen rate-limits cold browser
// sessions; a shared warm browser prevents the second request from being blocked.
const TEST_TIMEOUT = 180_000;
describe('#kleinanzeigen testsuite()', () => {
it('should test kleinanzeigen provider', async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'kleinanzeigen',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
provider.init(providerConfig.kleinanzeigen, [], []);
return await new Promise((resolve, reject) => {
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
let browser;
let liveListings;
fredy.execute().then((listing) => {
if (listing == null || listing.length === 0) {
reject('Listings is empty!');
return;
}
beforeAll(async () => {
browser = await launchBrowser(providerConfig.kleinanzeigen.url);
}, TEST_TIMEOUT);
expect(listing).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('kleinanzeigen');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
expect(notify.title).not.toBe('');
expect(notify.link).toContain('https://www.kleinanzeigen.de');
expect(notify.address).not.toBe('');
});
resolve();
});
});
afterAll(async () => {
await closeBrowser(browser);
});
it(
'should test kleinanzeigen provider',
async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'kleinanzeigen',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
provider.init(providerConfig.kleinanzeigen, [], []);
return await new Promise((resolve, reject) => {
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
fredy.execute().then((listing) => {
if (listing == null || listing.length === 0) {
reject('Listings is empty!');
return;
}
liveListings = listing;
expect(listing).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('kleinanzeigen');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
expect(notify.title).not.toBe('');
expect(notify.link).toContain('https://www.kleinanzeigen.de');
expect(notify.address).not.toBe('');
});
resolve();
});
});
},
TEST_TIMEOUT,
);
describe('with provider_details enabled', () => {
beforeEach(() => {
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
});
it(
'should enrich listings with details',
async () => {
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
afterEach(() => {
vi.restoreAllMocks();
});
// Call fetchDetails directly on the first live listing — no need to
// re-scrape the search page. The shared browser keeps the session warm.
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
it('should enrich listings with details', async () => {
const Fredy = await mockFredy();
provider.init(providerConfig.kleinanzeigen, [], []);
const mockedJob = { id: 'kleinanzeigen', notificationAdapter: null, specFilter: null, spatialFilter: null };
const fredy = new Fredy(
provider.config,
mockedJob,
provider.metaInformation.id,
{ checkAndAddEntry: () => false },
undefined,
);
const listings = await fredy.execute();
expect(listings).toBeInstanceOf(Array);
listings.forEach((listing) => {
expect(listing.link).toContain('https://www.kleinanzeigen.de');
expect(listing.address).toBeTypeOf('string');
expect(listing.address).not.toBe('');
expect(listing.description).toBeTypeOf('string');
expect(listing.description).not.toBe('');
});
});
expect(enriched).toBeTruthy();
expect(enriched.link).toContain('https://www.kleinanzeigen.de');
expect(enriched.address).toBeTypeOf('string');
expect(enriched.address).not.toBe('');
expect(enriched.description).toBeTypeOf('string');
expect(enriched.description).not.toBe('');
},
TEST_TIMEOUT,
);
});
});

View File

@@ -9,81 +9,97 @@ import { mockFredy, providerConfig } from '../utils.js';
import { expect, vi } from 'vitest';
import * as provider from '../../lib/provider/sparkasse.js';
import * as mockStore from '../mocks/mockStore.js';
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
// One browser shared across the whole suite so both requests (search + detail)
// come from the same warm session. This prevents the second request from being
// flagged as a cold-start bot hit.
const TEST_TIMEOUT = 120_000;
describe('#sparkasse testsuite()', () => {
it('should test sparkasse provider', async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'sparkasse',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
provider.init(providerConfig.sparkasse, []);
let browser;
let liveListings;
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
beforeAll(async () => {
browser = await launchBrowser(providerConfig.sparkasse.url);
}, TEST_TIMEOUT);
const listing = await fredy.execute();
if (listing == null || listing.length === 0) {
throw new Error('Listings is empty!');
}
expect(listing).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('sparkasse');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.price).toBeTypeOf('string');
expect(notify.price).toContain('€');
expect(notify.size).toBeTypeOf('string');
expect(notify.size).toContain('m²');
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
expect(notify.size).toBeTypeOf('string');
expect(notify.title).not.toBe('');
expect(notify.address).not.toBe('');
});
afterAll(async () => {
await closeBrowser(browser);
});
it(
'should test sparkasse provider',
async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'sparkasse',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
provider.init(providerConfig.sparkasse, []);
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
liveListings = await fredy.execute();
if (liveListings == null || liveListings.length === 0) {
throw new Error('Listings is empty!');
}
expect(liveListings).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj).toBeTypeOf('object');
expect(notificationObj.serviceName).toBe('sparkasse');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.price).toBeTypeOf('string');
expect(notify.price).toContain('€');
expect(notify.size).toBeTypeOf('string');
expect(notify.size).toContain('m²');
expect(notify.title).toBeTypeOf('string');
expect(notify.link).toBeTypeOf('string');
expect(notify.address).toBeTypeOf('string');
/** check the values if possible **/
expect(notify.size).toBeTypeOf('string');
expect(notify.title).not.toBe('');
expect(notify.address).not.toBe('');
});
},
TEST_TIMEOUT,
);
describe('with provider_details enabled', () => {
beforeEach(() => {
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
});
afterEach(() => {
vi.restoreAllMocks();
});
it('should enrich listings with details', async () => {
const Fredy = await mockFredy();
provider.init(providerConfig.sparkasse, []);
const mockedJob = { id: 'sparkasse', notificationAdapter: null, specFilter: null, spatialFilter: null };
it(
'should enrich listings with details',
async () => {
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
const fredy = new Fredy(
provider.config,
mockedJob,
provider.metaInformation.id,
{ checkAndAddEntry: () => false },
undefined,
);
const listings = await fredy.execute();
expect(listings).toBeInstanceOf(Array);
listings.forEach((listing) => {
expect(listing.link).toContain('https://immobilien.sparkasse.de');
expect(listing.address).toBeTypeOf('string');
expect(listing.address).not.toBe('');
// description is enriched from the detail page; falls back gracefully if bot-detected
if (listing.description != null) {
expect(listing.description).toBeTypeOf('string');
expect(listing.description).not.toBe('');
// Call fetchDetails directly on the first live listing — no need to
// re-scrape the search page. The shared browser keeps the session warm.
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
expect(enriched).toBeTruthy();
expect(enriched.link).toContain('https://immobilien.sparkasse.de');
expect(enriched.address).toBeTypeOf('string');
expect(enriched.address).not.toBe('');
// description is enriched from the detail page; falls back gracefully if blocked
if (enriched.description != null) {
expect(enriched.description).toBeTypeOf('string');
expect(enriched.description).not.toBe('');
}
});
});
},
TEST_TIMEOUT,
);
});
});

View File

@@ -6,77 +6,85 @@
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
import { get } from '../mocks/mockNotification.js';
import { mockFredy, providerConfig } from '../utils.js';
import { expect, vi } from 'vitest';
import { expect } from 'vitest';
import * as provider from '../../lib/provider/wgGesucht.js';
import * as mockStore from '../mocks/mockStore.js';
import { launchBrowser, closeBrowser } from '../../lib/services/extractor/puppeteerExtractor.js';
// One browser shared across the whole suite so both requests (search + detail)
// come from the same warm session, avoiding double cold-start bot detection.
const TEST_TIMEOUT = 120_000;
describe('#wgGesucht testsuite()', () => {
provider.init(providerConfig.wgGesucht, [], []);
it('should test wgGesucht provider', { timeout: 120000 }, async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'wgGesucht',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
return await new Promise((resolve, reject) => {
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, undefined);
let browser;
let liveListings;
fredy.execute().then((listing) => {
if (listing == null || listing.length === 0) {
reject('Listings is empty!');
return;
}
beforeAll(async () => {
browser = await launchBrowser(providerConfig.wgGesucht.url);
}, TEST_TIMEOUT);
expect(listing).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj.serviceName).toBe('wgGesucht');
notificationObj.payload.forEach((notify) => {
expect(notify).toBeTypeOf('object');
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.title).toBeTypeOf('string');
// expect(notify.details).toBeTypeOf('string');
expect(notify.price).toBeTypeOf('string');
expect(notify.price).toContain('€');
expect(notify.link).toBeTypeOf('string');
});
resolve();
});
});
afterAll(async () => {
await closeBrowser(browser);
});
it(
'should test wgGesucht provider',
async () => {
const Fredy = await mockFredy();
const mockedJob = {
id: 'wgGesucht',
notificationAdapter: null,
spatialFilter: null,
specFilter: null,
};
return await new Promise((resolve, reject) => {
const fredy = new Fredy(provider.config, mockedJob, provider.metaInformation.id, similarityCache, browser);
fredy.execute().then((listing) => {
if (listing == null || listing.length === 0) {
reject('Listings is empty!');
return;
}
liveListings = listing;
expect(listing).toBeInstanceOf(Array);
const notificationObj = get();
expect(notificationObj.serviceName).toBe('wgGesucht');
notificationObj.payload.forEach((notify) => {
expect(notify).toBeTypeOf('object');
/** check the actual structure **/
expect(notify.id).toBeTypeOf('string');
expect(notify.title).toBeTypeOf('string');
// expect(notify.details).toBeTypeOf('string');
expect(notify.price).toBeTypeOf('string');
expect(notify.price).toContain('€');
expect(notify.link).toBeTypeOf('string');
});
resolve();
});
});
},
TEST_TIMEOUT,
);
describe('with provider_details enabled', () => {
beforeEach(() => {
vi.spyOn(mockStore, 'getUserSettings').mockReturnValue({ provider_details: [provider.metaInformation.id] });
vi.spyOn(mockStore, 'getKnownListingHashesForJobAndProvider').mockReturnValue([]);
});
it(
'should enrich listings with details',
async () => {
if (!liveListings?.length) throw new Error('No listings from first test to enrich');
afterEach(() => {
vi.restoreAllMocks();
});
// Call fetchDetails directly on the first live listing — no need to
// re-scrape the search page. The shared browser keeps the session warm.
const enriched = await provider.config.fetchDetails(liveListings[0], browser);
it('should enrich listings with details', async () => {
const Fredy = await mockFredy();
provider.init(providerConfig.wgGesucht, [], []);
const mockedJob = { id: 'wgGesucht', notificationAdapter: null, specFilter: null, spatialFilter: null };
const fredy = new Fredy(
provider.config,
mockedJob,
provider.metaInformation.id,
{ checkAndAddEntry: () => false },
undefined,
);
const listings = await fredy.execute();
expect(listings).toBeInstanceOf(Array);
listings.forEach((listing) => {
expect(listing.link).toContain('https://www.wg-gesucht.de');
expect(listing.description).toBeTypeOf('string');
expect(listing.description).not.toBe('');
});
});
expect(enriched).toBeTruthy();
expect(enriched.link).toContain('https://www.wg-gesucht.de');
expect(enriched.description).toBeTypeOf('string');
expect(enriched.description).not.toBe('');
},
TEST_TIMEOUT,
);
});
});

View File

@@ -0,0 +1,37 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { vi, describe, it, expect, beforeEach } from 'vitest';
// Mock the CloakBrowser launcher so no real Chromium binary is needed and we can
// assert which options get forwarded to it.
const { launchMock } = vi.hoisted(() => ({ launchMock: vi.fn() }));
vi.mock('cloakbrowser/puppeteer', () => ({
launch: launchMock,
}));
const { launchBrowser } = await import('../../../lib/services/extractor/puppeteerExtractor.js');
describe('launchBrowser proxy forwarding', () => {
beforeEach(() => {
launchMock.mockReset();
launchMock.mockResolvedValue({ close: async () => {} });
});
it('forwards proxyUrl to CloakBrowser as the proxy option', async () => {
await launchBrowser('https://www.immowelt.de/', { proxyUrl: 'http://user:pass@host:8080' });
expect(launchMock).toHaveBeenCalledTimes(1);
expect(launchMock.mock.calls[0][0]).toMatchObject({ proxy: 'http://user:pass@host:8080' });
});
it('does not set a proxy when no proxyUrl is given', async () => {
await launchBrowser('https://www.immowelt.de/', {});
expect(launchMock).toHaveBeenCalledTimes(1);
expect(launchMock.mock.calls[0][0].proxy).toBeUndefined();
});
});

View File

@@ -4,12 +4,28 @@
*/
import { convertWebToMobile } from '../../../lib/services/immoscout/immoscout-web-translator.js';
import { expect } from 'vitest';
import { expect, vi } from 'vitest';
import { readFile } from 'fs/promises';
import { buildFetchMock } from '../../offlineFixtures.js';
export const testData = JSON.parse(await readFile(new URL('./testdata.json', import.meta.url)));
if (process.env.TEST_MODE === 'offline') {
vi.stubGlobal('fetch', buildFetchMock());
}
describe('#immoscout-mobile URL conversion', () => {
// Test shape URL conversion
it('should convert a full web URL with shape to mobile URL', () => {
const webUrl =
'https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list';
const expectedMobileUrl =
'https://api.mobile.immobilienscout24.de/search/list?ground=240.0-&price=-600000.0&realestatetype=housebuy&searchType=shape&shape=ior~H_kxmAr%60Pig%60%40fzHm%7Bp%40qs%40%7D%60l%40crCidPuol%40wy~%40onVb~Ek%60KhdPcoEhkS%7CHf%7B%7CAtpDtqLjiGngQ%7CxLrlOxyVbzR';
const actualMobileUrl = convertWebToMobile(webUrl);
expect(actualMobileUrl).toBe(expectedMobileUrl);
});
// Test URL conversion
it('should convert a full web URL to mobile URL', () => {
const webUrl =
@@ -30,6 +46,60 @@ describe('#immoscout-mobile URL conversion', () => {
expect(queryParams.get('equipment').split(',')).toEqual(expect.arrayContaining(['garden', 'balcony']));
});
// Test URL conversion of SEO web path for max warmrent. The ImmoScout web UI
// generates this special SEO slug instead of explicit price/pricetype params
// when the user configures a "Warmmiete" filter (real-world URL).
it('should convert a SEO apartment max warmrent path to rent + price + pricetype', () => {
const webUrl =
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-bis-800-euro-warm?livingspace=-800.0&enteredFrom=result_list';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
expect(queryParams.get('price')).toBe('-800');
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
expect(queryParams.get('livingspace')).toBe('-800.0');
});
// Same SEO pattern for houses ("haus-bis-X-euro-warm" → houserent).
it('should convert a SEO house max warmrent path to rent + price + pricetype', () => {
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/haus-bis-1500-euro-warm';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('houserent');
expect(queryParams.get('price')).toBe('-1500');
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
});
// Sanity check: max coldrent ("Kaltmiete") does NOT use an SEO slug. The web
// UI keeps the regular "wohnung-mieten" path and passes explicit
// price + pricetype query params, which the existing translator already
// handles (real-world URL).
it('should convert a max coldrent search via the regular wohnung-mieten path', () => {
const webUrl =
'https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/wohnung-mieten?price=-800.0&livingspace=-800.0&pricetype=rentpermonth&enteredFrom=result_list';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
expect(queryParams.get('price')).toBe('-800.0');
expect(queryParams.get('pricetype')).toBe('rentpermonth');
expect(queryParams.get('geocodes')).toBe('/de/nordrhein-westfalen/duesseldorf');
});
// Explicit query params win over the SEO slug's implicit defaults.
it('should let explicit query params override SEO path price defaults', () => {
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-bis-800-euro-warm?price=100-500';
const converted = convertWebToMobile(webUrl);
const queryParams = new URL(converted).searchParams;
expect(queryParams.get('realestatetype')).toBe('apartmentrent');
expect(queryParams.get('price')).toBe('100-500');
expect(queryParams.get('pricetype')).toBe('calculatedtotalrent');
});
// Test URL conversion with unsupported query parameters
it('should remove unsupported query parameters', () => {
const webUrl = 'https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?minimuminternetspeed=100000';

View File

@@ -18,5 +18,9 @@
"rentHouse": {
"url": "https://www.immobilienscout24.de/Suche/de/nordrhein-westfalen/duesseldorf/haus-mieten?enteredFrom=one_step_search",
"type": "houserent"
},
"buyHouseWithShape": {
"url": "https://www.immobilienscout24.de/Suche/shape/haus-kaufen?shape=aW9yfkhfa3htQXJgUGlnYEBmekhte3BAcXNAfWBsQGNyQ2lkUHVvbEB3eX5Ab25WYn5Fa2BLaGRQY29FaGtTfEhme3xBdHBEdHFMamlHbmdRfHhMcmxPeHlWYnpS&price=-600000.0&ground=240.0-&enteredFrom=result_list",
"type": "housebuy"
}
}

View File

@@ -18,6 +18,7 @@ describe('services/jobs/jobExecutionService', () => {
const busPath = root + '/lib/services/events/event-bus.js';
const jobStoragePath = root + '/lib/services/storage/jobStorage.js';
const userStoragePath = root + '/lib/services/storage/userStorage.js';
const settingsStoragePath = root + '/lib/services/storage/settingsStorage.js';
const brokerPath = root + '/lib/services/sse/sse-broker.js';
const utilsPath = root + '/lib/utils.js';
const loggerPath = root + '/lib/services/logger.js';
@@ -33,11 +34,15 @@ describe('services/jobs/jobExecutionService', () => {
getUsers: () => state.users.slice(),
getUser: (id) => state.users.find((u) => u.id === id) || null,
}));
vi.doMock(settingsStoragePath, () => ({
getSettings: async () => ({}),
}));
vi.doMock(brokerPath, () => ({
sendToUsers: (...args) => calls.sent.push(args),
}));
vi.doMock(utilsPath, () => ({
duringWorkingHoursOrNotSet: () => false,
getPackageVersion: async () => '0.0.0-test',
}));
vi.doMock(loggerPath, () => {
const m = { debug: () => {}, info: () => {}, warn: () => {}, error: () => {} };

View File

@@ -0,0 +1,193 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { vi, describe, it, expect, beforeEach } from 'vitest';
// We mock SqliteConnection so we can assert which SQL the storage layer
// runs and with which params, without spinning up a real SQLite DB.
const calls = {
execute: [],
query: [],
};
const sqliteMock = {
execute: (sql, params) => {
calls.execute.push({ sql, params });
// Default: pretend 1 row was affected (so setListingStatus reports success).
return { changes: 1 };
},
query: (sql, params) => {
calls.query.push({ sql, params });
// Return shape varies by test — overridden via queryHandler when needed.
if (sqliteMock.__queryHandler) return sqliteMock.__queryHandler(sql, params);
return [];
},
__queryHandler: null,
};
vi.mock('../../lib/services/storage/SqliteConnection.js', () => ({
default: sqliteMock,
}));
describe('listingsStorage.setListingStatus', () => {
let listingsStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
sqliteMock.__queryHandler = null;
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
});
it('runs an UPDATE storing a JSON payload with status and setAt', () => {
const before = Date.now();
const changes = listingsStorage.setListingStatus('listing-1', 'Applied');
const after = Date.now();
expect(changes).toBe(1);
expect(calls.execute).toHaveLength(1);
expect(calls.execute[0].sql).toMatch(/UPDATE listings SET status = @status WHERE id = @id/);
expect(calls.execute[0].params.id).toBe('listing-1');
const parsed = JSON.parse(calls.execute[0].params.status);
expect(parsed.status).toBe('applied');
expect(parsed.setAt).toBeGreaterThanOrEqual(before);
expect(parsed.setAt).toBeLessThanOrEqual(after);
});
it('accepts null to clear the status (no JSON wrapping)', () => {
listingsStorage.setListingStatus('listing-2', null);
expect(calls.execute[0].params).toEqual({ id: 'listing-2', status: null });
});
it('rejects invalid statuses', () => {
expect(() => listingsStorage.setListingStatus('listing-3', 'maybe')).toThrow(/Invalid listing status/);
expect(calls.execute).toHaveLength(0);
});
it('returns 0 when no id is supplied (no SQL is run)', () => {
const result = listingsStorage.setListingStatus(null, 'applied');
expect(result).toBe(0);
expect(calls.execute).toHaveLength(0);
});
});
describe('listingsStorage.queryListings statusFilter', () => {
let listingsStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
// Return empty rows for both the count and the page-fetch queries.
sqliteMock.__queryHandler = (sql) => {
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 0 }];
return [];
};
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
});
it("adds 'l.status IS NULL' to WHERE when statusFilter is 'none'", () => {
listingsStorage.queryListings({ statusFilter: 'none', userId: 'u1', isAdmin: true });
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
expect(pageQuery.sql).toMatch(/\(l\.status IS NULL\)/);
});
it('extracts the inner status field via json_extract for a concrete status', () => {
listingsStorage.queryListings({ statusFilter: 'applied', userId: 'u1', isAdmin: true });
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
expect(pageQuery.sql).toMatch(/json_extract\(l\.status, '\$\.status'\) = @statusValue/);
expect(pageQuery.params.statusValue).toBe('applied');
});
it('ignores unknown statusFilter values silently', () => {
listingsStorage.queryListings({ statusFilter: 'bogus', userId: 'u1', isAdmin: true });
const pageQuery = calls.query.find((c) => !/COUNT\(1\)/.test(c.sql));
expect(pageQuery.sql).not.toMatch(/status/i);
});
it('parses the JSON status payload of returned rows into an object', () => {
sqliteMock.__queryHandler = (sql) => {
if (/COUNT\(1\)/.test(sql)) return [{ cnt: 2 }];
return [
{ id: 'a', status: JSON.stringify({ status: 'applied', setAt: 1700000000000 }) },
{ id: 'b', status: null },
];
};
const result = listingsStorage.queryListings({ userId: 'u1', isAdmin: true });
expect(result.result[0].status).toEqual({ status: 'applied', setAt: 1700000000000 });
expect(result.result[1].status).toBeNull();
});
});
describe('listingsStorage.getListingById', () => {
let listingsStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
listingsStorage = await import('../../lib/services/storage/listingsStorage.js');
});
it('parses the JSON status payload of the returned row', () => {
sqliteMock.__queryHandler = () => [
{ id: 'a', status: JSON.stringify({ status: 'rejected', setAt: 1700000000001 }) },
];
const row = listingsStorage.getListingById('a', 'u1', true);
expect(row.status).toEqual({ status: 'rejected', setAt: 1700000000001 });
});
it('returns null status untouched', () => {
sqliteMock.__queryHandler = () => [{ id: 'a', status: null }];
const row = listingsStorage.getListingById('a', 'u1', true);
expect(row.status).toBeNull();
});
it('returns null when no row is found', () => {
sqliteMock.__queryHandler = () => [];
const row = listingsStorage.getListingById('missing', 'u1', true);
expect(row).toBeNull();
});
});
describe('watchListStorage.ensureWatch', () => {
let watchListStorage;
beforeEach(async () => {
calls.execute.length = 0;
calls.query.length = 0;
sqliteMock.__queryHandler = null;
watchListStorage = await import('../../lib/services/storage/watchListStorage.js');
});
it('inserts and reports watched=true on first call', () => {
// After INSERT, createWatch queries for existence and gets a row back.
sqliteMock.__queryHandler = () => [{ ok: 1 }];
const result = watchListStorage.ensureWatch('listing-1', 'user-1');
expect(result).toEqual({ watched: true });
// INSERT should have been issued.
expect(calls.execute.some((c) => /INSERT INTO watch_list/.test(c.sql))).toBe(true);
});
it('returns watched=true when an entry already exists', () => {
// Simulate ON CONFLICT being a no-op: execute reports no changes, then SELECT confirms row exists.
sqliteMock.execute = (sql, params) => {
calls.execute.push({ sql, params });
return { changes: 0 };
};
sqliteMock.__queryHandler = () => [{ ok: 1 }];
const result = watchListStorage.ensureWatch('listing-2', 'user-2');
expect(result).toEqual({ watched: true });
// Restore execute to default for subsequent tests.
sqliteMock.execute = (sql, params) => {
calls.execute.push({ sql, params });
return { changes: 1 };
};
});
it('returns watched=false when listingId or userId is missing', () => {
expect(watchListStorage.ensureWatch(null, 'u')).toEqual({ watched: false });
expect(watchListStorage.ensureWatch('l', null)).toEqual({ watched: false });
expect(calls.execute).toHaveLength(0);
});
});

View File

@@ -713,7 +713,7 @@
</div><!-- /srb-filters -->
<!-- Search button CMS handles via onclick="{{submit()}}"; srbDoSearch() is
<!-- Search button - CMS handles via onclick="{{submit()}}"; srbDoSearch() is
a standalone fallback that skips empty params (used outside CMS context) -->
<button type="button" class="srb-search-btn" id="srbSearchBtn">
<svg width="17" height="17" viewBox="0 0 24 24" fill="none" aria-hidden="true">
@@ -740,7 +740,7 @@
</div><!-- /srb-wrap -->
<!-- ═══════════════════════════════════════════════════════════
MODAL WEITERE FILTER
MODAL - WEITERE FILTER
═══════════════════════════════════════════════════════════ -->
<!-- Screen-reader live region: announces filter changes and modal state -->
<div id="srbLiveRegion" role="status" aria-live="polite" aria-atomic="true" style="position:absolute;width:1px;height:1px;overflow:hidden;clip:rect(0,0,0,0);white-space:nowrap"></div>
@@ -768,7 +768,7 @@
<circle cx="12" cy="10" r="3" stroke="currentColor" stroke-width="2"></circle>
</svg>
<input type="text" class="srb-modal-location-input" id="srbModalLocation" placeholder="Ort, Postleitzahl, Stadt" autocomplete="off" b-event="keyup" role="combobox" aria-autocomplete="list" aria-controls="srbModalLocationList" aria-expanded="false">
<!-- Mirror of the desktop grbWoHidden Piglet updates this via b-model
<!-- Mirror of the desktop grbWoHidden - Piglet updates this via b-model
when select(entry) fires in the modal controller's own scope. -->
<input type="hidden" id="srbModalWoHidden" value="district:2434,2695,2621,2700,2967,2734,2909,2955,2392,2746,2767,2982,2904,2612,2892,2587,2871,2975,2591,2887,2569,2640,2735">
<div b-redrawable="autocomplete" class="srb-autocomplete-wrapper">
@@ -899,7 +899,7 @@
</div>
</div>
<!-- ── Additional criteria CMS server-side rendered per typ+objektart ── -->
<!-- ── Additional criteria - CMS server-side rendered per typ+objektart ── -->
<div class="srb-modal-criteria-groups">
<div class="srb-modal-section srb-modal-section--animated srb-modal-section--criteria" data-valid-searches="[&quot;kaufen_grundstueck&quot;,&quot;kaufen_haus&quot;,&quot;kaufen_rendite&quot;,&quot;kaufen_wohnung&quot;,&quot;mieten_grundstueck&quot;,&quot;mieten_haus&quot;,&quot;mieten_waz&quot;,&quot;mieten_wohnung&quot;]">
<div class="srb-section-body">
@@ -4393,7 +4393,7 @@
void g.offsetHeight;
g.classList.remove('lr-card__gallery--no-transition');
// counter total (CMS doesn't evaluate expressions in attrs read from hidden span)
// counter total (CMS doesn't evaluate expressions in attrs - read from hidden span)
var totalEl = g.querySelector('.lr-card__gallery-total');
var counter = g.querySelector('.lr-card__gallery-counter');
if (counter && totalEl) counter.dataset.total = totalEl.textContent.trim();

View File

@@ -1523,7 +1523,7 @@
<div class="hidden print:flex print:items-center print:justify-between mb-2">
<img src="/static/img/brand/logo.b187f1e54302.svg" class="w-32" alt="ohne-makler Immobilien selbst vermarkten">
<img src="/static/img/brand/logo.b187f1e54302.svg" class="w-32" alt="ohne-makler - Immobilien selbst vermarkten">
<br>
Diese Seite wurde ausgedruckt von:<br>
https://www.ohne-makler.net/immobilien/wohnung-kaufen/nordrhein-westfalen/dusseldorf/

View File

@@ -29,7 +29,7 @@ vi.mock('../lib/services/extractor/puppeteerExtractor.js', async (importOriginal
const { readFixture } = await import('./offlineFixtures.js');
return {
default: (url) => readFixture(url),
launchBrowser: async () => ({ close: async () => {}, __fredy_removeUserDataDir: false }),
launchBrowser: async () => ({ close: async () => {}, isConnected: () => true }),
closeBrowser: async () => {},
};
});

View File

@@ -155,6 +155,7 @@ const routes = {
'GET /api/dashboard': dashboard,
'GET /api/demo': { demoMode: false },
'POST /api/user/settings/news-hash': {},
'POST /api/user/settings/listing-deletion-preference': {},
};
const server = http.createServer((req, res) => {

View File

@@ -95,7 +95,10 @@ async function downloadHtmlProvider(name, providerConfig, launchBrowser, closeBr
const browser = await launchBrowser(providerConfig.url, {});
try {
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, { browser });
const html = await puppeteerExtractor(providerConfig.url, providerConfig.waitForSelector, {
browser,
name: 'dowload_fixtures',
});
if (!html) {
console.warn(` Failed to download ${name}`);

View File

@@ -97,6 +97,7 @@ export default function FredyApp() {
<Route path="/dashboard" element={<Dashboard />} />
<Route path="/jobs" element={<Jobs />} />
<Route path="/listings" element={<Listings />} />
<Route path="/listings/watchlist" element={<Listings mode="watchlist" />} />
<Route path="/listings/listing/:listingId" element={<ListingDetail />} />
<Route path="/map" element={<MapView />} />
<Route path="/watchlistManagement" element={<WatchlistManagement />} />

View File

@@ -12,7 +12,7 @@ import './Index.less';
// Semi UI uses react-dom (not react-dom/client) internally for imperative renders
// like Toast, Notification, etc. In React 19, createRoot was removed from react-dom
// and lives only in react-dom/client inject it so Toast can create its own root.
// and lives only in react-dom/client - inject it so Toast can create its own root.
semiGlobal.config.createRoot = createRoot;
const container = document.getElementById('fredy');

View File

@@ -71,7 +71,7 @@ body {
vertical-align: middle;
}
// Suppress focus outlines Semi uses --semi-color-primary (our red) for all rings
// Suppress focus outlines - Semi uses --semi-color-primary (our red) for all rings
button:focus,
button:focus-visible,
.semi-button:focus,

BIN
ui/src/assets/news/1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 835 KiB

BIN
ui/src/assets/news/2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 367 KiB

View File

@@ -1,10 +1,16 @@
{
"key": "00e6b81777a275f5a140fc9101cb943810db6a69f6eb3927319c5aee0c876542",
"key": "00e6b81777a275f5a140fc9101cb943810db6a69f6eb3927319c5aee0c876221",
"content":
[
{
"title": "Open in...Fredy ;)",
"text": "With the latest version of Fredy, every notification now comes with a link that opens the listing directly inside Fredy. This is also a key step toward an upcoming...milestone :).<br/>To make this work, Fredy needs to know where it lives on the network. We try to guess the public base URL, but lets be honest, you probably know better. Take a quick look at the baseUrl in the system settings and fix it if it looks off."
"title": "Table overview for listings",
"text": "Thanks to https://github.com/datenwurm, we now have a table overview for listings. If you decide to use the table view, the decision will be stored.",
"media": "1.png"
},
{
"title": "Table overview for jobs",
"text": "Based on datenwurm's, work, I created a table overview for jobs. If you decide to use the table view, the decision will be stored.",
"media": "2.png"
}
]
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 242 KiB

BIN
ui/src/assets/no_image.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

View File

@@ -3,8 +3,8 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { useState } from 'react';
import { Modal, Radio, RadioGroup, Typography } from '@douyinfe/semi-ui-19';
import { useState, useEffect } from 'react';
import { Modal, Radio, RadioGroup, Typography, Checkbox } from '@douyinfe/semi-ui-19';
const { Text } = Typography;
@@ -15,11 +15,24 @@ const ListingDeletionModal = ({
title = 'Delete Listings',
showOptions = true,
message = 'How would you like to delete the selected listing(s)?',
defaultDeleteType = 'soft',
}) => {
const [deleteType, setDeleteType] = useState('soft');
const [remember, setRemember] = useState(false);
useEffect(() => {
if (visible) {
setDeleteType(defaultDeleteType);
setRemember(false);
}
}, [visible, defaultDeleteType]);
const handleOk = () => {
onConfirm(!showOptions || deleteType === 'hard');
if (showOptions) {
onConfirm(deleteType === 'hard', remember);
} else {
onConfirm(true);
}
};
return (
@@ -36,32 +49,37 @@ const ListingDeletionModal = ({
<Text>{message}</Text>
</div>
{showOptions && (
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Mark as deleted (Soft Delete)</Text>
<br />
<Text type="secondary">
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
scraping session.
</Text>
</div>
</Radio>
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Remove from database (Hard Delete)</Text>
<br />
<Text type="secondary">
Listings are completely removed from the database.
<>
<RadioGroup value={deleteType} onChange={(e) => setDeleteType(e.target.value)} style={{ width: '100%' }}>
<Radio value="soft" style={{ alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Mark as deleted (Soft Delete)</Text>
<br />
<Text type="warning">
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
previously found.
<Text type="secondary">
Listings are kept in the database but marked as hidden. They will <b>not</b> re-appear during the next
scraping session.
</Text>
</Text>
</div>
</Radio>
</RadioGroup>
</div>
</Radio>
<Radio value="hard" style={{ marginTop: 16, alignItems: 'flex-start', width: '100%' }}>
<div style={{ marginLeft: 8 }}>
<Text strong>Remove from database (Hard Delete)</Text>
<br />
<Text type="secondary">
Listings are completely removed from the database.
<br />
<Text type="warning">
Consequence: They might re-appear when scraping the next time because Fredy won't know they were
previously found.
</Text>
</Text>
</div>
</Radio>
</RadioGroup>
<Checkbox checked={remember} onChange={(e) => setRemember(e.target.checked)} style={{ marginTop: 16 }}>
Remember my choice and skip this dialog next time
</Checkbox>
</>
)}
</Modal>
);

View File

@@ -21,6 +21,7 @@ import {
Empty,
Radio,
RadioGroup,
Tooltip,
} from '@douyinfe/semi-ui-19';
import {
IconAlertTriangle,
@@ -35,6 +36,8 @@ import {
IconArrowUp,
IconArrowDown,
IconHome,
IconGridView,
IconList,
} from '@douyinfe/semi-icons';
import { useNavigate } from 'react-router-dom';
import ListingDeletionModal from '../../ListingDeletionModal.jsx';
@@ -42,6 +45,7 @@ import { useActions, useSelector } from '../../../services/state/store.js';
import { xhrDelete, xhrPut, xhrPost } from '../../../services/xhr.js';
import { debounce } from '../../../utils';
import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-illustrations';
import JobsTable from '../../table/JobsTable.jsx';
import './JobGrid.less';
@@ -54,6 +58,11 @@ const JobGrid = () => {
const actions = useActions();
const navigate = useNavigate();
const userSettings = useSelector((state) => state.userSettings.settings);
const viewMode = userSettings?.jobs_view_mode ?? 'grid';
const listingDeletionPref = userSettings?.listing_deletion_preference;
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
const [page, setPage] = useState(1);
const pageSize = 12;
@@ -135,13 +144,21 @@ const JobGrid = () => {
};
const onListingRemoval = (jobId) => {
setPendingDeletion({ type: 'listings', jobId });
const deletion = { type: 'listings', jobId };
if (listingDeletionPref?.skipPrompt) {
confirmDeletion(listingDeletionPref.hardDelete, false, deletion);
return;
}
setPendingDeletion(deletion);
setDeleteModalVisible(true);
};
const confirmDeletion = async (hardDelete) => {
const { type, jobId } = pendingDeletion;
const confirmDeletion = async (hardDelete, remember, deletion = pendingDeletion) => {
const { type, jobId } = deletion;
try {
if (remember && type === 'listings') {
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
}
if (type === 'job') {
await xhrDelete('/api/jobs', { jobId });
Toast.success('Job and listings successfully removed');
@@ -167,7 +184,7 @@ const JobGrid = () => {
Toast.success('Job status successfully changed');
loadData();
} catch (error) {
Toast.error(error);
Toast.error(error.error);
}
};
@@ -234,6 +251,27 @@ const JobGrid = () => {
onClick={() => setSortDir((d) => (d === 'asc' ? 'desc' : 'asc'))}
title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
/>
<div className="jobGrid__topbar__view-toggle">
<Tooltip content="Grid view">
<Button
icon={<IconGridView />}
theme={viewMode === 'grid' ? 'solid' : 'borderless'}
onClick={() => actions.userSettings.setJobsViewMode('grid')}
aria-label="Grid view"
aria-pressed={viewMode === 'grid'}
/>
</Tooltip>
<Tooltip content="Table view">
<Button
icon={<IconList />}
theme={viewMode === 'table' ? 'solid' : 'borderless'}
onClick={() => actions.userSettings.setJobsViewMode('table')}
aria-label="Table view"
aria-pressed={viewMode === 'table'}
/>
</Tooltip>
</div>
</div>
{(jobsData?.result || []).length === 0 && (
@@ -244,136 +282,144 @@ const JobGrid = () => {
/>
)}
<Row gutter={[16, 16]}>
{(jobsData?.result || []).map((job) => (
<Col key={job.id} xs={24} sm={12} md={12} lg={8} xl={8} xxl={6}>
<Card className="jobGrid__card" bodyStyle={{ padding: '16px' }}>
<div className="jobGrid__card__header">
<div className="jobGrid__card__name">
<span className={`jobGrid__card__dot${job.enabled ? ' jobGrid__card__dot--active' : ''}`} />
<Title heading={5} ellipsis={{ showTooltip: true }} className="jobGrid__title">
{job.name}
</Title>
{viewMode === 'grid' ? (
<Row gutter={[16, 16]}>
{(jobsData?.result || []).map((job) => (
<Col key={job.id} xs={24} sm={12} md={12} lg={8} xl={8} xxl={6}>
<Card className="jobGrid__card" bodyStyle={{ padding: '16px' }}>
<div className="jobGrid__card__header">
<div className="jobGrid__card__name">
<span className={`jobGrid__card__dot${job.enabled ? ' jobGrid__card__dot--active' : ''}`} />
<Title heading={5} ellipsis={{ showTooltip: true }} className="jobGrid__title">
{job.name}
</Title>
</div>
<div style={{ display: 'flex', alignItems: 'center', gap: 6, flexShrink: 0 }}>
{job.isOnlyShared && (
<Popover content={getPopoverContent('This job has been shared with you — read only.')}>
<div>
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
</div>
</Popover>
)}
{job.running && (
<Tag color="green" variant="light" size="small">
RUNNING
</Tag>
)}
</div>
</div>
<div style={{ display: 'flex', alignItems: 'center', gap: 6, flexShrink: 0 }}>
{job.isOnlyShared && (
<Popover
content={getPopoverContent(
'This job has been shared with you by another user, therefor it is read-only.',
)}
>
<div className="jobGrid__card__stats">
<div className="jobGrid__card__stat jobGrid__card__stat--blue">
<span className="jobGrid__card__stat__number">{job.numberOfFoundListings || 0}</span>
<span className="jobGrid__card__stat__label">
<IconHome size="small" /> Listings
</span>
</div>
<div className="jobGrid__card__stat jobGrid__card__stat--orange">
<span className="jobGrid__card__stat__number">{job.provider?.length || 0}</span>
<span className="jobGrid__card__stat__label">
<IconBriefcase size="small" /> Providers
</span>
</div>
<div className="jobGrid__card__stat jobGrid__card__stat--purple">
<span className="jobGrid__card__stat__number">{job.notificationAdapter?.length || 0}</span>
<span className="jobGrid__card__stat__label">
<IconBell size="small" /> Adapters
</span>
</div>
</div>
<Divider margin="12px" />
<div className="jobGrid__card__footer">
<div style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
<Switch
onChange={(checked) => onJobStatusChanged(job.id, checked)}
checked={job.enabled}
disabled={job.isOnlyShared}
size="small"
/>
<Text type="secondary" size="small">
Active
</Text>
</div>
<div className="jobGrid__actions">
<Popover content={getPopoverContent('Run Job')}>
<div>
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
<Button
type="primary"
style={{ background: '#21aa21b5' }}
size="small"
theme="solid"
icon={<IconPlayCircle />}
disabled={job.isOnlyShared || job.running}
onClick={() => onJobRun(job.id)}
/>
</div>
</Popover>
)}
{job.running && (
<Tag color="green" variant="light" size="small">
RUNNING
</Tag>
)}
<Popover content={getPopoverContent('Edit a Job')}>
<div>
<Button
type="secondary"
size="small"
icon={<IconEdit />}
disabled={job.isOnlyShared}
onClick={() => navigate(`/jobs/edit/${job.id}`)}
/>
</div>
</Popover>
<Popover content={getPopoverContent('Clone Job')}>
<div>
<Button
type="tertiary"
size="small"
icon={<IconCopy />}
disabled={job.isOnlyShared}
onClick={() => navigate('/jobs/new', { state: { cloneFrom: job.id } })}
/>
</div>
</Popover>
<Popover content={getPopoverContent('Delete all found Listings of this Job')}>
<div>
<Button
type="danger"
size="small"
icon={<IconDescend2 />}
disabled={job.isOnlyShared}
onClick={() => onListingRemoval(job.id)}
/>
</div>
</Popover>
<Popover content={getPopoverContent('Delete Job')}>
<div>
<Button
type="danger"
size="small"
icon={<IconDelete />}
disabled={job.isOnlyShared}
onClick={() => onJobRemoval(job.id)}
/>
</div>
</Popover>
</div>
</div>
</div>
<div className="jobGrid__card__stats">
<div className="jobGrid__card__stat jobGrid__card__stat--blue">
<span className="jobGrid__card__stat__number">{job.numberOfFoundListings || 0}</span>
<span className="jobGrid__card__stat__label">
<IconHome size="small" /> Listings
</span>
</div>
<div className="jobGrid__card__stat jobGrid__card__stat--orange">
<span className="jobGrid__card__stat__number">{job.provider.length || 0}</span>
<span className="jobGrid__card__stat__label">
<IconBriefcase size="small" /> Providers
</span>
</div>
<div className="jobGrid__card__stat jobGrid__card__stat--purple">
<span className="jobGrid__card__stat__number">{job.notificationAdapter.length || 0}</span>
<span className="jobGrid__card__stat__label">
<IconBell size="small" /> Adapters
</span>
</div>
</div>
<Divider margin="12px" />
<div className="jobGrid__card__footer">
<div style={{ display: 'flex', alignItems: 'center', gap: 6 }}>
<Switch
onChange={(checked) => onJobStatusChanged(job.id, checked)}
checked={job.enabled}
disabled={job.isOnlyShared}
size="small"
/>
<Text type="secondary" size="small">
Active
</Text>
</div>
<div className="jobGrid__actions">
<Popover content={getPopoverContent('Run Job')}>
<div>
<Button
type="primary"
style={{ background: '#21aa21b5' }}
size="small"
theme="solid"
icon={<IconPlayCircle />}
disabled={job.isOnlyShared || job.running}
onClick={() => onJobRun(job.id)}
/>
</div>
</Popover>
<Popover content={getPopoverContent('Edit a Job')}>
<div>
<Button
type="secondary"
size="small"
icon={<IconEdit />}
disabled={job.isOnlyShared}
onClick={() => navigate(`/jobs/edit/${job.id}`)}
/>
</div>
</Popover>
<Popover content={getPopoverContent('Clone Job')}>
<div>
<Button
type="tertiary"
size="small"
icon={<IconCopy />}
disabled={job.isOnlyShared}
onClick={() => navigate('/jobs/new', { state: { cloneFrom: job.id } })}
/>
</div>
</Popover>
<Popover content={getPopoverContent('Delete all found Listings of this Job')}>
<div>
<Button
type="danger"
size="small"
icon={<IconDescend2 />}
disabled={job.isOnlyShared}
onClick={() => onListingRemoval(job.id)}
/>
</div>
</Popover>
<Popover content={getPopoverContent('Delete Job')}>
<div>
<Button
type="danger"
size="small"
icon={<IconDelete />}
disabled={job.isOnlyShared}
onClick={() => onJobRemoval(job.id)}
/>
</div>
</Popover>
</div>
</div>
</Card>
</Col>
))}
</Row>
</Card>
</Col>
))}
</Row>
) : (
<JobsTable
jobs={jobsData?.result || []}
onRun={onJobRun}
onEdit={(id) => navigate(`/jobs/edit/${id}`)}
onClone={(id) => navigate('/jobs/new', { state: { cloneFrom: id } })}
onDeleteListings={onListingRemoval}
onDeleteJob={onJobRemoval}
onStatusChange={onJobStatusChanged}
/>
)}
{(jobsData?.result || []).length > 0 && jobsData?.totalNumber > 12 && (
<div className="jobGrid__pagination">
<Pagination
@@ -389,6 +435,7 @@ const JobGrid = () => {
visible={deleteModalVisible}
title={pendingDeletion?.type === 'job' ? 'Delete Job' : 'Delete Listings'}
showOptions={pendingDeletion?.type !== 'job'}
defaultDeleteType={defaultDeleteType}
message={
pendingDeletion?.type === 'job'
? 'Are you sure you want to delete this job? All associated listings will be removed from the database.'

View File

@@ -17,6 +17,12 @@
flex: 1;
min-width: 160px;
}
&__view-toggle {
display: flex;
gap: 4px;
flex-shrink: 0;
}
}
&__card {

View File

@@ -3,14 +3,7 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { useState, useEffect, useMemo } from 'react';
import {
useSearchParamState,
parseNumber,
parseString,
parseNullableBoolean,
} from '../../../hooks/useSearchParamState.js';
import { Button, Pagination, Toast, Input, Select, Empty, Radio, RadioGroup, Tooltip } from '@douyinfe/semi-ui-19';
import { Button, Tooltip } from '@douyinfe/semi-ui-19';
import {
IconBriefcase,
IconCart,
@@ -19,323 +12,126 @@ import {
IconMapPin,
IconStar,
IconStarStroked,
IconSearch,
IconEyeOpened,
IconArrowUp,
IconArrowDown,
} from '@douyinfe/semi-icons';
import { useNavigate, useSearchParams } from 'react-router-dom';
import ListingDeletionModal from '../../ListingDeletionModal.jsx';
import no_image from '../../../assets/no_image.jpg';
import no_image from '../../../assets/no_image.png';
import * as timeService from '../../../services/time/timeService.js';
import { xhrDelete, xhrPost } from '../../../services/xhr.js';
import { useActions, useSelector } from '../../../services/state/store.js';
import { debounce } from '../../../utils';
import StatusControl from '../../listings/StatusControl.jsx';
import './ListingsGrid.less';
import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-illustrations';
const ListingsGrid = () => {
const listingsData = useSelector((state) => state.listingsData);
const providers = useSelector((state) => state.provider);
const jobs = useSelector((state) => state.jobsData.jobs);
const actions = useActions();
const navigate = useNavigate();
const sp = useSearchParams();
const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
const pageSize = 40;
const [sortField, setSortField] = useSearchParamState(sp, 'sort', 'created_at', parseString);
const [sortDir, setSortDir] = useSearchParamState(sp, 'dir', 'desc', parseString);
const [freeTextFilter, setFreeTextFilter] = useSearchParamState(sp, 'q', null, parseString);
const [watchListFilter, setWatchListFilter] = useSearchParamState(sp, 'watch', null, parseNullableBoolean);
const [jobNameFilter, setJobNameFilter] = useSearchParamState(sp, 'job', null, parseString);
const [activityFilter, setActivityFilter] = useSearchParamState(sp, 'active', null, parseNullableBoolean);
const [providerFilter, setProviderFilter] = useSearchParamState(sp, 'provider', null, parseString);
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
const [listingToDelete, setListingToDelete] = useState(null);
const loadData = () => {
actions.listingsData.getListingsData({
page,
pageSize,
sortfield: sortField,
sortdir: sortDir,
freeTextFilter,
filter: { watchListFilter, jobNameFilter, activityFilter, providerFilter },
});
};
useEffect(() => {
loadData();
}, [page, sortField, sortDir, freeTextFilter, providerFilter, activityFilter, jobNameFilter, watchListFilter]);
const handleFilterChange = useMemo(
() =>
debounce((value) => {
setFreeTextFilter(value || null);
setPage(1);
}, 500),
[],
);
useEffect(() => {
return () => {
// cleanup debounced handler to avoid memory leaks
handleFilterChange.cancel && handleFilterChange.cancel();
};
}, [handleFilterChange]);
const handleWatch = async (e, item) => {
e.preventDefault();
e.stopPropagation();
try {
await xhrPost('/api/listings/watch', { listingId: item.id });
Toast.success(item.isWatched === 1 ? 'Listing removed from Watchlist' : 'Listing added to Watchlist');
loadData();
} catch (e) {
console.error(e);
Toast.error('Failed to operate Watchlist');
}
};
const handlePageChange = (_page) => {
setPage(_page);
};
const confirmDeletion = async (hardDelete) => {
try {
await xhrDelete('/api/listings/', { ids: [listingToDelete], hardDelete });
Toast.success('Listing successfully removed');
loadData();
} catch (error) {
Toast.error(error.message || 'Error deleting listing');
} finally {
setDeleteModalVisible(false);
setListingToDelete(null);
}
};
return (
<div className="listingsGrid">
<div className="listingsGrid__topbar">
<Input
className="listingsGrid__topbar__search"
prefix={<IconSearch />}
showClear
placeholder="Search"
defaultValue={freeTextFilter ?? ''}
onChange={handleFilterChange}
/>
<RadioGroup
type="button"
buttonSize="middle"
value={activityFilter === null ? 'all' : String(activityFilter)}
onChange={(e) => {
const v = e.target.value;
setActivityFilter(v === 'all' ? null : v === 'true');
setPage(1);
}}
>
<Radio value="all">All</Radio>
<Radio value="true">Active</Radio>
<Radio value="false">Inactive</Radio>
</RadioGroup>
<RadioGroup
type="button"
buttonSize="middle"
value={watchListFilter === null ? 'all' : String(watchListFilter)}
onChange={(e) => {
const v = e.target.value;
setWatchListFilter(v === 'all' ? null : v === 'true');
setPage(1);
}}
>
<Radio value="all">All</Radio>
<Radio value="true">Watched</Radio>
<Radio value="false">Unwatched</Radio>
</RadioGroup>
<Select
placeholder="Provider"
showClear
onChange={(val) => {
setProviderFilter(val);
setPage(1);
}}
value={providerFilter}
style={{ width: 130 }}
>
{providers?.map((p) => (
<Select.Option key={p.id} value={p.id}>
{p.name}
</Select.Option>
))}
</Select>
<Select
placeholder="Job"
showClear
onChange={(val) => {
setJobNameFilter(val);
setPage(1);
}}
value={jobNameFilter}
style={{ width: 130 }}
>
{jobs?.map((j) => (
<Select.Option key={j.id} value={j.id}>
{j.name}
</Select.Option>
))}
</Select>
<Select prefix="Sort by" style={{ width: 185 }} value={sortField} onChange={(val) => setSortField(val)}>
<Select.Option value="job_name">Job Name</Select.Option>
<Select.Option value="created_at">Listing Date</Select.Option>
<Select.Option value="price">Price</Select.Option>
<Select.Option value="provider">Provider</Select.Option>
</Select>
<Button
icon={sortDir === 'asc' ? <IconArrowUp /> : <IconArrowDown />}
onClick={() => setSortDir((d) => (d === 'asc' ? 'desc' : 'asc'))}
title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
/>
</div>
{(listingsData?.result || []).length === 0 && (
<Empty
image={<IllustrationNoResult />}
darkModeImage={<IllustrationNoResultDark />}
description="No listings available yet..."
/>
)}
<div className="listingsGrid__grid">
{(listingsData?.result || []).map((item) => (
<div
key={item.id}
className="listingsGrid__card"
style={{ cursor: 'pointer' }}
role="button"
tabIndex={0}
onClick={() => navigate(`/listings/listing/${item.id}`)}
onKeyDown={(e) => {
if (e.key === 'Enter' || e.key === ' ') navigate(`/listings/listing/${item.id}`);
}}
>
<div className="listingsGrid__card__image-wrapper">
<img
src={item.image_url || no_image}
alt={item.title}
onError={(e) => {
e.target.src = no_image;
}}
/>
{!item.is_active && (
<div className="listingsGrid__card__inactive-watermark">
<span>Inactive</span>
</div>
)}
<button
type="button"
className="listingsGrid__card__star"
onClick={(e) => handleWatch(e, item)}
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
>
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
</button>
</div>
<div className="listingsGrid__card__body">
<div className="listingsGrid__card__title" title={item.title}>
{item.title}
</div>
{item.price && (
<div className="listingsGrid__card__price">
<IconCart size="small" />
{item.price}
</div>
)}
{item.address && (
<div className="listingsGrid__card__meta">
<IconMapPin />
{item.address}
</div>
)}
<div className="listingsGrid__card__meta">
<IconBriefcase />
{item.provider}
</div>
<div className="listingsGrid__card__provider">{timeService.format(item.created_at, false)}</div>
</div>
<div className="listingsGrid__card__actions" onClick={(e) => e.stopPropagation()}>
<Tooltip content="Original Listing">
<Button
size="small"
icon={<IconLink />}
style={{ color: '#60a5fa' }}
theme="borderless"
onClick={(e) => {
e.stopPropagation();
window.open(item.link, '_blank');
}}
/>
</Tooltip>
<Tooltip content="View in Fredy">
<Button
size="small"
icon={<IconEyeOpened />}
style={{ color: '#34d399' }}
theme="borderless"
onClick={(e) => {
e.stopPropagation();
navigate(`/listings/listing/${item.id}`);
}}
/>
</Tooltip>
<Tooltip content="Remove">
<Button
size="small"
icon={<IconDelete />}
style={{ color: '#fb7185' }}
theme="borderless"
onClick={(e) => {
e.stopPropagation();
setListingToDelete(item.id);
setDeleteModalVisible(true);
}}
/>
</Tooltip>
</div>
</div>
))}
</div>
{(listingsData?.result || []).length > 0 && (
<div className="listingsGrid__pagination">
<Pagination
currentPage={page}
pageSize={pageSize}
total={listingsData?.totalNumber || 0}
onPageChange={handlePageChange}
showSizeChanger={false}
/>
</div>
)}
<ListingDeletionModal
visible={deleteModalVisible}
onConfirm={confirmDeletion}
onCancel={() => {
setDeleteModalVisible(false);
setListingToDelete(null);
/**
* @param {{ listings: object[], onWatch: Function, onNavigate: Function, onDelete: Function, onStatusChange: Function }} props
*/
const ListingsGrid = ({ listings, onWatch, onNavigate, onDelete, onStatusChange }) => (
<div className="listingsGrid__grid">
{listings.map((item) => (
<div
key={item.id}
className="listingsGrid__card"
style={{ cursor: 'pointer' }}
role="button"
tabIndex={0}
onClick={() => onNavigate(item.id)}
onKeyDown={(e) => {
if (e.key === 'Enter' || e.key === ' ') onNavigate(item.id);
}}
/>
</div>
);
};
>
<div className="listingsGrid__card__image-wrapper">
<img
src={item.image_url || no_image}
alt={item.title}
onError={(e) => {
e.target.src = no_image;
}}
/>
{!item.is_active && (
<div className="listingsGrid__card__inactive-watermark">
<span>Inactive</span>
</div>
)}
<Tooltip content={item.isWatched === 1 ? 'Remove from Watchlist' : 'Add to Watchlist'}>
<button
type="button"
className="listingsGrid__card__star"
onClick={(e) => onWatch(e, item)}
aria-label={item.isWatched === 1 ? 'Remove from watchlist' : 'Add to watchlist'}
>
{item.isWatched === 1 ? <IconStar /> : <IconStarStroked />}
</button>
</Tooltip>
</div>
<div className="listingsGrid__card__body">
<div className="listingsGrid__card__title" title={item.title}>
{item.title}
</div>
{item.price && (
<div className="listingsGrid__card__price">
<IconCart size="small" />
{item.price}
</div>
)}
{item.address && (
<div className="listingsGrid__card__meta">
<IconMapPin />
{item.address}
</div>
)}
<div className="listingsGrid__card__meta">
<IconBriefcase />
{item.provider}
</div>
<div className="listingsGrid__card__provider">{timeService.format(item.created_at, false)}</div>
</div>
<div className="listingsGrid__card__actions" onClick={(e) => e.stopPropagation()}>
<StatusControl
status={item.status?.status ?? null}
compact
onChange={(next) => onStatusChange?.(item, next)}
onTriggerClick={(e) => e.stopPropagation()}
/>
<Tooltip content="Original Listing">
<Button
size="small"
icon={<IconLink />}
style={{ color: '#60a5fa' }}
theme="borderless"
onClick={(e) => {
e.stopPropagation();
window.open(item.link, '_blank');
}}
/>
</Tooltip>
<Tooltip content="View in Fredy">
<Button
size="small"
icon={<IconEyeOpened />}
style={{ color: '#34d399' }}
theme="borderless"
onClick={(e) => {
e.stopPropagation();
onNavigate(item.id);
}}
/>
</Tooltip>
<Tooltip content="Remove">
<Button
size="small"
icon={<IconDelete />}
style={{ color: '#fb7185' }}
theme="borderless"
onClick={(e) => {
e.stopPropagation();
onDelete(item.id);
}}
/>
</Tooltip>
</div>
</div>
))}
</div>
);
export default ListingsGrid;

View File

@@ -1,181 +1,142 @@
@import '../../../tokens.less';
.listingsGrid {
&__topbar {
.listingsGrid__grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(240px, 1fr));
gap: 12px;
}
.listingsGrid__card {
background: @color-elevated !important;
border: 1px solid @color-border !important;
border-radius: @radius-card !important;
overflow: hidden;
transition: box-shadow @transition-card;
display: flex;
flex-direction: column;
&:hover {
box-shadow: 0 6px 24px -4px rgba(0,0,0,0.6);
}
&__image-wrapper {
position: relative;
height: 160px;
overflow: hidden;
flex-shrink: 0;
img {
width: 100%;
height: 100%;
object-fit: cover;
display: block;
}
}
&__inactive-watermark {
position: absolute;
inset: 0;
display: flex;
align-items: center;
gap: @space-3;
margin-bottom: @space-4;
flex-wrap: wrap;
justify-content: center;
background: rgba(0,0,0,0.35);
&__search {
min-width: 200px;
flex: 1;
}
@media (max-width: 768px) {
.listingsGrid__topbar__search {
width: 100%;
flex: unset;
}
.semi-radio-group {
flex: 1;
}
.semi-select {
flex: 1;
min-width: 100px;
width: auto !important;
}
span {
font-size: 18px;
font-weight: 800;
color: rgba(251,113,133,0.9);
text-transform: uppercase;
letter-spacing: 0.15em;
transform: rotate(-30deg);
border: 2px solid rgba(251,113,133,0.5);
padding: 4px 12px;
border-radius: @radius-chip;
backdrop-filter: blur(2px);
}
}
&__grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(240px, 1fr));
gap: 12px;
}
&__card {
background: @color-elevated !important;
border: 1px solid @color-border !important;
border-radius: @radius-card !important;
overflow: hidden;
transition: transform @transition-card, box-shadow @transition-card;
&__star {
position: absolute;
top: 8px;
right: 8px;
background: rgba(0,0,0,0.5);
border: none;
border-radius: 50%;
width: 28px;
height: 28px;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
cursor: pointer;
transition: background @transition-fast;
padding: 0;
&:hover {
transform: translateY(-2px);
box-shadow: 0 6px 24px -4px rgba(0,0,0,0.6);
background: rgba(0,0,0,0.75);
}
&__image-wrapper {
position: relative;
height: 160px;
overflow: hidden;
flex-shrink: 0;
img {
width: 100%;
height: 100%;
object-fit: cover;
display: block;
}
svg {
color: @color-accent;
font-size: 14px;
}
}
&__inactive-watermark {
position: absolute;
inset: 0;
display: flex;
align-items: center;
justify-content: center;
background: rgba(0,0,0,0.35);
&__body {
padding: 12px;
display: flex;
flex-direction: column;
gap: 4px;
flex: 1;
}
span {
font-size: 18px;
font-weight: 800;
color: rgba(251,113,133,0.9);
text-transform: uppercase;
letter-spacing: 0.15em;
transform: rotate(-30deg);
border: 2px solid rgba(251,113,133,0.5);
padding: 4px 12px;
border-radius: @radius-chip;
backdrop-filter: blur(2px);
}
}
&__title {
font-weight: 700;
font-size: @text-sm;
color: @color-text;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
&__star {
position: absolute;
top: 8px;
right: 8px;
background: rgba(0,0,0,0.5);
border: none;
border-radius: 50%;
width: 28px;
height: 28px;
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
transition: background @transition-fast;
padding: 0;
&__price {
font-size: @text-base;
font-weight: 600;
color: @color-success;
display: flex;
align-items: center;
gap: 4px;
}
&:hover {
background: rgba(0,0,0,0.75);
}
&__meta {
font-size: @text-xs;
color: @color-muted;
display: flex;
align-items: center;
gap: 4px;
svg {
color: @color-accent;
font-size: 14px;
}
}
&__body {
padding: 12px;
display: flex;
flex-direction: column;
gap: 4px;
flex: 1;
}
&__title {
font-weight: 700;
font-size: @text-sm;
color: @color-text;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
&__price {
font-size: @text-base;
font-weight: 600;
color: @color-success;
display: flex;
align-items: center;
gap: 4px;
}
&__meta {
font-size: @text-xs;
color: @color-muted;
display: flex;
align-items: center;
gap: 4px;
.semi-icon {
font-size: 11px;
color: @color-faint;
}
}
&__provider {
font-size: @text-xs;
.semi-icon {
font-size: 11px;
color: @color-faint;
}
}
&__actions {
display: flex;
justify-content: space-around;
padding: 8px 12px;
border-top: 1px solid @color-border;
gap: 4px;
margin-top: auto;
&__provider {
font-size: @text-xs;
color: @color-faint;
}
button {
flex: 1;
border: none !important;
border-radius: @radius-chip !important;
}
&__actions {
display: flex;
justify-content: space-around;
padding: 8px 12px;
border-top: 1px solid @color-border;
gap: 4px;
margin-top: auto;
button {
flex: 1;
border: none !important;
border-radius: @radius-chip !important;
}
}
&__pagination {
margin-top: @space-4;
display: flex;
justify-content: center;
}
}

View File

@@ -0,0 +1,343 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { useState, useEffect, useMemo } from 'react';
import {
useSearchParamState,
parseNumber,
parseString,
parseNullableBoolean,
} from '../../hooks/useSearchParamState.js';
import { Button, Pagination, Toast, Input, Select, Empty, Radio, RadioGroup, Tooltip } from '@douyinfe/semi-ui-19';
import { IconSearch, IconArrowUp, IconArrowDown, IconGridView, IconList } from '@douyinfe/semi-icons';
import { useNavigate, useSearchParams } from 'react-router-dom';
import ListingDeletionModal from '../ListingDeletionModal.jsx';
import { xhrDelete, xhrPost } from '../../services/xhr.js';
import { useActions, useSelector } from '../../services/state/store.js';
import { debounce } from '../../utils';
import ListingsGrid from '../grid/listings/ListingsGrid.jsx';
import ListingsTable from '../table/ListingsTable.jsx';
import { IllustrationNoResult, IllustrationNoResultDark } from '@douyinfe/semi-illustrations';
import './ListingsOverview.less';
const ListingsOverview = ({ mode = 'all' }) => {
const isWatchlistMode = mode === 'watchlist';
const listingsData = useSelector((state) => state.listingsData);
const providers = useSelector((state) => state.provider);
const jobs = useSelector((state) => state.jobsData.jobs);
const userSettings = useSelector((state) => state.userSettings.settings);
const actions = useActions();
const navigate = useNavigate();
const sp = useSearchParams();
const viewMode = userSettings?.listings_view_mode ?? 'grid';
const listingDeletionPref = userSettings?.listing_deletion_preference;
const defaultDeleteType = listingDeletionPref?.hardDelete ? 'hard' : 'soft';
const [page, setPage] = useSearchParamState(sp, 'page', 1, parseNumber);
const pageSize = 40;
const [sortField, setSortField] = useSearchParamState(sp, 'sort', 'created_at', parseString);
const [sortDir, setSortDir] = useSearchParamState(sp, 'dir', 'desc', parseString);
const [freeTextFilter, setFreeTextFilter] = useSearchParamState(sp, 'q', null, parseString);
const [watchListFilter, setWatchListFilter] = useSearchParamState(sp, 'watch', null, parseNullableBoolean);
const [jobNameFilter, setJobNameFilter] = useSearchParamState(sp, 'job', null, parseString);
const [activityFilter, setActivityFilter] = useSearchParamState(sp, 'active', null, parseNullableBoolean);
const [providerFilter, setProviderFilter] = useSearchParamState(sp, 'provider', null, parseString);
const [statusFilter, setStatusFilter] = useSearchParamState(sp, 'status', null, parseString);
const [deleteModalVisible, setDeleteModalVisible] = useState(false);
const [listingToDelete, setListingToDelete] = useState(null);
// In watchlist mode the watch filter is forced to "watched only" — regardless of the URL.
const effectiveWatchListFilter = isWatchlistMode ? true : watchListFilter;
const loadData = () => {
actions.listingsData.getListingsData({
page,
pageSize,
sortfield: sortField,
sortdir: sortDir,
freeTextFilter,
filter: {
watchListFilter: effectiveWatchListFilter,
jobNameFilter,
activityFilter,
providerFilter,
statusFilter,
},
});
};
useEffect(() => {
loadData();
}, [
page,
sortField,
sortDir,
freeTextFilter,
providerFilter,
activityFilter,
jobNameFilter,
watchListFilter,
statusFilter,
isWatchlistMode,
]);
const handleFilterChange = useMemo(
() =>
debounce((value) => {
setFreeTextFilter(value || null);
setPage(1);
}, 500),
[],
);
useEffect(() => {
return () => {
handleFilterChange.cancel && handleFilterChange.cancel();
};
}, [handleFilterChange]);
const handleWatch = async (e, item) => {
e.preventDefault();
e.stopPropagation();
try {
await xhrPost('/api/listings/watch', { listingId: item.id });
Toast.success(item.isWatched === 1 ? 'Listing removed from Watchlist' : 'Listing added to Watchlist');
loadData();
} catch (e) {
console.error(e);
Toast.error('Failed to operate Watchlist');
}
};
const handleStatusChange = async (item, nextStatus) => {
try {
await actions.listingsData.setListingStatus(item.id, nextStatus);
Toast.success(nextStatus ? `Marked as ${nextStatus}` : 'Status cleared');
loadData();
} catch (e) {
console.error(e);
Toast.error('Failed to update status');
}
};
const handleDelete = (id) => {
if (listingDeletionPref?.skipPrompt) {
confirmDeletion(listingDeletionPref.hardDelete, false, id);
return;
}
setListingToDelete(id);
setDeleteModalVisible(true);
};
const handleNavigate = (id) => navigate(`/listings/listing/${id}`);
const confirmDeletion = async (hardDelete, remember, id = listingToDelete) => {
try {
if (remember) {
await actions.userSettings.setListingDeletionPreference({ skipPrompt: true, hardDelete });
}
await xhrDelete('/api/listings/', { ids: [id], hardDelete });
Toast.success('Listing successfully removed');
loadData();
} catch (error) {
Toast.error(error.message || 'Error deleting listing');
} finally {
setDeleteModalVisible(false);
setListingToDelete(null);
}
};
const listings = listingsData?.result || [];
return (
<div className="listingsOverview">
<div className="listingsOverview__topbar">
<Input
className="listingsOverview__topbar__search"
prefix={<IconSearch />}
showClear
placeholder="Search"
defaultValue={freeTextFilter ?? ''}
onChange={handleFilterChange}
/>
<RadioGroup
type="button"
buttonSize="middle"
value={activityFilter === null ? 'all' : String(activityFilter)}
onChange={(e) => {
const v = e.target.value;
setActivityFilter(v === 'all' ? null : v === 'true');
setPage(1);
}}
>
<Radio value="all">All</Radio>
<Radio value="true">Active</Radio>
<Radio value="false">Inactive</Radio>
</RadioGroup>
{!isWatchlistMode && (
<RadioGroup
type="button"
buttonSize="middle"
value={watchListFilter === null ? 'all' : String(watchListFilter)}
onChange={(e) => {
const v = e.target.value;
setWatchListFilter(v === 'all' ? null : v === 'true');
setPage(1);
}}
>
<Radio value="all">All</Radio>
<Radio value="true">Watched</Radio>
<Radio value="false">Unwatched</Radio>
</RadioGroup>
)}
<Select
placeholder="Status"
showClear
onChange={(val) => {
setStatusFilter(val ?? null);
setPage(1);
}}
value={statusFilter}
style={{ width: 150 }}
>
<Select.Option value="applied">Applied</Select.Option>
<Select.Option value="rejected">Rejected</Select.Option>
<Select.Option value="accepted">Accepted</Select.Option>
<Select.Option value="none">No status</Select.Option>
</Select>
<Select
placeholder="Provider"
showClear
onChange={(val) => {
setProviderFilter(val);
setPage(1);
}}
value={providerFilter}
style={{ width: 130 }}
>
{providers?.map((p) => (
<Select.Option key={p.id} value={p.id}>
{p.name}
</Select.Option>
))}
</Select>
<Select
placeholder="Job"
showClear
onChange={(val) => {
setJobNameFilter(val);
setPage(1);
}}
value={jobNameFilter}
style={{ width: 130 }}
>
{jobs?.map((j) => (
<Select.Option key={j.id} value={j.id}>
{j.name}
</Select.Option>
))}
</Select>
<Select
prefix="Sort by"
className="listingsOverview__topbar__sort"
style={{ width: 220 }}
value={sortField}
onChange={(val) => setSortField(val)}
>
<Select.Option value="job_name">Job Name</Select.Option>
<Select.Option value="created_at">Listing Date</Select.Option>
<Select.Option value="price">Price</Select.Option>
<Select.Option value="provider">Provider</Select.Option>
</Select>
<Button
icon={sortDir === 'asc' ? <IconArrowUp /> : <IconArrowDown />}
onClick={() => setSortDir(sortDir === 'asc' ? 'desc' : 'asc')}
title={sortDir === 'asc' ? 'Ascending' : 'Descending'}
/>
<div className="listingsOverview__topbar__view-toggle">
<Tooltip content="Grid view">
<Button
icon={<IconGridView />}
theme={viewMode === 'grid' ? 'solid' : 'borderless'}
onClick={() => actions.userSettings.setListingsViewMode('grid')}
aria-label="Grid view"
aria-pressed={viewMode === 'grid'}
/>
</Tooltip>
<Tooltip content="Table view">
<Button
icon={<IconList />}
theme={viewMode === 'table' ? 'solid' : 'borderless'}
onClick={() => actions.userSettings.setListingsViewMode('table')}
aria-label="Table view"
aria-pressed={viewMode === 'table'}
/>
</Tooltip>
</div>
</div>
{listings.length === 0 && (
<Empty
image={<IllustrationNoResult />}
darkModeImage={<IllustrationNoResultDark />}
description="No listings available yet..."
/>
)}
{viewMode === 'grid' ? (
<ListingsGrid
listings={listings}
onWatch={handleWatch}
onNavigate={handleNavigate}
onDelete={handleDelete}
onStatusChange={handleStatusChange}
/>
) : (
<ListingsTable
listings={listings}
onWatch={handleWatch}
onNavigate={handleNavigate}
onDelete={handleDelete}
onStatusChange={handleStatusChange}
/>
)}
{listings.length > 0 && (
<div className="listingsOverview__pagination">
<Pagination
currentPage={page}
pageSize={pageSize}
total={listingsData?.totalNumber || 0}
onPageChange={setPage}
showSizeChanger={false}
/>
</div>
)}
<ListingDeletionModal
visible={deleteModalVisible}
defaultDeleteType={defaultDeleteType}
onConfirm={confirmDeletion}
onCancel={() => {
setDeleteModalVisible(false);
setListingToDelete(null);
}}
/>
</div>
);
};
export default ListingsOverview;

View File

@@ -0,0 +1,53 @@
@import '../../tokens.less';
.listingsOverview {
&__topbar {
display: flex;
align-items: center;
gap: @space-3;
margin-bottom: @space-4;
flex-wrap: wrap;
&__search {
min-width: 200px;
flex: 1;
}
&__view-toggle {
display: flex;
gap: 2px;
flex-shrink: 0;
}
&__sort {
flex-shrink: 0;
.semi-select-prefix {
white-space: nowrap;
}
}
@media (max-width: 768px) {
.listingsOverview__topbar__search {
width: 100%;
flex: unset;
}
.semi-radio-group {
flex: 1;
}
.semi-select {
flex: 1;
min-width: 100px;
width: auto !important;
}
}
}
&__pagination {
margin-top: @space-4;
display: flex;
justify-content: center;
}
}

View File

@@ -0,0 +1,115 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { useState } from 'react';
import { Dropdown, Button, Tooltip } from '@douyinfe/semi-ui-19';
import { IconChevronDown } from '@douyinfe/semi-icons';
import './StatusControl.less';
const STATUS_TOOLTIP =
'Track where you stand with this listing: Applied once you have reached out, Rejected if it did not work out, or Accepted if you got it.';
/**
* @typedef {('applied'|'rejected'|'accepted'|null)} ListingStatus
*/
const STATUS_OPTIONS = [
{ value: null, label: 'None' },
{ value: 'applied', label: 'Applied' },
{ value: 'rejected', label: 'Rejected' },
{ value: 'accepted', label: 'Accepted' },
];
/**
* Look up the option metadata for a status value.
* @param {ListingStatus} status
*/
const optionFor = (status) => STATUS_OPTIONS.find((o) => o.value === status) ?? STATUS_OPTIONS[0];
/**
* Shared control for setting a listing's user-decision status
* (Applied / Rejected / Accepted).
*
* Both compact (table/grid rows) and full (listing detail header) modes
* render a Button that picks up the project's CI tokens via the
* .status-btn classes, with a small size variant for compact contexts.
*
* @param {Object} props
* @param {ListingStatus} props.status - The current status value.
* @param {(next: ListingStatus) => void} props.onChange - Called with the new status when the user picks one.
* @param {boolean} [props.compact=false] - When true, renders smaller for table/grid rows; full size otherwise.
* @param {(e: React.MouseEvent) => void} [props.onTriggerClick] - Optional click handler to stop propagation on the trigger.
*/
export default function StatusControl({ status = null, onChange, compact = false, onTriggerClick }) {
const [open, setOpen] = useState(false);
const [tooltipOpen, setTooltipOpen] = useState(false);
const current = optionFor(status);
const handlePick = (next) => {
setOpen(false);
if (next === status) return;
onChange?.(next);
};
const menu = (
<Dropdown.Menu>
{STATUS_OPTIONS.map((opt) => (
<Dropdown.Item
key={opt.value ?? '__none__'}
active={opt.value === status}
onClick={() => handlePick(opt.value)}
>
{opt.label}
</Dropdown.Item>
))}
</Dropdown.Menu>
);
const className = ['status-btn', compact ? 'status-btn--compact' : null, status ? `status-btn--${status}` : null]
.filter(Boolean)
.join(' ');
const trigger = (
<Tooltip
content={STATUS_TOOLTIP}
position="top"
trigger="custom"
visible={tooltipOpen && !open}
onVisibleChange={setTooltipOpen}
>
<Button
size={compact ? 'small' : 'default'}
theme="borderless"
icon={<IconChevronDown />}
iconPosition="right"
onMouseEnter={() => setTooltipOpen(true)}
onMouseLeave={() => setTooltipOpen(false)}
onClick={(e) => {
onTriggerClick?.(e);
setTooltipOpen(false);
setOpen((o) => !o);
}}
className={className}
>
{status ? current.label : 'Status'}
</Button>
</Tooltip>
);
return (
<Dropdown
trigger="custom"
visible={open}
onVisibleChange={setOpen}
onClickOutSide={() => setOpen(false)}
position="bottom"
render={menu}
stopPropagation
>
<span className="status-btn__anchor">{trigger}</span>
</Dropdown>
);
}

View File

@@ -0,0 +1,64 @@
@import '../../tokens.less';
// Wrapper span used as the Dropdown's positioning anchor so the menu opens
// directly below the visible button rather than the implicit wrapper of the
// hover tooltip (which can have a different bounding box).
.status-btn__anchor {
display: inline-block;
line-height: 0;
}
// StatusControl shared base. Matches dimensions and border treatment
// of the surrounding Watched / Open listing / Delete buttons in the
// detail view, and shrinks via the --compact modifier for table rows
// and grid cards.
.status-btn {
color: @color-muted !important;
border: 1px solid @color-border-bright !important;
border-radius: @radius-btn !important;
background: transparent !important;
transition: color @transition-fast, border-color @transition-fast, background @transition-fast;
&:hover {
color: @color-text !important;
background: rgba(255, 255, 255, 0.06) !important;
}
&--compact {
height: 24px !important;
padding: 0 8px !important;
font-size: @text-sm !important;
border-radius: @radius-chip !important;
.semi-icon {
font-size: 12px !important;
}
}
&--applied {
color: @color-info !important;
border-color: rgba(96, 165, 250, 0.4) !important;
background: rgba(96, 165, 250, 0.08) !important;
&:hover {
background: rgba(96, 165, 250, 0.14) !important;
}
}
&--rejected {
color: @color-error !important;
border-color: rgba(251, 113, 133, 0.4) !important;
background: rgba(251, 113, 133, 0.08) !important;
&:hover {
background: rgba(251, 113, 133, 0.14) !important;
}
}
&--accepted {
color: @color-success !important;
border-color: rgba(52, 211, 153, 0.4) !important;
background: rgba(52, 211, 153, 0.08) !important;
&:hover {
background: rgba(52, 211, 153, 0.14) !important;
}
}
}

View File

@@ -8,6 +8,7 @@ import maplibregl from 'maplibre-gl';
import 'maplibre-gl/dist/maplibre-gl.css';
import '@mapbox/mapbox-gl-draw/dist/mapbox-gl-draw.css';
import { fixMapboxDrawCompatibility, addDrawingControl, setupAreaFilterEventListeners } from './MapDrawingExtension.js';
import { getBoundsFromCoords } from '../../views/listings/mapUtils.js';
import './Map.less';
export const GERMANY_BOUNDS = [
@@ -66,6 +67,7 @@ export default function Map({
const mapContainerRef = useRef(null);
const mapRef = useRef(null);
const drawRef = useRef(null);
const hasFittedToInitialAreaRef = useRef(false);
// Initialize map - ONLY when container changes, never reinitialize
useEffect(() => {
@@ -128,6 +130,17 @@ export default function Map({
} catch (error) {
console.error('Error loading spatial filter:', error);
}
if (!hasFittedToInitialAreaRef.current) {
const coords = initialSpatialFilter.features.flatMap((feature) =>
feature.geometry?.type === 'Polygon' ? feature.geometry.coordinates.flat() : [],
);
const bounds = getBoundsFromCoords(coords);
if (bounds) {
mapRef.current.fitBounds(bounds, { padding: 50, maxZoom: 15, duration: 0 });
hasFittedToInitialAreaRef.current = true;
}
}
}
// Setup drawing event listeners

View File

@@ -136,9 +136,9 @@
color: @color-muted !important;
}
// Collapsed state icons perfectly centered
// Collapsed state - icons perfectly centered
.semi-navigation-collapsed {
// Text span is display:block and takes up flex space must be removed so justify-content:center works
// Text span is display:block and takes up flex space - must be removed so justify-content:center works
.semi-navigation-item-text {
display: none !important;
}
@@ -165,7 +165,7 @@
min-width: 0 !important;
}
// Semi adds margin-right to icons for text spacing remove it when collapsed
// Semi adds margin-right to icons for text spacing - remove it when collapsed
.semi-navigation-item-icon,
.semi-navigation-sub-title-icon {
display: flex !important;
@@ -179,13 +179,13 @@
}
}
// Semi Nav.Footer full width, no extra padding (our BEM class controls it)
// Semi Nav.Footer - full width, no extra padding (our BEM class controls it)
.semi-navigation-footer {
width: 100% !important;
box-sizing: border-box !important;
}
// Collapsed submenu popup actual class used by Semi UI is .semi-navigation-popover
// Collapsed submenu popup - actual class used by Semi UI is .semi-navigation-popover
.semi-navigation-popover {
background: @color-elevated !important;
border: 1px solid @color-border !important;

View File

@@ -37,6 +37,7 @@ export default function Navigation({ isAdmin }) {
items: [
{ itemKey: '/listings', text: 'Overview' },
{ itemKey: '/map', text: 'Map View' },
{ itemKey: '/listings/watchlist', text: 'Watchlist' },
],
},
];
@@ -61,6 +62,22 @@ export default function Navigation({ isAdmin }) {
}
function parsePathName(name) {
// Collect every leaf itemKey that looks like a route (starts with '/').
// Prefer the longest exact-prefix match so nested routes like
// '/listings/watchlist' resolve to themselves instead of being collapsed
// to '/listings'.
const allKeys = [];
const collect = (nodes) => {
for (const n of nodes) {
if (typeof n.itemKey === 'string' && n.itemKey.startsWith('/')) allKeys.push(n.itemKey);
if (Array.isArray(n.items)) collect(n.items);
}
};
collect(items);
const longestMatch = allKeys
.filter((k) => name === k || name.startsWith(k + '/'))
.sort((a, b) => b.length - a.length)[0];
if (longestMatch) return longestMatch;
const split = name.split('/').filter((s) => s.length !== 0);
return '/' + split[0];
}

View File

@@ -32,7 +32,7 @@
padding: 16px 20px !important;
}
// Semi input focus subtle, not accent
// Semi input focus - subtle, not accent
.semi-input-wrapper:focus-within,
.semi-select:focus-within {
border-color: @color-border-bright !important;

View File

@@ -0,0 +1,128 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { Button, Tag, Tooltip, Switch } from '@douyinfe/semi-ui-19';
import {
IconAlertTriangle,
IconBell,
IconBriefcase,
IconCopy,
IconDelete,
IconDescend2,
IconEdit,
IconHome,
IconPlayCircle,
} from '@douyinfe/semi-icons';
import './JobsTable.less';
/**
* @param {{ jobs: object[], onRun: Function, onEdit: Function, onClone: Function, onDeleteListings: Function, onDeleteJob: Function, onStatusChange: Function }} props
*/
const JobsTable = ({ jobs, onRun, onEdit, onClone, onDeleteListings, onDeleteJob, onStatusChange }) => (
<div className="jobsTable">
{jobs.map((job) => (
<div key={job.id} className={`jobsTable__row${!job.enabled ? ' jobsTable__row--inactive' : ''}`}>
<div className="jobsTable__row__dot">
<span
className={`jobsTable__row__dot__indicator${job.enabled ? ' jobsTable__row__dot__indicator--active' : ''}`}
/>
</div>
<div className="jobsTable__row__name" title={job.name}>
{job.name}
</div>
<div className="jobsTable__row__stat jobsTable__row__stat--blue">
<IconHome size="small" />
{job.numberOfFoundListings || 0}
</div>
<div className="jobsTable__row__stat jobsTable__row__stat--orange">
<IconBriefcase size="small" />
{job.provider?.length || 0}
</div>
<div className="jobsTable__row__stat jobsTable__row__stat--purple">
<IconBell size="small" />
{job.notificationAdapter?.length || 0}
</div>
<div className="jobsTable__row__badges">
<Switch
size="small"
checked={job.enabled}
disabled={job.isOnlyShared}
onChange={(checked) => onStatusChange(job.id, checked)}
/>
{job.running && (
<Tag color="green" variant="light" size="small">
RUNNING
</Tag>
)}
{job.isOnlyShared && (
<Tooltip content="Shared with you - read only">
<span style={{ display: 'flex', alignItems: 'center' }}>
<IconAlertTriangle style={{ color: 'rgba(var(--semi-yellow-7), 1)' }} />
</span>
</Tooltip>
)}
</div>
<div className="jobsTable__row__actions">
<Tooltip content="Run Job">
<Button
type="primary"
style={{ background: '#21aa21b5' }}
size="small"
theme="solid"
icon={<IconPlayCircle />}
disabled={job.isOnlyShared || job.running}
onClick={() => onRun(job.id)}
/>
</Tooltip>
<Tooltip content="Edit Job">
<Button
type="secondary"
size="small"
icon={<IconEdit />}
disabled={job.isOnlyShared}
onClick={() => onEdit(job.id)}
/>
</Tooltip>
<Tooltip content="Clone Job">
<Button
type="tertiary"
size="small"
icon={<IconCopy />}
disabled={job.isOnlyShared}
onClick={() => onClone(job.id)}
/>
</Tooltip>
<Tooltip content="Delete all found Listings">
<Button
type="danger"
size="small"
icon={<IconDescend2 />}
disabled={job.isOnlyShared}
onClick={() => onDeleteListings(job.id)}
/>
</Tooltip>
<Tooltip content="Delete Job">
<Button
type="danger"
size="small"
icon={<IconDelete />}
disabled={job.isOnlyShared}
onClick={() => onDeleteJob(job.id)}
/>
</Tooltip>
</div>
</div>
))}
</div>
);
export default JobsTable;

View File

@@ -0,0 +1,105 @@
@import '../../tokens.less';
.jobsTable {
display: flex;
flex-direction: column;
gap: 4px;
&__row {
display: grid;
grid-template-columns: 24px 1fr 80px 80px 80px auto auto;
align-items: center;
gap: @space-3;
padding: 8px 12px;
background: @color-elevated;
border: 1px solid @color-border;
border-radius: @radius-chip;
transition: background @transition-fast;
&:hover {
background: #252525;
}
&--inactive {
opacity: 0.6;
}
&__dot {
display: flex;
align-items: center;
justify-content: center;
&__indicator {
width: 8px;
height: 8px;
border-radius: 50%;
flex-shrink: 0;
background-color: rgba(251, 113, 133, 0.7);
&--active {
background-color: rgba(52, 211, 153, 0.8);
}
}
}
&__name {
font-weight: 600;
font-size: @text-sm;
color: @color-text;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
&__stat {
font-size: @text-sm;
font-weight: 600;
display: flex;
align-items: center;
gap: 4px;
white-space: nowrap;
&--blue {
color: @color-blue-text;
}
&--orange {
color: @color-orange-text;
}
&--purple {
color: @color-purple-text;
}
}
&__badges {
display: flex;
align-items: center;
gap: 4px;
min-width: 0;
}
&__actions {
display: flex;
align-items: center;
gap: 2px;
}
@media (max-width: 900px) {
grid-template-columns: 24px 1fr 80px auto auto;
.jobsTable__row__stat--orange,
.jobsTable__row__stat--purple {
display: none;
}
}
@media (max-width: 560px) {
grid-template-columns: 24px 1fr auto auto;
.jobsTable__row__stat--blue {
display: none;
}
}
}
}

Some files were not shown because too many files have changed in this diff Show More