Compare commits

..

38 Commits

Author SHA1 Message Date
orangecoding
19d4721f9f improve welcome screen 2026-02-17 14:03:15 +01:00
orangecoding
a794645393 fixing login route 2026-02-17 12:50:21 +01:00
orangecoding
fd7e228972 adding welcome screen 2026-02-17 12:35:39 +01:00
orangecoding
b86e351007 fixing lint even harder 2026-02-16 13:50:50 +01:00
orangecoding
19c4860da7 fixing eslint harder 2026-02-16 12:59:34 +01:00
orangecoding
d98e06cfdf fixing eslint 2026-02-16 12:40:41 +01:00
orangecoding
6ae0c9749b update dependencies 2026-02-16 12:30:59 +01:00
orangecoding
10e40e038e adding check if fredy is running in docker 2026-02-16 12:29:02 +01:00
orangecoding
4ba6828939 adding release tool 2026-02-05 12:02:18 +01:00
orangecoding
d09770dae2 fancy, almost impossible to see animation on dashboard 2026-02-05 09:54:42 +01:00
orangecoding
248e4d2562 improve tracking 2026-02-04 14:41:55 +01:00
orangecoding
7b8e961b49 adding confirmation dialog if to remove listing entirely from db or just hide it 2026-02-03 14:04:40 +01:00
orangecoding
f66ceccbb4 next release version 2026-01-29 13:01:39 +01:00
orangecoding
a3db725af6 fixing image rendering 2026-01-29 13:01:07 +01:00
orangecoding
0663bd945f smaller demo improvements 2026-01-29 09:46:23 +01:00
orangecoding
bc355fb5fe fixing some bugs the wife found ;) 2026-01-28 21:25:48 +01:00
orangecoding
797421f0d5 hardening demo handling 2026-01-28 16:29:59 +01:00
orangecoding
0b2b42fc75 improve geocoding 2026-01-28 15:55:23 +01:00
Christian Kellner
472169693f Improvements 01 28 (#264)
* improving footer

* improve ui

* upgrading dependencies

* adding glow to all boxes on dashboard

* introducing single listing view

* next release version

* improve screenshots and login page
2026-01-28 14:27:03 +01:00
orangecoding
3117044139 fixing immoscout scraper 2026-01-26 19:52:37 +01:00
orangecoding
7879d0e94a next release version 2026-01-26 12:35:57 +01:00
orangecoding
afd1048c9e hardening the check if a listing is active 2026-01-26 12:34:49 +01:00
orangecoding
acbaab05ed next release version 2026-01-26 12:07:43 +01:00
orangecoding
72fffc526b deleting a listing now sets it to deleted in the db, preventing it from reappearing when scraping happens 2026-01-26 12:07:21 +01:00
orangecoding
9e5989ece3 zoom into map where most markers are 2026-01-26 11:54:47 +01:00
orangecoding
afc200c9e1 improved tooltip in map, improved user-settings handling 2026-01-26 11:50:16 +01:00
orangecoding
59226491f2 improved tooltip in map, improved user-settings handling 2026-01-26 11:20:02 +01:00
orangecoding
28f7760120 adapt link to listing in grid view to behave like a real link 2026-01-26 10:43:38 +01:00
orangecoding
2465514b7a fixing immoscout translator, allowing balcony and garden for purchases 2026-01-26 10:20:21 +01:00
Christian Kellner
9dde377fe6 possibility to display distance (#262) 2026-01-25 13:52:56 +01:00
Katrin Leinweber
28a3a7f372 Use EUR-symbol to match Map.jsx (see d43c5b3) (#261)
Co-authored-by: Katrin Leinweber <katrinleinweber@noreply.github.com>
2026-01-25 12:32:11 +01:00
orangecoding
e859250545 next release version 2026-01-22 15:10:31 +00:00
Christian Kellner
4dd0370ec1 Calculating the distance (#255)
* migra for distance

* adding distance calculator

* adding ability to store home address

* improve distance calculation

* calculating distance

* show distance in grid view

* upgrading dependencies

* moving to react 19

* ability to clone a job

* fixing tests

* polishing
2026-01-22 16:09:36 +01:00
orangecoding
51b4e51f3f fixing setting kleinanzeigen listings to inactive if not available anymore 2026-01-16 11:36:51 +01:00
orangecoding
fa1899765c fixing some rendering issues in map 2026-01-16 10:46:50 +01:00
Christian Kellner
d43c5b3f97 Map View in Fredy :D (#253)
* init map view

* switching off 3d buildings when sattelite view is on

* rename menu items

* upgrading dependencies, adding provider to popups

* adding screenshot for map view

* fixing readme

* next release version
2026-01-12 15:00:36 +01:00
orangecoding
7fd8be07a2 adding wohnungsboerse provider 2026-01-09 11:37:03 +01:00
orangecoding
2926ee7e08 upgrading dependencies 2026-01-06 09:51:04 +01:00
206 changed files with 19705 additions and 2674 deletions

1
.gitignore vendored
View File

@@ -6,3 +6,4 @@ npm-debug.log
.DS_Store
.idea
.vscode
tools/release/config.json

View File

@@ -1,94 +0,0 @@
Newer release changelog see https://github.com/orangecoding/fredy/releases
---
###### [V5.5.0]
- Upgrading dependencies
- fixing provider
- allow multiple instances of 1 provider
- **BREAKING**: Minimum node version is now 16
###### [V5.4.6]
- Adding Instana node.js monitoring
-
###### [V5.4.5]
- Adding Instana node.js monitoring
###### [V5.4.4]
- Add support for Immo Südwest Presse (immo.swp.de)
- Telegram: Use job name instead of ID and link in title
- Fix race condition if user ID is in session but not in user store
- Allow visiting the original provider URL
###### [V5.4.3]
- re-writing readme
- improving docker build
- using github's actions to build docker and test automatically
###### [V5.4.2]
- Fixing prod build
###### [V5.4.1]
- Upgrading dependencies
- Provider urls are now automagically been changed to include the correct sort order for search results
```
Note: It has been an point of confusion since the very beginning of Fredy, that people simply copied the url, but
did not take care of sorting the search results by date. If this is not done, Fredy will most likely not see the latest
results, thus cannot report them. This release fixes it by adding the necessary params (or replaces them).
```
###### [V5.3.0]
- Upgrading dependencies
- It's now possible to send mails to multiple receiver using comma separation for MailJet & Sendgrid
- Fixing Immowelt scraping
###### [V5.2.0]
- Upgrading dependencies
- Adding new similarity check layer (Duplicates are being removed now)
- Adding paging for search results
###### [V5.1.0]
- Upgrading dependencies
- NodeJS 12.13 is now the minimum supported version
- Adding general settings as new configuration page to ui
- Adding new feature working hours
###### [V5.0.0]
- Upgrading dependencies
- NodeJS 12 is now the minimum supported version
###### [V4.0.0]
Bringing back Immoscout :tada:
###### [V3.0.0]
This is basically a re-write, your old config file will not be compatible anymore. Please re-created your search jobs
on the new ui and use the values from your previous config file if needed.
```
- We're getting rid of manual config changes, Fredy, now ships with a UI so that it's easy for you to create and edit jobs
```
###### [V2.0.0]
```
- Fredy can now run multiple search job on one instance
- Changed lot's of the structure of Fredy to make this happen
[BREAKING CHANGES]
- The config has been changed, the config of V1.x will not work any longer
- Sources have been renamed to provider
```

View File

@@ -35,6 +35,7 @@ WORKDIR /fredy
RUN apk add --no-cache chromium curl
ENV NODE_ENV=production \
IS_DOCKER=true \
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser

View File

@@ -119,7 +119,7 @@ Should you use [Unraid](https://unraid.net/), you can now install Fredy from the
## 📸 Screenshots
| Fredy Main Overview | Job Configuration | Found Listings |
| Fredy Maps View | Dashboard | Found Listings |
|--------------------------------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------------------------|
| ![Screenshot showing Fredy](doc/screenshot1.png) | ![Screenshot showing job configuration in Fredy](doc/screenshot3.png) | ![Screenshot showing found listings in Fredy](doc/screenshot2.png) |

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -30,12 +30,16 @@ async function getAllFiles(dir = '.') {
/* eslint-disable no-console */
async function addCopyright(files) {
const oldCopyrightRegex =
/^(\/\*\n \* Copyright \(c\) \d{4} by Christian Kellner\.\n \* Licensed under Apache-2.0 with Commons Clause and Attribution\/Naming Clause\n \*\/\n\n)+/;
for (let file of files) {
try {
let content = await fs.readFile(file, 'utf8');
if (!content.startsWith(COPYRIGHT)) {
await fs.writeFile(file, COPYRIGHT + content);
console.log(`Added copyright to ${file}`);
const strippedContent = content.replace(oldCopyrightRegex, '');
const newContent = COPYRIGHT + strippedContent;
if (content !== newContent) {
await fs.writeFile(file, newContent);
console.log(`Added/Updated copyright in ${file}`);
}
} catch (err) {
console.error(`Error processing ${file}: ${err}`);

Binary file not shown.

Before

Width:  |  Height:  |  Size: 248 KiB

After

Width:  |  Height:  |  Size: 3.7 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.7 MiB

After

Width:  |  Height:  |  Size: 4.8 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 402 KiB

After

Width:  |  Height:  |  Size: 531 KiB

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -8,20 +8,20 @@ import js from '@eslint/js';
import prettier from 'eslint-config-prettier';
import globals from 'globals';
import react from 'eslint-plugin-react';
import babelParser from '@babel/eslint-parser';
export default [
{
files: ['**/*.{js,jsx,ts,tsx}'],
ignores: ['**/node_modules/**', '**/dist/**', '**/build/**', '**/public/**', 'db/**', 'conf/**'],
},
js.configs.recommended,
prettier,
{
files: ['**/*.{js,jsx}'],
languageOptions: {
parser: babelParser,
ecmaVersion: 'latest',
sourceType: 'module',
ecmaVersion: 2021,
parserOptions: {
ecmaFeatures: { jsx: true },
},
globals: {
...globals.browser,
...globals.node,
@@ -32,70 +32,14 @@ export default [
after: 'readonly',
it: 'readonly',
},
parserOptions: { requireConfigFile: false },
},
plugins: { react },
rules: {
eqeqeq: [2, 'allow-null'],
strict: 0,
'no-redeclare': [2, { builtinGlobals: false }],
'class-methods-use-this': 'off',
indent: ['off', 2],
'linebreak-style': ['error', 'unix'],
quotes: ['error', 'single', { avoidEscape: true, allowTemplateLiterals: true }],
semi: ['error', 'always'],
'no-console': ['error', { allow: ['warn', 'error'] }],
'jsx-quotes': ['error', 'prefer-double'],
'react/display-name': 'off',
'react/forbid-prop-types': 'off',
'react/jsx-closing-bracket-location': 'off',
'react/jsx-curly-spacing': 'off',
'react/jsx-handler-names': ['off', { eventHandlerPrefix: 'handle', eventHandlerPropPrefix: 'on' }],
'react/jsx-indent-props': 'off',
'react/jsx-key': 'off',
'react/jsx-max-props-per-line': 'off',
'react/jsx-no-bind': ['error', { ignoreRefs: true, allowArrowFunctions: true, allowBind: false }],
'react/jsx-no-duplicate-props': ['error', { ignoreCase: true }],
'react/jsx-no-literals': 'off',
'react/jsx-no-undef': 'error',
'react/jsx-pascal-case': ['error', { allowAllCaps: true, ignore: [] }],
'react/sort-prop-types': ['off', { ignoreCase: true, callbacksLast: false, requiredFirst: false }],
'react/jsx-sort-prop-types': 'off',
'react/jsx-sort-props': 'off',
'react/jsx-uses-react': 'error',
'react/jsx-uses-vars': 'error',
'react/no-danger': 'warn',
'react/no-deprecated': 'error',
'react/no-did-mount-set-state': 'error',
'react/no-did-update-set-state': 'warn',
'react/no-direct-mutation-state': 'off',
'react/no-is-mounted': 'error',
'react/no-set-state': 'off',
'react/no-string-refs': 'warn',
'react/no-unknown-property': 'error',
'react/prop-types': ['error', { ignore: [], customValidators: [], skipUndeclared: true }],
'react/react-in-jsx-scope': 'error',
'react/require-extension': 'off',
'react/require-render-return': 'error',
'react/self-closing-comp': 'warn',
'react/sort-comp': 'off',
'react/jsx-wrap-multilines': ['warn', { declaration: true, assignment: true, return: true }],
'react/wrap-multilines': 'off',
'react/jsx-first-prop-new-line': 'off',
'react/jsx-equals-spacing': ['warn', 'never'],
'react/jsx-no-target-blank': 'error',
'react/jsx-filename-extension': ['error', { extensions: ['.jsx'] }],
'react/jsx-no-comment-textnodes': 'error',
'react/no-comment-textnodes': 'off',
'react/no-render-return-value': 'error',
'react/require-optimization': ['off', { allowDecorators: [] }],
'react/no-find-dom-node': 'warn',
'react/forbid-component-props': ['off', { forbid: [] }],
'react/no-danger-with-children': 'error',
'react/no-unused-prop-types': ['warn', { customValidators: [], skipShapeProps: true }],
'react/style-prop-object': 'error',
'react/no-children-prop': 'warn',
},
settings: { react: { version: 'detect' } },
rules: {
...js.configs.recommended.rules,
'no-console': ['error', { allow: ['warn', 'error'] }],
},
},
prettier,
];

View File

@@ -7,7 +7,7 @@
content="user-scalable=no, width=device-width, initial-scale=1, maximum-scale=1"
/>
<meta name="google" content="notranslate" />
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<title>Fredy || Real Estate Finder</title>

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -8,10 +8,10 @@ import { checkIfConfigIsAccessible, getProviders, refreshConfig } from './lib/ut
import * as similarityCache from './lib/services/similarity-check/similarityCache.js';
import { runMigrations } from './lib/services/storage/migrations/migrate.js';
import { ensureDemoUserExists, ensureAdminUserExists } from './lib/services/storage/userStorage.js';
import { cleanupDemoAtMidnight } from './lib/services/crons/demoCleanup-cron.js';
import { initTrackerCron } from './lib/services/crons/tracker-cron.js';
import logger from './lib/services/logger.js';
import { initActiveCheckerCron } from './lib/services/crons/listing-alive-cron.js';
import { initGeocodingCron } from './lib/services/crons/geocoding-cron.js';
import { getSettings } from './lib/services/storage/settingsStorage.js';
import SqliteConnection, { computeDbPath } from './lib/services/storage/SqliteConnection.js';
import { initJobExecutionService } from './lib/services/jobs/jobExecutionService.js';
@@ -53,7 +53,6 @@ await import('./lib/api/api.js');
if (settings.demoMode) {
logger.info('Running in demo mode');
cleanupDemoAtMidnight();
}
ensureAdminUserExists();
@@ -61,6 +60,7 @@ ensureDemoUserExists();
await initTrackerCron();
//do not wait for this to finish, let it run in the background
initActiveCheckerCron();
initGeocodingCron();
logger.info(`Started Fredy successfully. Ui can be accessed via http://localhost:${settings.port}`);

View File

@@ -1,14 +1,19 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { NoNewListingsWarning } from './errors.js';
import { storeListings, getKnownListingHashesForJobAndProvider } from './services/storage/listingsStorage.js';
import { getJob } from './services/storage/jobStorage.js';
import * as notify from './notification/notify.js';
import Extractor from './services/extractor/extractor.js';
import urlModifier from './services/queryStringMutator.js';
import logger from './services/logger.js';
import { geocodeAddress } from './services/geocoding/geoCodingService.js';
import { distanceMeters } from './services/listings/distanceCalculator.js';
import { getUserSettings } from './services/storage/settingsStorage.js';
import { updateListingDistance } from './services/storage/listingsStorage.js';
/**
* @typedef {Object} Listing
@@ -79,12 +84,33 @@ class FredyPipelineExecutioner {
.then(this._normalize.bind(this))
.then(this._filter.bind(this))
.then(this._findNew.bind(this))
.then(this._geocode.bind(this))
.then(this._save.bind(this))
.then(this._calculateDistance.bind(this))
.then(this._filterBySimilarListings.bind(this))
.then(this._notify.bind(this))
.catch(this._handleError.bind(this));
}
/**
* Geocode new listings.
*
* @param {Listing[]} newListings New listings to geocode.
* @returns {Promise<Listing[]>} Resolves with the listings (potentially with added coordinates).
*/
async _geocode(newListings) {
for (const listing of newListings) {
if (listing.address) {
const coords = await geocodeAddress(listing.address);
if (coords) {
listing.latitude = coords.lat;
listing.longitude = coords.lng;
}
}
}
return newListings;
}
/**
* Fetch listings from the provider, using the default Extractor flow unless
* a provider-specific getListings override is supplied.
@@ -180,6 +206,42 @@ class FredyPipelineExecutioner {
return newListings;
}
/**
* Calculate distance for new listings.
*
* @param {Listing[]} listings
* @returns {Listing[]}
* @private
*/
_calculateDistance(listings) {
if (listings.length === 0) return [];
const job = getJob(this._jobKey);
const userId = job?.userId;
if (userId == null || typeof userId !== 'string') {
logger.debug('Skipping distance calculation: userId is missing or invalid');
return listings;
}
const userSettings = getUserSettings(userId);
const homeAddress = userSettings?.home_address;
if (!homeAddress || !homeAddress.coords) {
return listings;
}
const { lat, lng } = homeAddress.coords;
for (const listing of listings) {
if (listing.latitude != null && listing.longitude != null) {
const dist = distanceMeters(lat, lng, listing.latitude, listing.longitude);
updateListingDistance(listing.id, dist);
listing.distance_to_destination = dist;
}
}
return listings;
}
/**
* Remove listings that are similar to already known entries according to the similarity cache.
* Adds the remaining listings to the cache.

10
lib/TRACKING_POIS.js Normal file
View File

@@ -0,0 +1,10 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
export const TRACKING_POIS = {
DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
WELCOME_FINISHED: 'WELCOME_FINISHED',
WELCOME_SKIPPED: 'WELCOME_SKIPPED',
};

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -10,6 +10,7 @@ import { providerRouter } from './routes/providerRouter.js';
import { versionRouter } from './routes/versionRouter.js';
import { loginRouter } from './routes/loginRoute.js';
import { userRouter } from './routes/userRoute.js';
import { userSettingsRouter } from './routes/userSettingsRoute.js';
import { jobRouter } from './routes/jobRouter.js';
import bodyParser from 'body-parser';
import restana from 'restana';
@@ -20,9 +21,9 @@ import { demoRouter } from './routes/demoRouter.js';
import logger from '../services/logger.js';
import { listingsRouter } from './routes/listingsRouter.js';
import { getSettings } from '../services/storage/settingsStorage.js';
import { featureRouter } from './routes/featureRouter.js';
import { dashboardRouter } from './routes/dashboardRouter.js';
import { backupRouter } from './routes/backupRouter.js';
import { trackingRouter } from './routes/trackingRoute.js';
const service = restana();
const staticService = files(path.join(getDirName(), '../ui/public'));
const PORT = (await getSettings()).port || 9998;
@@ -35,7 +36,8 @@ service.use('/api/jobs', authInterceptor());
service.use('/api/version', authInterceptor());
service.use('/api/listings', authInterceptor());
service.use('/api/dashboard', authInterceptor());
service.use('/api/features', authInterceptor());
service.use('/api/user/settings', authInterceptor());
service.use('/api/tracking', authInterceptor());
// /admin can only be accessed when user is having admin permissions
service.use('/api/admin', adminInterceptor());
@@ -44,12 +46,13 @@ service.use('/api/admin/generalSettings', generalSettingsRouter);
service.use('/api/admin/backup', backupRouter);
service.use('/api/jobs/provider', providerRouter);
service.use('/api/admin/users', userRouter);
service.use('/api/user/settings', userSettingsRouter);
service.use('/api/version', versionRouter);
service.use('/api/jobs', jobRouter);
service.use('/api/login', loginRouter);
service.use('/api/listings', listingsRouter);
service.use('/api/features', featureRouter);
service.use('/api/dashboard', dashboardRouter);
service.use('/api/tracking', trackingRouter);
//this route is unsecured intentionally as it is being queried from the login page
service.use('/api/demo', demoRouter);

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,17 +0,0 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import getFeatures from '../../features.js';
const service = restana();
const featureRouter = service.newRouter();
featureRouter.get('/', async (req, res) => {
const features = getFeatures();
res.body = Object.assign({}, { features });
res.send();
});
export { featureRouter };

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -11,10 +11,13 @@ import logger from '../../services/logger.js';
import { bus } from '../../services/events/event-bus.js';
import { isRunning as isJobRunning } from '../../services/jobs/run-state.js';
import { addClient as addSseClient, removeClient } from '../../services/sse/sse-broker.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
const service = restana();
const jobRouter = service.newRouter();
const DEMO_JOB_NAME = 'Demo-Job';
function doesJobBelongsToUser(job, req) {
const userId = req.session.currentUser;
if (userId == null) {
@@ -161,6 +164,7 @@ jobRouter.post('/:jobId/run', async (req, res) => {
jobRouter.post('/', async (req, res) => {
const { provider, notificationAdapter, name, blacklist = [], jobId, enabled, shareWithUsers = [] } = req.body;
const settings = await getSettings();
try {
let jobFromDb = jobStorage.getJob(jobId);
@@ -169,6 +173,11 @@ jobRouter.post('/', async (req, res) => {
return;
}
if (settings.demoMode && jobFromDb && jobFromDb.name === DEMO_JOB_NAME) {
res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
return;
}
jobStorage.upsertJob({
userId: req.session.currentUser,
jobId,
@@ -188,8 +197,14 @@ jobRouter.post('/', async (req, res) => {
jobRouter.delete('', async (req, res) => {
const { jobId } = req.body;
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (settings.demoMode && job.name === DEMO_JOB_NAME) {
res.send(new Error('Sorry, but you cannot remove the Demo Job ;)'));
return;
}
if (!doesJobBelongsToUser(job, req)) {
res.send(new Error('You are trying to remove a job that is not associated to your user'));
} else {
@@ -204,8 +219,15 @@ jobRouter.delete('', async (req, res) => {
jobRouter.put('/:jobId/status', async (req, res) => {
const { status } = req.body;
const { jobId } = req.params;
const settings = await getSettings();
try {
const job = jobStorage.getJob(jobId);
if (settings.demoMode && job.name === DEMO_JOB_NAME) {
res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
return;
}
if (!doesJobBelongsToUser(job, req)) {
res.send(new Error('You are trying change a job that is not associated to your user'));
} else {

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -10,6 +10,7 @@ import { isAdmin as isAdminFn } from '../security.js';
import logger from '../../services/logger.js';
import { nullOrEmpty } from '../../utils.js';
import { getJobs } from '../../services/storage/jobStorage.js';
import { getSettings } from '../../services/storage/settingsStorage.js';
const service = restana();
@@ -63,6 +64,29 @@ listingsRouter.get('/table', async (req, res) => {
res.send();
});
listingsRouter.get('/map', async (req, res) => {
const { jobId } = req.query || {};
res.body = listingStorage.getListingsForMap({
jobId: nullOrEmpty(jobId) ? null : jobId,
userId: req.session.currentUser,
isAdmin: isAdminFn(req),
});
res.send();
});
listingsRouter.get('/:listingId', async (req, res) => {
const { listingId } = req.params;
const listing = listingStorage.getListingById(listingId, req.session.currentUser, isAdminFn(req));
if (!listing) {
res.statusCode = 404;
res.body = { message: 'Listing not found' };
return res.send();
}
res.body = listing;
res.send();
});
// Toggle watch state for the current user on a listing
listingsRouter.post('/watch', async (req, res) => {
try {
@@ -83,9 +107,15 @@ listingsRouter.post('/watch', async (req, res) => {
});
listingsRouter.delete('/job', async (req, res) => {
const { jobId } = req.body;
const { jobId, hardDelete = false } = req.body;
const settings = await getSettings();
try {
listingStorage.deleteListingsByJobId(jobId);
if (settings.demoMode) {
res.send(new Error('Sorry, but you cannot remove listings in demo mode ;)'));
return;
}
listingStorage.deleteListingsByJobId(jobId, hardDelete);
} catch (error) {
res.send(new Error(error));
logger.error(error);
@@ -94,10 +124,10 @@ listingsRouter.delete('/job', async (req, res) => {
});
listingsRouter.delete('/', async (req, res) => {
const { ids } = req.body;
const { ids, hardDelete = false } = req.body;
try {
if (Array.isArray(ids) && ids.length > 0) {
listingStorage.deleteListingsById(ids);
listingStorage.deleteListingsById(ids, hardDelete);
}
} catch (error) {
res.send(new Error(error));

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -0,0 +1,37 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
import logger from '../../services/logger.js';
const service = restana();
const trackingRouter = service.newRouter();
trackingRouter.get('/trackingPois', async (req, res) => {
res.body = TRACKING_POIS;
res.send();
});
trackingRouter.post('/poi', async (req, res) => {
const { poi } = req.body;
if (!poi) {
res.statusCode = 400;
res.send({ error: 'Feature name is required' });
return;
}
try {
await trackPoi(poi);
res.send({ success: true });
} catch (error) {
logger.error('Error tracking feature', error);
res.statusCode = 500;
res.send({ error: error.message });
}
});
export { trackingRouter };

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -0,0 +1,100 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import restana from 'restana';
import SqliteConnection from '../../services/storage/SqliteConnection.js';
import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
import { resetGeocoordinatesAndDistanceForUser } from '../../services/storage/listingsStorage.js';
import { geocodeAddress } from '../../services/geocoding/geoCodingService.js';
import { autocompleteAddress } from '../../services/geocoding/autocompleteService.js';
import { fromJson } from '../../utils.js';
import { trackPoi } from '../../services/tracking/Tracker.js';
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
import logger from '../../services/logger.js';
import { runGeoCordTask } from '../../services/crons/geocoding-cron.js';
const service = restana();
const userSettingsRouter = service.newRouter();
userSettingsRouter.get('/', async (req, res) => {
const userId = req.session.currentUser;
const rows = SqliteConnection.query('SELECT name, value FROM settings WHERE user_id = @userId', { userId });
const settings = {};
for (const r of rows) {
settings[r.name] = fromJson(r.value, null);
}
res.body = settings;
res.send();
});
userSettingsRouter.get('/autocomplete', async (req, res) => {
const { q } = req.query;
try {
const results = await autocompleteAddress(q);
res.body = results;
res.send();
} catch (error) {
res.statusCode = 500;
res.send({ error: error.message });
}
});
userSettingsRouter.post('/home-address', async (req, res) => {
const userId = req.session.currentUser;
const { home_address } = req.body;
const settings = await getSettings();
if (settings.demoMode) {
res.send(new Error('In demo mode, it is not allowed to change the home address.'));
return;
}
try {
if (home_address) {
await trackPoi(TRACKING_POIS.DISTANCE_ADDRESS_ENTERED);
const coords = await geocodeAddress(home_address);
if (coords && coords.lat !== -1) {
upsertSettings({ home_address: { address: home_address, coords } }, userId);
resetGeocoordinatesAndDistanceForUser(userId);
//we do NOT wait for this to finish, as we don't want to block the response
runGeoCordTask();
res.send({ success: true, coords });
} else {
res.statusCode = 400;
res.send({ error: 'Could not geocode address' });
}
} else {
upsertSettings({ home_address: null }, userId);
res.send({ success: true });
}
} catch (error) {
logger.error('Error updating home address settings', error);
res.statusCode = 500;
res.send({ error: error.message });
}
});
userSettingsRouter.post('/news-hash', async (req, res) => {
const userId = req.session.currentUser;
const { news_hash } = req.body;
const globalSettings = await getSettings();
if (globalSettings.demoMode) {
res.statusCode = 403;
res.send({ error: 'In demo mode, it is not allowed to change settings.' });
return;
}
try {
upsertSettings({ news_hash }, userId);
res.send({ success: true });
} catch (error) {
logger.error('Error updating news hash', error);
res.statusCode = 500;
res.send({ error: error.message });
}
});
export { userSettingsRouter };

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,14 +0,0 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
const FEATURES = {
WATCHLIST_MANAGEMENT: false,
};
export default function getFeatures() {
return {
...FEATURES,
};
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -8,7 +8,7 @@
*
* The mobile API provides the following endpoints:
* - GET /search/total?{search parameters}: Returns the total number of listings for the given query
* Example: `curl -H "User-Agent: ImmoScout_27.3_26.0_._" https://api.mobile.immobilienscout24.de/search/total?searchType=region&realestatetype=apartmentrent&pricetype=calculatedtotalrent&geocodes=%2Fde%2Fberlin%2Fberlin `
* Example: `curl -H "User-Agent: ImmoScout_27.12_26.2_._" https://api.mobile.immobilienscout24.de/search/total?searchType=region&realestatetype=apartmentrent&pricetype=calculatedtotalrent&geocodes=%2Fde%2Fberlin%2Fberlin `
*
* - POST /search/list?{search parameters}: Actually retrieves the listings. Body is json encoded and contains
* data specifying additional results (advertisements) to return. The format is as follows:
@@ -20,12 +20,12 @@
* ```
* It is not necessary to provide data for the specified keys.
*
* Example: `curl -X POST 'https://api.mobile.immobilienscout24.de/search/list?pricetype=calculatedtotalrent&realestatetype=apartmentrent&searchType=region&geocodes=%2Fde%2Fberlin%2Fberlin&pagenumber=1' -H "Connection: keep-alive" -H "User-Agent: ImmoScout_27.3_26.0_._" -H "Accept: application/json" -H "Content-Type: application/json" -d '{"supportedResultListType": [], "userData": {}}'`
* Example: `curl -X POST 'https://api.mobile.immobilienscout24.de/search/list?pricetype=calculatedtotalrent&realestatetype=apartmentrent&searchType=region&geocodes=%2Fde%2Fberlin%2Fberlin&pagenumber=1' -H "Connection: keep-alive" -H "User-Agent: ImmoScout_27.12_26.2_._" -H "Accept: application/json" -H "Content-Type: application/json" -d '{"supportedResultListType": [], "userData": {}}'`
* - GET /expose/{id} - Returns the details of a listing. The response contains additional details not included in the
* listing response.
*
* Example: `curl -H "User-Agent: ImmoScout_27.3_26.0_._" "https://api.mobile.immobilienscout24.de/expose/158382494"`
* Example: `curl -H "User-Agent: ImmoScout_27.12_26.2_._" "https://api.mobile.immobilienscout24.de/expose/158382494"`
*
*
* It is necessary to set the correct User Agent (see `getListings`) in the request header.
@@ -52,7 +52,7 @@ async function getListings(url) {
const response = await fetch(url, {
method: 'POST',
headers: {
'User-Agent': 'ImmoScout_27.3_26.0_._',
'User-Agent': 'ImmoScout_27.12_26.2_._',
'Content-Type': 'application/json',
},
body: JSON.stringify({
@@ -88,7 +88,7 @@ async function getListings(url) {
async function isListingActive(link) {
const result = await fetch(convertImmoscoutListingToMobileListing(link), {
headers: {
'User-Agent': 'ImmoScout_27.3_26.0_._',
'User-Agent': 'ImmoScout_27.12_26.2_._',
},
});

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -36,7 +36,7 @@ const config = {
},
normalize: normalize,
filter: applyBlacklist,
activeTester: checkIfListingIsActive,
activeTester: (url) => checkIfListingIsActive(url, 'Angebot nicht gefunden'),
};
export const init = (sourceConfig, blacklist) => {
config.enabled = sourceConfig.enabled;

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -0,0 +1,58 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import * as utils from '../utils.js';
import checkIfListingIsActive from '../services/listings/listingActiveTester.js';
let appliedBlackList = [];
function normalize(o) {
const id = o.link.split('/').pop();
const price = o.price;
const size = o.size;
const rooms = o.rooms;
const [city = '', part = ''] = (o.description || '').split('-').map((v) => v.trim());
const address = `${part}, ${city}`;
return Object.assign(o, { id, price, size, rooms, address });
}
function applyBlacklist(o) {
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
return o.id != null && o.title != null && titleNotBlacklisted && descNotBlacklisted && o.link.startsWith(o.link);
}
const config = {
url: null,
sortByDateParam: null,
waitForSelector: 'body',
crawlContainer: '.search_result_container > a',
crawlFields: {
id: '*',
title: 'h3 | trim',
price: 'dl:nth-of-type(1) dd | removeNewline | trim',
rooms: 'dl:nth-of-type(2) dd | removeNewline | trim',
size: 'dl:nth-of-type(3) dd | removeNewline | trim',
description: 'div.before\\:icon-location_marker | trim',
link: '@href',
imageUrl: 'img@src',
},
normalize: normalize,
filter: applyBlacklist,
activeTester: checkIfListingIsActive,
};
export const init = (sourceConfig, blacklistTerms) => {
config.url = sourceConfig.url;
appliedBlackList = blacklistTerms || [];
};
export const metaInformation = {
name: 'Wohnungsboerse',
baseUrl: 'https://www.wohnungsboerse.net',
id: 'wohnungsboerse',
};
export { config };

View File

@@ -1,29 +0,0 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { removeJobsByUserId } from '../storage/jobStorage.js';
import { getUsers } from '../storage/userStorage.js';
import logger from '../logger.js';
import cron from 'node-cron';
import { getSettings } from '../storage/settingsStorage.js';
/**
* if we are running in demo environment, we have to cleanup the db files (specifically the jobs table)
*/
export function cleanupDemoAtMidnight() {
cron.schedule('0 0 * * *', cleanup);
}
async function cleanup() {
const settings = await getSettings();
if (settings.demoMode) {
const demoUser = getUsers(false).find((user) => user.username === 'demo');
if (demoUser == null) {
logger.error('Demo user not found, cannot remove Jobs');
return Promise.resolve();
}
removeJobsByUserId(demoUser.id);
}
}

View File

@@ -0,0 +1,46 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import cron from 'node-cron';
import { getListingsToGeocode, updateListingGeocoordinates } from '../storage/listingsStorage.js';
import { geocodeAddress, isGeocodingPaused } from '../geocoding/geoCodingService.js';
import { getJobs } from '../storage/jobStorage.js';
import { calculateDistanceForJob } from '../geocoding/distanceService.js';
import { getSettings } from '../storage/settingsStorage.js';
import logger from '../logger.js';
export async function runGeoCordTask() {
const listings = getListingsToGeocode();
if (listings.length > 0) {
for (const listing of listings) {
if (isGeocodingPaused()) {
break;
}
const coords = await geocodeAddress(listing.address);
if (coords) {
updateListingGeocoordinates(listing.id, coords.lat, coords.lng);
}
}
}
//additional run
const jobs = getJobs();
for (const job of jobs) {
calculateDistanceForJob(job.id, job.userId);
}
}
export async function initGeocodingCron() {
const settings = await getSettings();
if (settings.demoMode) {
logger.info('Do not start geo service as we are in demo mode');
return;
}
// run directly on start
await runGeoCordTask();
// then every 6 hours
cron.schedule('0 */6 * * *', runGeoCordTask);
}

View File

@@ -1,16 +1,23 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import cron from 'node-cron';
import runActiveChecker from '../listings/listingActiveService.js';
import logger from '../logger.js';
import { getSettings } from '../storage/settingsStorage.js';
async function runTask() {
await runActiveChecker();
}
export async function initActiveCheckerCron() {
const settings = await getSettings();
if (settings.demoMode) {
logger.info('Do not start listing active checker as we are in demo mode');
return;
}
//run directly on start
await runTask();
// then every day at 1 am

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -22,7 +22,7 @@ puppeteer.use(StealthPlugin());
export default async function execute(url, waitForSelector, options) {
let browser;
let page;
let result = null;
let result;
let userDataDir;
let removeUserDataDir = false;
try {

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -0,0 +1,26 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { autocomplete as nominatimAutocomplete } from './client/nominatimClient.js';
import logger from '../logger.js';
/**
* Autocompletes an address using Nominatim.
*
* @param {string} query - The search query.
* @returns {Promise<string[]>} List of matching addresses.
*/
export async function autocompleteAddress(query) {
if (!query) {
return [];
}
try {
return await nominatimAutocomplete(query);
} catch (error) {
logger.error('Error during address autocomplete:', error);
return [];
}
}

View File

@@ -0,0 +1,152 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import os from 'os';
import crypto from 'crypto';
import https from 'https';
import fetch from 'node-fetch';
import pThrottle from 'p-throttle';
import logger from '../../logger.js';
const API_URL = 'https://nominatim.openstreetmap.org/search';
const agent = new https.Agent({
keepAlive: true,
keepAliveMsecs: 1000,
});
const throttle = pThrottle({
limit: 1,
interval: 1000,
});
function computeMachineId() {
const hostname = os.hostname() || 'unknown-host';
const nets = os.networkInterfaces?.() || {};
const macs = [];
for (const ifname of Object.keys(nets)) {
for (const addr of nets[ifname] || []) {
if (!addr) continue;
if (addr.internal) continue;
if (addr.mac && addr.mac !== '00:00:00:00:00:00') macs.push(addr.mac);
}
}
macs.sort();
const raw = [hostname, os.platform(), os.arch(), ...macs].join('|');
return crypto.createHash('sha256').update(raw).digest('hex').slice(0, 20);
}
/**
* Nominatim requires a specific User-Agent.
* Since Fredy is self-hosted, we use a unique machine ID to make it specific.
*/
const userAgent = `Fredy-Self-Hosted (${computeMachineId()}; https://github.com/orangecoding/fredy)`;
let last429 = 0;
const PAUSE_DURATION = 3600000; // 1 hour
/**
* Geocodes an address using Nominatim.
*
* @param {string} address - The address to geocode.
* @returns {Promise<{lat: number, lng: number}|null>} The geocoordinates or null if error. {lat: -1, lng: -1} if not found.
*/
async function doGeocode(address) {
if (Date.now() - last429 < PAUSE_DURATION) {
return null;
}
const url = `${API_URL}?q=${encodeURIComponent(address)}&format=json&countrycodes=de`;
try {
const response = await fetch(url, {
agent,
headers: {
'User-Agent': userAgent,
},
});
if (response.status === 429) {
logger.warn('Nominatim rate limit hit. Pausing for 1 hour.');
last429 = Date.now();
return null;
}
if (!response.ok) {
logger.error(`Nominatim API error: ${response.status} ${response.statusText}`);
return null;
}
const data = await response.json();
if (Array.isArray(data) && data.length > 0) {
const result = data[0];
return {
lat: parseFloat(result.lat),
lng: parseFloat(result.lon),
};
}
return { lat: -1, lng: -1 };
} catch (error) {
logger.error('Error during Nominatim geocoding:', error);
return null;
}
}
/**
* Autocompletes an address using Nominatim.
*
* @param {string} query - The search query.
* @returns {Promise<string[]>} List of matching addresses.
*/
async function doAutocomplete(query) {
if (Date.now() - last429 < PAUSE_DURATION) {
return [];
}
const url = `${API_URL}?q=${encodeURIComponent(query)}&format=json&addressdetails=1&limit=5&countrycodes=de`;
try {
const response = await fetch(url, {
agent,
headers: {
'User-Agent': userAgent,
},
});
if (response.status === 429) {
logger.warn('Nominatim rate limit hit. Pausing for 1 hour.');
last429 = Date.now();
return [];
}
if (!response.ok) {
logger.error(`Nominatim API error: ${response.status} ${response.statusText}`);
return [];
}
const data = await response.json();
if (Array.isArray(data)) {
return data.map((item) => item.display_name);
}
return [];
} catch (error) {
logger.error('Error during Nominatim autocomplete:', error);
return [];
}
}
export const geocode = throttle(doGeocode);
export const autocomplete = throttle(doAutocomplete);
export const isPaused = () => Date.now() - last429 < PAUSE_DURATION;

View File

@@ -0,0 +1,61 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { distanceMeters } from '../listings/distanceCalculator.js';
import {
getListingsToCalculateDistance,
getListingsForUserToCalculateDistance,
updateListingDistance,
} from '../storage/listingsStorage.js';
import { getUserSettings } from '../storage/settingsStorage.js';
/**
* Calculates and updates distances for listings of a specific job.
* Only processes listings where distance_to_destination is null.
*
* @param {string} jobId
* @param {string} userId
* @returns {void}
*/
export function calculateDistanceForJob(jobId, userId) {
const userSettings = getUserSettings(userId);
const homeAddress = userSettings.home_address;
if (!homeAddress || !homeAddress.coords) {
return;
}
const listings = getListingsToCalculateDistance(jobId);
const { lat, lng } = homeAddress.coords;
for (const listing of listings) {
const dist = distanceMeters(lat, lng, listing.latitude, listing.longitude);
updateListingDistance(listing.id, dist);
}
}
/**
* Calculates and updates distances for all active listings of a user.
* Usually called when the user updates their home address.
*
* @param {string} userId
* @returns {void}
*/
export function calculateDistanceForUser(userId) {
const userSettings = getUserSettings(userId);
const homeAddress = userSettings.home_address;
if (!homeAddress || !homeAddress.coords) {
return;
}
const listings = getListingsForUserToCalculateDistance(userId);
const { lat, lng } = homeAddress.coords;
for (const listing of listings) {
const dist = distanceMeters(lat, lng, listing.latitude, listing.longitude);
updateListingDistance(listing.id, dist);
}
}

View File

@@ -0,0 +1,43 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
import { getGeocoordinatesByAddress } from '../storage/listingsStorage.js';
import { geocode as nominatimGeocode, isPaused as isNominatimPaused } from './client/nominatimClient.js';
import logger from '../logger.js';
/**
* Geocodes an address using Nominatim or cached results from the database.
*
* @param {string} address - The address to geocode.
* @returns {Promise<{lat: number, lng: number}|null>} The geocoordinates or null if error. {lat: -1, lng: -1} if not found.
*/
export async function geocodeAddress(address) {
if (!address) {
return null;
}
try {
// 1. Check if we already have this address geocoded in our database
const cachedCoordinates = getGeocoordinatesByAddress(address);
if (cachedCoordinates) {
logger.debug(`Found cached geocoordinates for address: ${address}`);
return cachedCoordinates;
}
// 2. If not, use Nominatim
return await nominatimGeocode(address);
} catch (error) {
logger.error('Error during geocoding:', error);
return null;
}
}
/**
* Checks if we are currently in a rate limit pause.
* @returns {boolean}
*/
export function isGeocodingPaused() {
return isNominatimPaused();
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -103,13 +103,17 @@ const REAL_ESTATE_TYPE = {
'haus-mieten': 'houserent',
'wohnung-mieten': 'apartmentrent',
'wohnung-kaufen': 'apartmentbuy',
'wohnung-kaufen-mit-balkon': 'apartmentbuy',
'eigentumswohnung-mit-garten': 'apartmentbuy',
'haus-kaufen': 'housebuy',
};
const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
// Category "Balkon/Terrasse"
'wohnung-mit-balkon-mieten': { equipment: ['balcony'] },
'wohnung-kaufen-mit-balkon': { equipment: ['balcony'] },
'wohnung-mit-garten-mieten': { equipment: ['garden'] },
'eigentumswohnung-mit-garten': { equipment: ['garden'] },
// Category "Wohnungstyp"
'souterrainwohnung-mieten': { apartmenttypes: ['halfbasement'] },
'erdgeschosswohnung-mieten': { apartmenttypes: ['groundfloor'] },
@@ -144,7 +148,7 @@ export function convertWebToMobile(webUrl) {
const realTypeKey = segments.at(-1);
let realType = REAL_ESTATE_TYPE[realTypeKey];
let additionalParamsFromWebPath;
let additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey] || null;
if (!realType) {
// Test for seo optimized apartment path (only used on the ImmoScout web app)
@@ -165,7 +169,7 @@ export function convertWebToMobile(webUrl) {
Object.entries(rawParams).filter(([key]) => key !== 'enteredFrom' && PARAM_NAME_MAP[key]),
);
const geocodes = `/${segments.slice(2, 5).join('/')}`;
const geocodes = `/${segments.slice(2, segments.length - 1).join('/')}`;
const isRadius = segments.includes('radius');
const mobileParams = {
searchType: isRadius ? 'radius' : 'region',

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -0,0 +1,35 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
const R = 6371000; // Earth radius in meters
/**
* Calculate the great-circle distance between two points on Earth using the Haversine formula.
* This is to calculate the distance between the listing address & the address provided by the user. I know, it is only
* a rough estimation as this calculates the distance as a straight line, but it's more convenient than using an external
* service and still gives a good approximation for sorting purposes.
* Returns distance in meters.
*
* @param {number} lat1
* @param {number} lon1
* @param {number} lat2
* @param {number} lon2
* @returns {number}
*/
export function distanceMeters(lat1, lon1, lat2, lon2) {
const toRad = (deg) => (deg * Math.PI) / 180;
const phi1 = toRad(lat1);
const phi2 = toRad(lat2);
const dPhi = toRad(lat2 - lat1);
const dLambda = toRad(lon2 - lon1);
const a =
Math.sin(dPhi / 2) * Math.sin(dPhi / 2) +
Math.cos(phi1) * Math.cos(phi2) * Math.sin(dLambda / 2) * Math.sin(dLambda / 2);
const c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
return Math.round(R * c * 10) / 10;
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -8,37 +8,71 @@ import { randomBetween, sleep } from '../../utils.js';
const maxAttempts = 3;
const userAgents = [
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.2 Safari/605.1.15',
'Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1',
];
/**
* Check if a listing is still active with up to 3 attempts and exponential backoff.
* Backoff waits are capped and the last wait is at most 2000 ms.
* Check if a listing is still active with up to 5 attempts and exponential backoff.
* Backoff waits are randomized and capped.
*
* Rules:
* - HTTP 200 => return 1
* - HTTP 200 => return 1 (if checkForText is provided and found, returns 0)
* - HTTP 401/403 => return -1 (most certainly detected as a bot)
* - HTTP 404 => return 0
* - Other statuses or network errors => retry until attempts are exhausted
*
* @returns {Promise<Integer>} 1 if active, o if not active and -1 if detected as bot
* @returns {Promise<Integer>} 1 if active, 0 if not active and -1 if detected as bot
*/
export default async function checkIfListingIsActive(link) {
export default async function checkIfListingIsActive(link, checkForText = null) {
await sleep(randomBetween(50, 100));
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
const userAgent = userAgents[Math.floor(Math.random() * userAgents.length)];
const res = await fetch(link, {
redirect: 'manual',
headers: {
'User-Agent':
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
'Accept-Language': 'de-DE,de;q=0.9,en;q=0.8',
'User-Agent': userAgent,
Accept:
'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
'Accept-Language': 'de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7',
'Accept-Encoding': 'gzip, deflate, br',
'Cache-Control': 'max-age=0',
'Sec-Ch-Ua': '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"',
'Sec-Ch-Ua-Mobile': '?0',
'Sec-Ch-Ua-Platform': '"macOS"',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'none',
'Sec-Fetch-User': '?1',
'Upgrade-Insecure-Requests': '1',
Referer: 'https://www.google.com/',
},
});
if (res.status === 200) {
if (checkForText) {
const htmText = await res.text();
if (htmText.includes(checkForText)) {
return 0;
}
}
return 1;
}
if (res.status === 401) return -1;
if (res.status === 403) return -1;
if (res.status === 404) return 0;
if (res.status === 401 || res.status === 403) {
if (attempt < maxAttempts) {
await sleep(backoffDelay(attempt));
continue;
}
return -1;
}
if (res.status === 404 || res.status === 410) return 0;
// For any other status, only retry if attempts remain
if (attempt < maxAttempts) {
@@ -61,13 +95,13 @@ export default async function checkIfListingIsActive(link) {
}
/**
* Exponential backoff delay with cap.
* attempt: 1 -> 500ms, 2 -> 1000ms, 3 -> 2000ms (cap)
* Exponential backoff delay with cap and jitter.
* @param {number} attempt 1-based attempt index
* @returns {number} delay in ms
*/
function backoffDelay(attempt) {
const base = 500;
const cap = 2000;
return Math.min(base * 2 ** (attempt - 1), cap);
const delay = Math.min(base * 2 ** (attempt - 1), cap);
return delay + randomBetween(0, 1000);
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -37,12 +37,12 @@ export const upsertJob = ({
if (existing) {
SqliteConnection.execute(
`UPDATE jobs
SET enabled = @enabled,
name = @name,
blacklist = @blacklist,
provider = @provider,
notification_adapter = @notification_adapter,
shared_with_user = @shareWithUsers
SET enabled = @enabled,
name = @name,
blacklist = @blacklist,
provider = @provider,
notification_adapter = @notification_adapter,
shared_with_user = @shareWithUsers
WHERE id = @id`,
{
id,
@@ -87,10 +87,10 @@ export const getJob = (jobId) => {
j.provider,
j.shared_with_user,
j.notification_adapter AS notificationAdapter,
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id) AS numberOfFoundListings
FROM jobs j
WHERE j.id = @id
LIMIT 1`,
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id AND l.is_active = 1 AND l.manually_deleted = 0) AS numberOfFoundListings
FROM jobs j
WHERE j.id = @id
LIMIT 1`,
{ id: jobId },
)[0];
if (!row) return null;
@@ -150,9 +150,10 @@ export const getJobs = () => {
j.provider,
j.shared_with_user,
j.notification_adapter AS notificationAdapter,
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id) AS numberOfFoundListings
FROM jobs j
ORDER BY j.name IS NULL, j.name`,
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id AND l.is_active = 1 AND l.manually_deleted = 0) AS numberOfFoundListings
FROM jobs j
WHERE j.enabled = 1
ORDER BY j.name IS NULL, j.name`,
);
return rows.map((row) => ({
...row,
@@ -250,11 +251,11 @@ export const queryJobs = ({
j.provider,
j.shared_with_user,
j.notification_adapter AS notificationAdapter,
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id) AS numberOfFoundListings
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id AND l.is_active = 1 AND l.manually_deleted = 0) AS numberOfFoundListings
FROM jobs j
${whereSql}
${orderSql}
LIMIT @limit OFFSET @offset`,
${orderSql}
LIMIT @limit OFFSET @offset`,
params,
);

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -48,7 +48,8 @@ export const getListingsKpisForJobIds = (jobIds = []) => {
SUM(CASE WHEN is_active = 1 THEN 1 ELSE 0 END) AS activeCount,
AVG(price) AS avgPrice
FROM listings
WHERE job_id IN (${placeholders})`,
WHERE job_id IN (${placeholders})
AND manually_deleted = 0`,
jobIds,
)[0] || {};
@@ -80,6 +81,7 @@ export const getProviderDistributionForJobIds = (jobIds = []) => {
`SELECT provider, COUNT(*) AS cnt
FROM listings
WHERE job_id IN (${placeholders})
AND manually_deleted = 0
GROUP BY provider
ORDER BY cnt DESC`,
jobIds,
@@ -118,8 +120,8 @@ export const getActiveOrUnknownListings = () => {
return SqliteConnection.query(
`SELECT *
FROM listings
WHERE is_active is null
OR is_active = 1
WHERE (is_active is null OR is_active = 1)
AND manually_deleted = 0
ORDER BY provider`,
);
};
@@ -173,9 +175,9 @@ export const storeListings = (jobId, providerId, listings) => {
SqliteConnection.withTransaction((db) => {
const stmt = db.prepare(
`INSERT INTO listings (id, hash, provider, job_id, price, size, title, image_url, description, address,
link, created_at, is_active)
link, created_at, is_active, latitude, longitude)
VALUES (@id, @hash, @provider, @job_id, @price, @size, @title, @image_url, @description, @address, @link,
@created_at, 1)
@created_at, 1, @latitude, @longitude)
ON CONFLICT(job_id, hash) DO NOTHING`,
);
@@ -193,6 +195,8 @@ export const storeListings = (jobId, providerId, listings) => {
address: removeParentheses(item.address),
link: item.link,
created_at: Date.now(),
latitude: item.latitude || null,
longitude: item.longitude || null,
};
stmt.run(params);
}
@@ -304,6 +308,9 @@ export const queryListings = ({
whereParts.push('(wl.id IS NULL)');
}
// Build whereSql (filtering by manually_deleted = 0)
whereParts.push('(l.manually_deleted = 0)');
const whereSql = whereParts.length ? `WHERE ${whereParts.join(' AND ')}` : '';
const whereSqlWithAlias = whereSql
.replace(/\btitle\b/g, 'l.title')
@@ -363,13 +370,21 @@ export const queryListings = ({
* Delete all listings for a given job id.
*
* @param {string} jobId - The job identifier whose listings should be removed.
* @returns {any} The result from SqliteConnection.execute (may contain changes count).
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
* @returns {any} The result from SqliteConnection.execute.
*/
export const deleteListingsByJobId = (jobId) => {
export const deleteListingsByJobId = (jobId, hardDelete = false) => {
if (!jobId) return;
if (hardDelete) {
return SqliteConnection.execute(
`DELETE FROM listings
WHERE job_id = @jobId`,
{ jobId },
);
}
return SqliteConnection.execute(
`DELETE
FROM listings
`UPDATE listings
SET manually_deleted = 1
WHERE job_id = @jobId`,
{ jobId },
);
@@ -379,19 +394,107 @@ export const deleteListingsByJobId = (jobId) => {
* Delete listings by a list of listing IDs.
*
* @param {string[]} ids - Array of listing IDs to delete.
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
* @returns {any} The result from SqliteConnection.execute.
*/
export const deleteListingsById = (ids) => {
export const deleteListingsById = (ids, hardDelete = false) => {
if (!Array.isArray(ids) || ids.length === 0) return;
const placeholders = ids.map(() => '?').join(',');
if (hardDelete) {
return SqliteConnection.execute(
`DELETE FROM listings
WHERE id IN (${placeholders})`,
ids,
);
}
return SqliteConnection.execute(
`DELETE
FROM listings
WHERE id IN (${placeholders})`,
`UPDATE listings
SET manually_deleted = 1
WHERE id IN (${placeholders})`,
ids,
);
};
/**
* Return all listings that are active, have an address, and do not yet have geocoordinates.
*
* @returns {Object[]} Array of listing objects {id, address}.
*/
export const getListingsToGeocode = () => {
return SqliteConnection.query(
`SELECT id, address
FROM listings
WHERE is_active = 1
AND manually_deleted = 0
AND address IS NOT NULL
AND (latitude IS NULL OR longitude IS NULL)`,
);
};
/**
* Update the geocoordinates for a listing.
*
* @param {string} id - The listing ID.
* @param {number} latitude
* @param {number} longitude
* @returns {void}
*/
export const updateListingGeocoordinates = (id, latitude, longitude) => {
SqliteConnection.execute(
`UPDATE listings
SET latitude = @latitude,
longitude = @longitude
WHERE id = @id`,
{ id, latitude, longitude },
);
};
/**
* Return listings with geocoordinates for the map view, with optional filtering.
*
* @param {Object} params
* @param {string} [params.jobId]
* @param {string} [params.userId]
* @param {boolean} [params.isAdmin=false]
* @returns {{listings: Object[], maxPrice: number}} Object containing listings and maxPrice.
*/
export const getListingsForMap = ({ jobId, userId = null, isAdmin = false } = {}) => {
const baseWhereParts = [
'l.latitude IS NOT NULL',
'l.longitude IS NOT NULL',
'l.latitude != -1',
'l.longitude != -1',
'l.is_active = 1',
'l.manually_deleted = 0',
];
const params = { userId: userId || '__NO_USER__' };
if (!isAdmin) {
baseWhereParts.push(
`(j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`,
);
}
if (jobId) {
params.jobId = jobId;
baseWhereParts.push('l.job_id = @jobId');
}
const wherePartsForListings = [...baseWhereParts];
const listings = SqliteConnection.query(
`SELECT l.*, j.name AS job_name
FROM listings l
LEFT JOIN jobs j ON j.id = l.job_id
WHERE ${wherePartsForListings.join(' AND ')}`,
params,
);
return {
listings,
};
};
/**
* Return all listings with only the fields: title, address, and price.
* This is the single helper requested for simple consumers.
@@ -399,5 +502,129 @@ export const deleteListingsById = (ids) => {
* @returns {{title: string|null, address: string|null, price: number|null}[]}
*/
export const getAllEntriesFromListings = () => {
return SqliteConnection.query(`SELECT title, address, price FROM listings`);
return SqliteConnection.query(`SELECT title, address, price FROM listings WHERE manually_deleted = 0`);
};
/**
* Return geocoordinates for a given address if it has been geocoded before.
*
* @param {string} address
* @returns {{lat: number, lng: number}|null}
*/
export const getGeocoordinatesByAddress = (address) => {
const row = SqliteConnection.query(
`SELECT latitude, longitude
FROM listings
WHERE address = @address
AND manually_deleted = 0
AND latitude IS NOT NULL
AND longitude IS NOT NULL
AND latitude != -1
AND longitude != -1
LIMIT 1`,
{ address },
)[0];
return row ? { lat: row.latitude, lng: row.longitude } : null;
};
/**
* Return all active listings for a given job that have geocoordinates but no distance set.
*
* @param {string} jobId
* @returns {Object[]}
*/
export const getListingsToCalculateDistance = (jobId) => {
return SqliteConnection.query(
`SELECT id, latitude, longitude
FROM listings
WHERE job_id = @jobId
AND is_active = 1
AND manually_deleted = 0
AND latitude IS NOT NULL
AND longitude IS NOT NULL
AND distance_to_destination IS NULL`,
{ jobId },
);
};
/**
* Return all active listings for a given user (across all jobs) that have geocoordinates.
*
* @param {string} userId
* @returns {Object[]}
*/
export const getListingsForUserToCalculateDistance = (userId) => {
return SqliteConnection.query(
`SELECT l.id, l.latitude, l.longitude
FROM listings l
JOIN jobs j ON l.job_id = j.id
WHERE j.user_id = @userId
AND l.is_active = 1
AND l.manually_deleted = 0
AND l.latitude IS NOT NULL
AND l.longitude IS NOT NULL`,
{ userId },
);
};
/**
* Update the distance to destination for a listing.
*
* @param {string} id
* @param {number} distance
* @returns {void}
*/
export const updateListingDistance = (id, distance) => {
SqliteConnection.execute(
`UPDATE listings
SET distance_to_destination = @distance
WHERE id = @id`,
{ id, distance },
);
};
/**
* Return a single listing by id.
*
* @param {string} id
* @param {string} userId
* @param {boolean} isAdmin
* @returns {Object|null}
*/
export const getListingById = (id, userId = null, isAdmin = false) => {
const params = { id, userId: userId || '__NO_USER__' };
let whereScoping = '';
if (!isAdmin) {
whereScoping = `AND (j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`;
}
return (
SqliteConnection.query(
`SELECT l.*, j.name AS job_name, CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END AS isWatched
FROM listings l
LEFT JOIN jobs j ON j.id = l.job_id
LEFT JOIN watch_list wl ON wl.listing_id = l.id AND wl.user_id = @userId
WHERE l.id = @id AND l.manually_deleted = 0 ${whereScoping}`,
params,
)[0] || null
);
};
/**
* Resets geocoordinates and distance for all listings related to a user.
*
* @param {string} userId
* @returns {void}
*/
export const resetGeocoordinatesAndDistanceForUser = (userId) => {
SqliteConnection.execute(
`UPDATE listings
SET latitude = NULL,
longitude = NULL,
distance_to_destination = NULL
WHERE job_id IN (
SELECT id FROM jobs j
WHERE j.user_id = @userId
)`,
{ userId },
);
};

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
@@ -88,7 +88,7 @@ export function up(db) {
}
} catch (e) {
// If parsing fails, let it throw to rollback the migration
throw new Error(`Failed to import users from ${usersJsonPath}: ${e.message}`);
throw new Error(`Failed to import users from ${usersJsonPath}: ${e.message}`, { cause: e });
}
}
@@ -116,7 +116,7 @@ export function up(db) {
}
}
} catch (e) {
throw new Error(`Failed to import jobs from ${jobsJsonPath}: ${e.message}`);
throw new Error(`Failed to import jobs from ${jobsJsonPath}: ${e.message}`, { cause: e });
}
}
}

View File

@@ -0,0 +1,16 @@
/*
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/
export function up(db) {
// 1. Add manually_deleted column
db.exec(`ALTER TABLE listings ADD COLUMN manually_deleted INTEGER NOT NULL DEFAULT 0;`);
// 2. Remove change_set column
try {
db.exec(`ALTER TABLE listings DROP COLUMN change_set;`);
} catch {
// if column does not exists for whatever reason
}
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

View File

@@ -1,5 +1,5 @@
/*
* Copyright (c) 2025 by Christian Kellner.
* Copyright (c) 2026 by Christian Kellner.
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

Some files were not shown because too many files have changed in this diff Show More