mirror of
https://github.com/orangecoding/fredy.git
synced 2026-06-16 12:31:07 +00:00
Compare commits
50 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
be5c4af3cf | ||
|
|
a460b813c1 | ||
|
|
4596442f64 | ||
|
|
0bcfa1d4ad | ||
|
|
0cad05124a | ||
|
|
eb53b68d45 | ||
|
|
ba0732e1f6 | ||
|
|
aa67647bbb | ||
|
|
7a9d49899b | ||
|
|
9a87c58d3e | ||
|
|
fdd7e835e8 | ||
|
|
00d6a12b30 | ||
|
|
05218800d2 | ||
|
|
19d4721f9f | ||
|
|
a794645393 | ||
|
|
fd7e228972 | ||
|
|
b86e351007 | ||
|
|
19c4860da7 | ||
|
|
d98e06cfdf | ||
|
|
6ae0c9749b | ||
|
|
10e40e038e | ||
|
|
4ba6828939 | ||
|
|
d09770dae2 | ||
|
|
248e4d2562 | ||
|
|
7b8e961b49 | ||
|
|
f66ceccbb4 | ||
|
|
a3db725af6 | ||
|
|
0663bd945f | ||
|
|
bc355fb5fe | ||
|
|
797421f0d5 | ||
|
|
0b2b42fc75 | ||
|
|
472169693f | ||
|
|
3117044139 | ||
|
|
7879d0e94a | ||
|
|
afd1048c9e | ||
|
|
acbaab05ed | ||
|
|
72fffc526b | ||
|
|
9e5989ece3 | ||
|
|
afc200c9e1 | ||
|
|
59226491f2 | ||
|
|
28f7760120 | ||
|
|
2465514b7a | ||
|
|
9dde377fe6 | ||
|
|
28a3a7f372 | ||
|
|
e859250545 | ||
|
|
4dd0370ec1 | ||
|
|
51b4e51f3f | ||
|
|
fa1899765c | ||
|
|
d43c5b3f97 | ||
|
|
7fd8be07a2 |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -6,3 +6,4 @@ npm-debug.log
|
|||||||
.DS_Store
|
.DS_Store
|
||||||
.idea
|
.idea
|
||||||
.vscode
|
.vscode
|
||||||
|
tools/release/config.json
|
||||||
|
|||||||
94
CHANGELOG.md
94
CHANGELOG.md
@@ -1,94 +0,0 @@
|
|||||||
Newer release changelog see https://github.com/orangecoding/fredy/releases
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
###### [V5.5.0]
|
|
||||||
|
|
||||||
- Upgrading dependencies
|
|
||||||
- fixing provider
|
|
||||||
- allow multiple instances of 1 provider
|
|
||||||
- **BREAKING**: Minimum node version is now 16
|
|
||||||
|
|
||||||
###### [V5.4.6]
|
|
||||||
|
|
||||||
- Adding Instana node.js monitoring
|
|
||||||
-
|
|
||||||
|
|
||||||
###### [V5.4.5]
|
|
||||||
|
|
||||||
- Adding Instana node.js monitoring
|
|
||||||
|
|
||||||
###### [V5.4.4]
|
|
||||||
|
|
||||||
- Add support for Immo Südwest Presse (immo.swp.de)
|
|
||||||
- Telegram: Use job name instead of ID and link in title
|
|
||||||
- Fix race condition if user ID is in session but not in user store
|
|
||||||
- Allow visiting the original provider URL
|
|
||||||
|
|
||||||
###### [V5.4.3]
|
|
||||||
|
|
||||||
- re-writing readme
|
|
||||||
- improving docker build
|
|
||||||
- using github's actions to build docker and test automatically
|
|
||||||
|
|
||||||
###### [V5.4.2]
|
|
||||||
|
|
||||||
- Fixing prod build
|
|
||||||
|
|
||||||
###### [V5.4.1]
|
|
||||||
|
|
||||||
- Upgrading dependencies
|
|
||||||
- Provider urls are now automagically been changed to include the correct sort order for search results
|
|
||||||
|
|
||||||
```
|
|
||||||
Note: It has been an point of confusion since the very beginning of Fredy, that people simply copied the url, but
|
|
||||||
did not take care of sorting the search results by date. If this is not done, Fredy will most likely not see the latest
|
|
||||||
results, thus cannot report them. This release fixes it by adding the necessary params (or replaces them).
|
|
||||||
```
|
|
||||||
|
|
||||||
###### [V5.3.0]
|
|
||||||
|
|
||||||
- Upgrading dependencies
|
|
||||||
- It's now possible to send mails to multiple receiver using comma separation for MailJet & Sendgrid
|
|
||||||
- Fixing Immowelt scraping
|
|
||||||
|
|
||||||
###### [V5.2.0]
|
|
||||||
|
|
||||||
- Upgrading dependencies
|
|
||||||
- Adding new similarity check layer (Duplicates are being removed now)
|
|
||||||
- Adding paging for search results
|
|
||||||
|
|
||||||
###### [V5.1.0]
|
|
||||||
|
|
||||||
- Upgrading dependencies
|
|
||||||
- NodeJS 12.13 is now the minimum supported version
|
|
||||||
- Adding general settings as new configuration page to ui
|
|
||||||
- Adding new feature working hours
|
|
||||||
|
|
||||||
###### [V5.0.0]
|
|
||||||
|
|
||||||
- Upgrading dependencies
|
|
||||||
- NodeJS 12 is now the minimum supported version
|
|
||||||
|
|
||||||
###### [V4.0.0]
|
|
||||||
|
|
||||||
Bringing back Immoscout :tada:
|
|
||||||
|
|
||||||
###### [V3.0.0]
|
|
||||||
|
|
||||||
This is basically a re-write, your old config file will not be compatible anymore. Please re-created your search jobs
|
|
||||||
on the new ui and use the values from your previous config file if needed.
|
|
||||||
|
|
||||||
```
|
|
||||||
- We're getting rid of manual config changes, Fredy, now ships with a UI so that it's easy for you to create and edit jobs
|
|
||||||
```
|
|
||||||
|
|
||||||
###### [V2.0.0]
|
|
||||||
|
|
||||||
```
|
|
||||||
- Fredy can now run multiple search job on one instance
|
|
||||||
- Changed lot's of the structure of Fredy to make this happen
|
|
||||||
[BREAKING CHANGES]
|
|
||||||
- The config has been changed, the config of V1.x will not work any longer
|
|
||||||
- Sources have been renamed to provider
|
|
||||||
```
|
|
||||||
@@ -35,6 +35,7 @@ WORKDIR /fredy
|
|||||||
RUN apk add --no-cache chromium curl
|
RUN apk add --no-cache chromium curl
|
||||||
|
|
||||||
ENV NODE_ENV=production \
|
ENV NODE_ENV=production \
|
||||||
|
IS_DOCKER=true \
|
||||||
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
|
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
|
||||||
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
|
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
|
||||||
|
|
||||||
|
|||||||
@@ -119,7 +119,7 @@ Should you use [Unraid](https://unraid.net/), you can now install Fredy from the
|
|||||||
|
|
||||||
## 📸 Screenshots
|
## 📸 Screenshots
|
||||||
|
|
||||||
| Fredy Main Overview | Job Configuration | Found Listings |
|
| Fredy Maps View | Dashboard | Found Listings |
|
||||||
|--------------------------------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------------------------|
|
|--------------------------------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------------------------|
|
||||||
|  |  |  |
|
|  |  |  |
|
||||||
|
|
||||||
@@ -154,6 +154,13 @@ to Slack + Telegram."\
|
|||||||
Jobs run automatically at the interval you configure (see
|
Jobs run automatically at the interval you configure (see
|
||||||
`/conf/config.json`).
|
`/conf/config.json`).
|
||||||
|
|
||||||
|
### MCP Server 🤖
|
||||||
|
|
||||||
|
Starting with **V20**, Fredy ships with a built-in **MCP Server **. This allows you to connect Fredy to LLMs (like Claude, ChatGPT, or local models via LM Studio) and query your real estate data using natural language.
|
||||||
|
The local LLM can even enrich existing listings by checking the listing online.
|
||||||
|
|
||||||
|
For more information on how to set it up and use it, please refer to the [MCP Readme](mcp/README.md).
|
||||||
|
|
||||||
------------------------------------------------------------------------
|
------------------------------------------------------------------------
|
||||||
|
|
||||||
## Immoscout
|
## Immoscout
|
||||||
|
|||||||
12
copyright.js
12
copyright.js
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -30,12 +30,16 @@ async function getAllFiles(dir = '.') {
|
|||||||
|
|
||||||
/* eslint-disable no-console */
|
/* eslint-disable no-console */
|
||||||
async function addCopyright(files) {
|
async function addCopyright(files) {
|
||||||
|
const oldCopyrightRegex =
|
||||||
|
/^(\/\*\n \* Copyright \(c\) \d{4} by Christian Kellner\.\n \* Licensed under Apache-2.0 with Commons Clause and Attribution\/Naming Clause\n \*\/\n\n)+/;
|
||||||
for (let file of files) {
|
for (let file of files) {
|
||||||
try {
|
try {
|
||||||
let content = await fs.readFile(file, 'utf8');
|
let content = await fs.readFile(file, 'utf8');
|
||||||
if (!content.startsWith(COPYRIGHT)) {
|
const strippedContent = content.replace(oldCopyrightRegex, '');
|
||||||
await fs.writeFile(file, COPYRIGHT + content);
|
const newContent = COPYRIGHT + strippedContent;
|
||||||
console.log(`Added copyright to ${file}`);
|
if (content !== newContent) {
|
||||||
|
await fs.writeFile(file, newContent);
|
||||||
|
console.log(`Added/Updated copyright in ${file}`);
|
||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.error(`Error processing ${file}: ${err}`);
|
console.error(`Error processing ${file}: ${err}`);
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 248 KiB After Width: | Height: | Size: 3.7 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.7 MiB After Width: | Height: | Size: 4.8 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 402 KiB After Width: | Height: | Size: 531 KiB |
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -8,20 +8,20 @@ import js from '@eslint/js';
|
|||||||
import prettier from 'eslint-config-prettier';
|
import prettier from 'eslint-config-prettier';
|
||||||
import globals from 'globals';
|
import globals from 'globals';
|
||||||
import react from 'eslint-plugin-react';
|
import react from 'eslint-plugin-react';
|
||||||
import babelParser from '@babel/eslint-parser';
|
|
||||||
|
|
||||||
export default [
|
export default [
|
||||||
{
|
{
|
||||||
files: ['**/*.{js,jsx,ts,tsx}'],
|
|
||||||
ignores: ['**/node_modules/**', '**/dist/**', '**/build/**', '**/public/**', 'db/**', 'conf/**'],
|
ignores: ['**/node_modules/**', '**/dist/**', '**/build/**', '**/public/**', 'db/**', 'conf/**'],
|
||||||
},
|
},
|
||||||
js.configs.recommended,
|
|
||||||
prettier,
|
|
||||||
{
|
{
|
||||||
|
files: ['**/*.{js,jsx}'],
|
||||||
languageOptions: {
|
languageOptions: {
|
||||||
parser: babelParser,
|
ecmaVersion: 'latest',
|
||||||
sourceType: 'module',
|
sourceType: 'module',
|
||||||
ecmaVersion: 2021,
|
parserOptions: {
|
||||||
|
ecmaFeatures: { jsx: true },
|
||||||
|
},
|
||||||
globals: {
|
globals: {
|
||||||
...globals.browser,
|
...globals.browser,
|
||||||
...globals.node,
|
...globals.node,
|
||||||
@@ -32,70 +32,14 @@ export default [
|
|||||||
after: 'readonly',
|
after: 'readonly',
|
||||||
it: 'readonly',
|
it: 'readonly',
|
||||||
},
|
},
|
||||||
parserOptions: { requireConfigFile: false },
|
|
||||||
},
|
},
|
||||||
plugins: { react },
|
plugins: { react },
|
||||||
rules: {
|
|
||||||
eqeqeq: [2, 'allow-null'],
|
|
||||||
strict: 0,
|
|
||||||
'no-redeclare': [2, { builtinGlobals: false }],
|
|
||||||
'class-methods-use-this': 'off',
|
|
||||||
indent: ['off', 2],
|
|
||||||
'linebreak-style': ['error', 'unix'],
|
|
||||||
quotes: ['error', 'single', { avoidEscape: true, allowTemplateLiterals: true }],
|
|
||||||
semi: ['error', 'always'],
|
|
||||||
'no-console': ['error', { allow: ['warn', 'error'] }],
|
|
||||||
'jsx-quotes': ['error', 'prefer-double'],
|
|
||||||
'react/display-name': 'off',
|
|
||||||
'react/forbid-prop-types': 'off',
|
|
||||||
'react/jsx-closing-bracket-location': 'off',
|
|
||||||
'react/jsx-curly-spacing': 'off',
|
|
||||||
'react/jsx-handler-names': ['off', { eventHandlerPrefix: 'handle', eventHandlerPropPrefix: 'on' }],
|
|
||||||
'react/jsx-indent-props': 'off',
|
|
||||||
'react/jsx-key': 'off',
|
|
||||||
'react/jsx-max-props-per-line': 'off',
|
|
||||||
'react/jsx-no-bind': ['error', { ignoreRefs: true, allowArrowFunctions: true, allowBind: false }],
|
|
||||||
'react/jsx-no-duplicate-props': ['error', { ignoreCase: true }],
|
|
||||||
'react/jsx-no-literals': 'off',
|
|
||||||
'react/jsx-no-undef': 'error',
|
|
||||||
'react/jsx-pascal-case': ['error', { allowAllCaps: true, ignore: [] }],
|
|
||||||
'react/sort-prop-types': ['off', { ignoreCase: true, callbacksLast: false, requiredFirst: false }],
|
|
||||||
'react/jsx-sort-prop-types': 'off',
|
|
||||||
'react/jsx-sort-props': 'off',
|
|
||||||
'react/jsx-uses-react': 'error',
|
|
||||||
'react/jsx-uses-vars': 'error',
|
|
||||||
'react/no-danger': 'warn',
|
|
||||||
'react/no-deprecated': 'error',
|
|
||||||
'react/no-did-mount-set-state': 'error',
|
|
||||||
'react/no-did-update-set-state': 'warn',
|
|
||||||
'react/no-direct-mutation-state': 'off',
|
|
||||||
'react/no-is-mounted': 'error',
|
|
||||||
'react/no-set-state': 'off',
|
|
||||||
'react/no-string-refs': 'warn',
|
|
||||||
'react/no-unknown-property': 'error',
|
|
||||||
'react/prop-types': ['error', { ignore: [], customValidators: [], skipUndeclared: true }],
|
|
||||||
'react/react-in-jsx-scope': 'error',
|
|
||||||
'react/require-extension': 'off',
|
|
||||||
'react/require-render-return': 'error',
|
|
||||||
'react/self-closing-comp': 'warn',
|
|
||||||
'react/sort-comp': 'off',
|
|
||||||
'react/jsx-wrap-multilines': ['warn', { declaration: true, assignment: true, return: true }],
|
|
||||||
'react/wrap-multilines': 'off',
|
|
||||||
'react/jsx-first-prop-new-line': 'off',
|
|
||||||
'react/jsx-equals-spacing': ['warn', 'never'],
|
|
||||||
'react/jsx-no-target-blank': 'error',
|
|
||||||
'react/jsx-filename-extension': ['error', { extensions: ['.jsx'] }],
|
|
||||||
'react/jsx-no-comment-textnodes': 'error',
|
|
||||||
'react/no-comment-textnodes': 'off',
|
|
||||||
'react/no-render-return-value': 'error',
|
|
||||||
'react/require-optimization': ['off', { allowDecorators: [] }],
|
|
||||||
'react/no-find-dom-node': 'warn',
|
|
||||||
'react/forbid-component-props': ['off', { forbid: [] }],
|
|
||||||
'react/no-danger-with-children': 'error',
|
|
||||||
'react/no-unused-prop-types': ['warn', { customValidators: [], skipShapeProps: true }],
|
|
||||||
'react/style-prop-object': 'error',
|
|
||||||
'react/no-children-prop': 'warn',
|
|
||||||
},
|
|
||||||
settings: { react: { version: 'detect' } },
|
settings: { react: { version: 'detect' } },
|
||||||
|
rules: {
|
||||||
|
...js.configs.recommended.rules,
|
||||||
|
'no-console': ['error', { allow: ['warn', 'error'] }],
|
||||||
|
},
|
||||||
},
|
},
|
||||||
|
|
||||||
|
prettier,
|
||||||
];
|
];
|
||||||
|
|||||||
@@ -7,7 +7,7 @@
|
|||||||
content="user-scalable=no, width=device-width, initial-scale=1, maximum-scale=1"
|
content="user-scalable=no, width=device-width, initial-scale=1, maximum-scale=1"
|
||||||
/>
|
/>
|
||||||
<meta name="google" content="notranslate" />
|
<meta name="google" content="notranslate" />
|
||||||
<meta name="apple-mobile-web-app-capable" content="yes" />
|
<meta name="mobile-web-app-capable" content="yes">
|
||||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||||
|
|
||||||
<title>Fredy || Real Estate Finder</title>
|
<title>Fredy || Real Estate Finder</title>
|
||||||
|
|||||||
6
index.js
6
index.js
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -8,10 +8,10 @@ import { checkIfConfigIsAccessible, getProviders, refreshConfig } from './lib/ut
|
|||||||
import * as similarityCache from './lib/services/similarity-check/similarityCache.js';
|
import * as similarityCache from './lib/services/similarity-check/similarityCache.js';
|
||||||
import { runMigrations } from './lib/services/storage/migrations/migrate.js';
|
import { runMigrations } from './lib/services/storage/migrations/migrate.js';
|
||||||
import { ensureDemoUserExists, ensureAdminUserExists } from './lib/services/storage/userStorage.js';
|
import { ensureDemoUserExists, ensureAdminUserExists } from './lib/services/storage/userStorage.js';
|
||||||
import { cleanupDemoAtMidnight } from './lib/services/crons/demoCleanup-cron.js';
|
|
||||||
import { initTrackerCron } from './lib/services/crons/tracker-cron.js';
|
import { initTrackerCron } from './lib/services/crons/tracker-cron.js';
|
||||||
import logger from './lib/services/logger.js';
|
import logger from './lib/services/logger.js';
|
||||||
import { initActiveCheckerCron } from './lib/services/crons/listing-alive-cron.js';
|
import { initActiveCheckerCron } from './lib/services/crons/listing-alive-cron.js';
|
||||||
|
import { initGeocodingCron } from './lib/services/crons/geocoding-cron.js';
|
||||||
import { getSettings } from './lib/services/storage/settingsStorage.js';
|
import { getSettings } from './lib/services/storage/settingsStorage.js';
|
||||||
import SqliteConnection, { computeDbPath } from './lib/services/storage/SqliteConnection.js';
|
import SqliteConnection, { computeDbPath } from './lib/services/storage/SqliteConnection.js';
|
||||||
import { initJobExecutionService } from './lib/services/jobs/jobExecutionService.js';
|
import { initJobExecutionService } from './lib/services/jobs/jobExecutionService.js';
|
||||||
@@ -53,7 +53,6 @@ await import('./lib/api/api.js');
|
|||||||
|
|
||||||
if (settings.demoMode) {
|
if (settings.demoMode) {
|
||||||
logger.info('Running in demo mode');
|
logger.info('Running in demo mode');
|
||||||
cleanupDemoAtMidnight();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
ensureAdminUserExists();
|
ensureAdminUserExists();
|
||||||
@@ -61,6 +60,7 @@ ensureDemoUserExists();
|
|||||||
await initTrackerCron();
|
await initTrackerCron();
|
||||||
//do not wait for this to finish, let it run in the background
|
//do not wait for this to finish, let it run in the background
|
||||||
initActiveCheckerCron();
|
initActiveCheckerCron();
|
||||||
|
initGeocodingCron();
|
||||||
|
|
||||||
logger.info(`Started Fredy successfully. Ui can be accessed via http://localhost:${settings.port}`);
|
logger.info(`Started Fredy successfully. Ui can be accessed via http://localhost:${settings.port}`);
|
||||||
|
|
||||||
|
|||||||
@@ -1,14 +1,24 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { NoNewListingsWarning } from './errors.js';
|
import { NoNewListingsWarning } from './errors.js';
|
||||||
import { storeListings, getKnownListingHashesForJobAndProvider } from './services/storage/listingsStorage.js';
|
import {
|
||||||
|
storeListings,
|
||||||
|
getKnownListingHashesForJobAndProvider,
|
||||||
|
deleteListingsById,
|
||||||
|
} from './services/storage/listingsStorage.js';
|
||||||
|
import { getJob } from './services/storage/jobStorage.js';
|
||||||
import * as notify from './notification/notify.js';
|
import * as notify from './notification/notify.js';
|
||||||
import Extractor from './services/extractor/extractor.js';
|
import Extractor from './services/extractor/extractor.js';
|
||||||
import urlModifier from './services/queryStringMutator.js';
|
import urlModifier from './services/queryStringMutator.js';
|
||||||
import logger from './services/logger.js';
|
import logger from './services/logger.js';
|
||||||
|
import { geocodeAddress } from './services/geocoding/geoCodingService.js';
|
||||||
|
import { distanceMeters } from './services/listings/distanceCalculator.js';
|
||||||
|
import { getUserSettings } from './services/storage/settingsStorage.js';
|
||||||
|
import { updateListingDistance } from './services/storage/listingsStorage.js';
|
||||||
|
import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @typedef {Object} Listing
|
* @typedef {Object} Listing
|
||||||
@@ -53,18 +63,21 @@ class FredyPipelineExecutioner {
|
|||||||
* @param {(raw:any)=>Listing} providerConfig.normalize Function to convert raw scraped data into a Listing shape.
|
* @param {(raw:any)=>Listing} providerConfig.normalize Function to convert raw scraped data into a Listing shape.
|
||||||
* @param {(listing:Listing)=>boolean} providerConfig.filter Function to filter out unwanted listings.
|
* @param {(listing:Listing)=>boolean} providerConfig.filter Function to filter out unwanted listings.
|
||||||
* @param {(url:string, waitForSelector?:string)=>Promise<void>|Promise<Listing[]>} [providerConfig.getListings] Optional override to fetch listings.
|
* @param {(url:string, waitForSelector?:string)=>Promise<void>|Promise<Listing[]>} [providerConfig.getListings] Optional override to fetch listings.
|
||||||
*
|
|
||||||
* @param {Object} notificationConfig Notification configuration passed to notification adapters.
|
* @param {Object} notificationConfig Notification configuration passed to notification adapters.
|
||||||
|
* @param {Object} spatialFilter Optional spatial filter configuration.
|
||||||
* @param {string} providerId The ID of the provider currently in use.
|
* @param {string} providerId The ID of the provider currently in use.
|
||||||
* @param {string} jobKey Key of the job that is currently running (from within the config).
|
* @param {string} jobKey Key of the job that is currently running (from within the config).
|
||||||
* @param {SimilarityCache} similarityCache Cache instance for checking similar entries.
|
* @param {SimilarityCache} similarityCache Cache instance for checking similar entries.
|
||||||
|
* @param browser
|
||||||
*/
|
*/
|
||||||
constructor(providerConfig, notificationConfig, providerId, jobKey, similarityCache) {
|
constructor(providerConfig, notificationConfig, spatialFilter, providerId, jobKey, similarityCache, browser) {
|
||||||
this._providerConfig = providerConfig;
|
this._providerConfig = providerConfig;
|
||||||
this._notificationConfig = notificationConfig;
|
this._notificationConfig = notificationConfig;
|
||||||
|
this._spatialFilter = spatialFilter;
|
||||||
this._providerId = providerId;
|
this._providerId = providerId;
|
||||||
this._jobKey = jobKey;
|
this._jobKey = jobKey;
|
||||||
this._similarityCache = similarityCache;
|
this._similarityCache = similarityCache;
|
||||||
|
this._browser = browser;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -79,12 +92,75 @@ class FredyPipelineExecutioner {
|
|||||||
.then(this._normalize.bind(this))
|
.then(this._normalize.bind(this))
|
||||||
.then(this._filter.bind(this))
|
.then(this._filter.bind(this))
|
||||||
.then(this._findNew.bind(this))
|
.then(this._findNew.bind(this))
|
||||||
|
.then(this._geocode.bind(this))
|
||||||
.then(this._save.bind(this))
|
.then(this._save.bind(this))
|
||||||
|
.then(this._calculateDistance.bind(this))
|
||||||
.then(this._filterBySimilarListings.bind(this))
|
.then(this._filterBySimilarListings.bind(this))
|
||||||
|
.then(this._filterByArea.bind(this))
|
||||||
.then(this._notify.bind(this))
|
.then(this._notify.bind(this))
|
||||||
.catch(this._handleError.bind(this));
|
.catch(this._handleError.bind(this));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Geocode new listings.
|
||||||
|
*
|
||||||
|
* @param {Listing[]} newListings New listings to geocode.
|
||||||
|
* @returns {Promise<Listing[]>} Resolves with the listings (potentially with added coordinates).
|
||||||
|
*/
|
||||||
|
async _geocode(newListings) {
|
||||||
|
for (const listing of newListings) {
|
||||||
|
if (listing.address) {
|
||||||
|
const coords = await geocodeAddress(listing.address);
|
||||||
|
if (coords) {
|
||||||
|
listing.latitude = coords.lat;
|
||||||
|
listing.longitude = coords.lng;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return newListings;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Filter listings by area using the provider's area filter if available.
|
||||||
|
* Only filters if areaFilter is set on the provider AND the listing has coordinates.
|
||||||
|
*
|
||||||
|
* @param {Listing[]} newListings New listings to filter by area.
|
||||||
|
* @returns {Promise<Listing[]>} Resolves with listings that are within the area (or not filtered if no area is set).
|
||||||
|
*/
|
||||||
|
_filterByArea(newListings) {
|
||||||
|
const polygonFeatures = this._spatialFilter?.features?.filter((f) => f.geometry?.type === 'Polygon');
|
||||||
|
|
||||||
|
// If no area filter is set, return all listings
|
||||||
|
if (!polygonFeatures?.length) {
|
||||||
|
return newListings;
|
||||||
|
}
|
||||||
|
|
||||||
|
const filteredIds = [];
|
||||||
|
// Filter listings by area - keep only those within the polygon
|
||||||
|
const keptListings = newListings.filter((listing) => {
|
||||||
|
// If listing doesn't have coordinates, keep it (don't filter out)
|
||||||
|
if (listing.latitude == null || listing.longitude == null) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if the point is inside the polygons
|
||||||
|
const point = [listing.longitude, listing.latitude]; // GeoJSON format: [lon, lat]
|
||||||
|
const isInPolygon = polygonFeatures.some((feature) => booleanPointInPolygon(point, feature));
|
||||||
|
|
||||||
|
if (!isInPolygon) {
|
||||||
|
filteredIds.push(listing.id);
|
||||||
|
}
|
||||||
|
|
||||||
|
return isInPolygon;
|
||||||
|
});
|
||||||
|
|
||||||
|
if (filteredIds.length > 0) {
|
||||||
|
deleteListingsById(filteredIds);
|
||||||
|
}
|
||||||
|
|
||||||
|
return keptListings;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Fetch listings from the provider, using the default Extractor flow unless
|
* Fetch listings from the provider, using the default Extractor flow unless
|
||||||
* a provider-specific getListings override is supplied.
|
* a provider-specific getListings override is supplied.
|
||||||
@@ -93,7 +169,7 @@ class FredyPipelineExecutioner {
|
|||||||
* @returns {Promise<Listing[]>} Resolves with an array of listings (empty when none found).
|
* @returns {Promise<Listing[]>} Resolves with an array of listings (empty when none found).
|
||||||
*/
|
*/
|
||||||
_getListings(url) {
|
_getListings(url) {
|
||||||
const extractor = new Extractor();
|
const extractor = new Extractor({ ...this._providerConfig.puppeteerOptions, browser: this._browser });
|
||||||
return new Promise((resolve, reject) => {
|
return new Promise((resolve, reject) => {
|
||||||
extractor
|
extractor
|
||||||
.execute(url, this._providerConfig.waitForSelector)
|
.execute(url, this._providerConfig.waitForSelector)
|
||||||
@@ -180,6 +256,42 @@ class FredyPipelineExecutioner {
|
|||||||
return newListings;
|
return newListings;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Calculate distance for new listings.
|
||||||
|
*
|
||||||
|
* @param {Listing[]} listings
|
||||||
|
* @returns {Listing[]}
|
||||||
|
* @private
|
||||||
|
*/
|
||||||
|
_calculateDistance(listings) {
|
||||||
|
if (listings.length === 0) return [];
|
||||||
|
|
||||||
|
const job = getJob(this._jobKey);
|
||||||
|
const userId = job?.userId;
|
||||||
|
|
||||||
|
if (userId == null || typeof userId !== 'string') {
|
||||||
|
logger.debug('Skipping distance calculation: userId is missing or invalid');
|
||||||
|
return listings;
|
||||||
|
}
|
||||||
|
|
||||||
|
const userSettings = getUserSettings(userId);
|
||||||
|
const homeAddress = userSettings?.home_address;
|
||||||
|
|
||||||
|
if (!homeAddress || !homeAddress.coords) {
|
||||||
|
return listings;
|
||||||
|
}
|
||||||
|
|
||||||
|
const { lat, lng } = homeAddress.coords;
|
||||||
|
for (const listing of listings) {
|
||||||
|
if (listing.latitude != null && listing.longitude != null) {
|
||||||
|
const dist = distanceMeters(lat, lng, listing.latitude, listing.longitude);
|
||||||
|
updateListingDistance(listing.id, dist);
|
||||||
|
listing.distance_to_destination = dist;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return listings;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Remove listings that are similar to already known entries according to the similarity cache.
|
* Remove listings that are similar to already known entries according to the similarity cache.
|
||||||
* Adds the remaining listings to the cache.
|
* Adds the remaining listings to the cache.
|
||||||
@@ -188,7 +300,8 @@ class FredyPipelineExecutioner {
|
|||||||
* @returns {Listing[]} Listings considered unique enough to keep.
|
* @returns {Listing[]} Listings considered unique enough to keep.
|
||||||
*/
|
*/
|
||||||
_filterBySimilarListings(listings) {
|
_filterBySimilarListings(listings) {
|
||||||
return listings.filter((listing) => {
|
const filteredIds = [];
|
||||||
|
const keptListings = listings.filter((listing) => {
|
||||||
const similar = this._similarityCache.checkAndAddEntry({
|
const similar = this._similarityCache.checkAndAddEntry({
|
||||||
title: listing.title,
|
title: listing.title,
|
||||||
address: listing.address,
|
address: listing.address,
|
||||||
@@ -198,9 +311,16 @@ class FredyPipelineExecutioner {
|
|||||||
logger.debug(
|
logger.debug(
|
||||||
`Filtering similar entry for title '${listing.title}' and address '${listing.address}' (Provider: '${this._providerId}')`,
|
`Filtering similar entry for title '${listing.title}' and address '${listing.address}' (Provider: '${this._providerId}')`,
|
||||||
);
|
);
|
||||||
|
filteredIds.push(listing.id);
|
||||||
}
|
}
|
||||||
return !similar;
|
return !similar;
|
||||||
});
|
});
|
||||||
|
|
||||||
|
if (filteredIds.length > 0) {
|
||||||
|
deleteListingsById(filteredIds);
|
||||||
|
}
|
||||||
|
|
||||||
|
return keptListings;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
10
lib/TRACKING_POIS.js
Normal file
10
lib/TRACKING_POIS.js
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
export const TRACKING_POIS = {
|
||||||
|
DISTANCE_ADDRESS_ENTERED: 'DISTANCE_ADDRESS_ENTERED',
|
||||||
|
WELCOME_FINISHED: 'WELCOME_FINISHED',
|
||||||
|
WELCOME_SKIPPED: 'WELCOME_SKIPPED',
|
||||||
|
};
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -10,6 +10,7 @@ import { providerRouter } from './routes/providerRouter.js';
|
|||||||
import { versionRouter } from './routes/versionRouter.js';
|
import { versionRouter } from './routes/versionRouter.js';
|
||||||
import { loginRouter } from './routes/loginRoute.js';
|
import { loginRouter } from './routes/loginRoute.js';
|
||||||
import { userRouter } from './routes/userRoute.js';
|
import { userRouter } from './routes/userRoute.js';
|
||||||
|
import { userSettingsRouter } from './routes/userSettingsRoute.js';
|
||||||
import { jobRouter } from './routes/jobRouter.js';
|
import { jobRouter } from './routes/jobRouter.js';
|
||||||
import bodyParser from 'body-parser';
|
import bodyParser from 'body-parser';
|
||||||
import restana from 'restana';
|
import restana from 'restana';
|
||||||
@@ -20,9 +21,10 @@ import { demoRouter } from './routes/demoRouter.js';
|
|||||||
import logger from '../services/logger.js';
|
import logger from '../services/logger.js';
|
||||||
import { listingsRouter } from './routes/listingsRouter.js';
|
import { listingsRouter } from './routes/listingsRouter.js';
|
||||||
import { getSettings } from '../services/storage/settingsStorage.js';
|
import { getSettings } from '../services/storage/settingsStorage.js';
|
||||||
import { featureRouter } from './routes/featureRouter.js';
|
|
||||||
import { dashboardRouter } from './routes/dashboardRouter.js';
|
import { dashboardRouter } from './routes/dashboardRouter.js';
|
||||||
import { backupRouter } from './routes/backupRouter.js';
|
import { backupRouter } from './routes/backupRouter.js';
|
||||||
|
import { trackingRouter } from './routes/trackingRoute.js';
|
||||||
|
import { registerMcpRoutes } from '../../mcp/mcpHttpRoute.js';
|
||||||
const service = restana();
|
const service = restana();
|
||||||
const staticService = files(path.join(getDirName(), '../ui/public'));
|
const staticService = files(path.join(getDirName(), '../ui/public'));
|
||||||
const PORT = (await getSettings()).port || 9998;
|
const PORT = (await getSettings()).port || 9998;
|
||||||
@@ -35,7 +37,8 @@ service.use('/api/jobs', authInterceptor());
|
|||||||
service.use('/api/version', authInterceptor());
|
service.use('/api/version', authInterceptor());
|
||||||
service.use('/api/listings', authInterceptor());
|
service.use('/api/listings', authInterceptor());
|
||||||
service.use('/api/dashboard', authInterceptor());
|
service.use('/api/dashboard', authInterceptor());
|
||||||
service.use('/api/features', authInterceptor());
|
service.use('/api/user/settings', authInterceptor());
|
||||||
|
service.use('/api/tracking', authInterceptor());
|
||||||
|
|
||||||
// /admin can only be accessed when user is having admin permissions
|
// /admin can only be accessed when user is having admin permissions
|
||||||
service.use('/api/admin', adminInterceptor());
|
service.use('/api/admin', adminInterceptor());
|
||||||
@@ -44,15 +47,19 @@ service.use('/api/admin/generalSettings', generalSettingsRouter);
|
|||||||
service.use('/api/admin/backup', backupRouter);
|
service.use('/api/admin/backup', backupRouter);
|
||||||
service.use('/api/jobs/provider', providerRouter);
|
service.use('/api/jobs/provider', providerRouter);
|
||||||
service.use('/api/admin/users', userRouter);
|
service.use('/api/admin/users', userRouter);
|
||||||
|
service.use('/api/user/settings', userSettingsRouter);
|
||||||
service.use('/api/version', versionRouter);
|
service.use('/api/version', versionRouter);
|
||||||
service.use('/api/jobs', jobRouter);
|
service.use('/api/jobs', jobRouter);
|
||||||
service.use('/api/login', loginRouter);
|
service.use('/api/login', loginRouter);
|
||||||
service.use('/api/listings', listingsRouter);
|
service.use('/api/listings', listingsRouter);
|
||||||
service.use('/api/features', featureRouter);
|
|
||||||
service.use('/api/dashboard', dashboardRouter);
|
service.use('/api/dashboard', dashboardRouter);
|
||||||
|
service.use('/api/tracking', trackingRouter);
|
||||||
//this route is unsecured intentionally as it is being queried from the login page
|
//this route is unsecured intentionally as it is being queried from the login page
|
||||||
service.use('/api/demo', demoRouter);
|
service.use('/api/demo', demoRouter);
|
||||||
|
|
||||||
|
// MCP Streamable HTTP endpoint (secured via Bearer token, not cookie-session)
|
||||||
|
registerMcpRoutes(service);
|
||||||
|
|
||||||
service.start(PORT).then(() => {
|
service.start(PORT).then(() => {
|
||||||
logger.debug(`Started API service on port ${PORT}`);
|
logger.debug(`Started API service on port ${PORT}`);
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,17 +0,0 @@
|
|||||||
/*
|
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
|
||||||
*/
|
|
||||||
|
|
||||||
import restana from 'restana';
|
|
||||||
import getFeatures from '../../features.js';
|
|
||||||
const service = restana();
|
|
||||||
const featureRouter = service.newRouter();
|
|
||||||
|
|
||||||
featureRouter.get('/', async (req, res) => {
|
|
||||||
const features = getFeatures();
|
|
||||||
res.body = Object.assign({}, { features });
|
|
||||||
res.send();
|
|
||||||
});
|
|
||||||
|
|
||||||
export { featureRouter };
|
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -11,10 +11,13 @@ import logger from '../../services/logger.js';
|
|||||||
import { bus } from '../../services/events/event-bus.js';
|
import { bus } from '../../services/events/event-bus.js';
|
||||||
import { isRunning as isJobRunning } from '../../services/jobs/run-state.js';
|
import { isRunning as isJobRunning } from '../../services/jobs/run-state.js';
|
||||||
import { addClient as addSseClient, removeClient } from '../../services/sse/sse-broker.js';
|
import { addClient as addSseClient, removeClient } from '../../services/sse/sse-broker.js';
|
||||||
|
import { getSettings } from '../../services/storage/settingsStorage.js';
|
||||||
|
|
||||||
const service = restana();
|
const service = restana();
|
||||||
const jobRouter = service.newRouter();
|
const jobRouter = service.newRouter();
|
||||||
|
|
||||||
|
const DEMO_JOB_NAME = 'Demo-Job';
|
||||||
|
|
||||||
function doesJobBelongsToUser(job, req) {
|
function doesJobBelongsToUser(job, req) {
|
||||||
const userId = req.session.currentUser;
|
const userId = req.session.currentUser;
|
||||||
if (userId == null) {
|
if (userId == null) {
|
||||||
@@ -160,7 +163,17 @@ jobRouter.post('/:jobId/run', async (req, res) => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
jobRouter.post('/', async (req, res) => {
|
jobRouter.post('/', async (req, res) => {
|
||||||
const { provider, notificationAdapter, name, blacklist = [], jobId, enabled, shareWithUsers = [] } = req.body;
|
const {
|
||||||
|
provider,
|
||||||
|
notificationAdapter,
|
||||||
|
name,
|
||||||
|
blacklist = [],
|
||||||
|
jobId,
|
||||||
|
enabled,
|
||||||
|
shareWithUsers = [],
|
||||||
|
spatialFilter = null,
|
||||||
|
} = req.body;
|
||||||
|
const settings = await getSettings();
|
||||||
try {
|
try {
|
||||||
let jobFromDb = jobStorage.getJob(jobId);
|
let jobFromDb = jobStorage.getJob(jobId);
|
||||||
|
|
||||||
@@ -169,6 +182,11 @@ jobRouter.post('/', async (req, res) => {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (settings.demoMode && jobFromDb && jobFromDb.name === DEMO_JOB_NAME) {
|
||||||
|
res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
jobStorage.upsertJob({
|
jobStorage.upsertJob({
|
||||||
userId: req.session.currentUser,
|
userId: req.session.currentUser,
|
||||||
jobId,
|
jobId,
|
||||||
@@ -178,6 +196,7 @@ jobRouter.post('/', async (req, res) => {
|
|||||||
provider,
|
provider,
|
||||||
notificationAdapter,
|
notificationAdapter,
|
||||||
shareWithUsers,
|
shareWithUsers,
|
||||||
|
spatialFilter,
|
||||||
});
|
});
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
res.send(new Error(error));
|
res.send(new Error(error));
|
||||||
@@ -188,8 +207,14 @@ jobRouter.post('/', async (req, res) => {
|
|||||||
|
|
||||||
jobRouter.delete('', async (req, res) => {
|
jobRouter.delete('', async (req, res) => {
|
||||||
const { jobId } = req.body;
|
const { jobId } = req.body;
|
||||||
|
const settings = await getSettings();
|
||||||
try {
|
try {
|
||||||
const job = jobStorage.getJob(jobId);
|
const job = jobStorage.getJob(jobId);
|
||||||
|
if (settings.demoMode && job.name === DEMO_JOB_NAME) {
|
||||||
|
res.send(new Error('Sorry, but you cannot remove the Demo Job ;)'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
if (!doesJobBelongsToUser(job, req)) {
|
if (!doesJobBelongsToUser(job, req)) {
|
||||||
res.send(new Error('You are trying to remove a job that is not associated to your user'));
|
res.send(new Error('You are trying to remove a job that is not associated to your user'));
|
||||||
} else {
|
} else {
|
||||||
@@ -204,8 +229,15 @@ jobRouter.delete('', async (req, res) => {
|
|||||||
jobRouter.put('/:jobId/status', async (req, res) => {
|
jobRouter.put('/:jobId/status', async (req, res) => {
|
||||||
const { status } = req.body;
|
const { status } = req.body;
|
||||||
const { jobId } = req.params;
|
const { jobId } = req.params;
|
||||||
|
const settings = await getSettings();
|
||||||
try {
|
try {
|
||||||
const job = jobStorage.getJob(jobId);
|
const job = jobStorage.getJob(jobId);
|
||||||
|
|
||||||
|
if (settings.demoMode && job.name === DEMO_JOB_NAME) {
|
||||||
|
res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
if (!doesJobBelongsToUser(job, req)) {
|
if (!doesJobBelongsToUser(job, req)) {
|
||||||
res.send(new Error('You are trying change a job that is not associated to your user'));
|
res.send(new Error('You are trying change a job that is not associated to your user'));
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -10,6 +10,7 @@ import { isAdmin as isAdminFn } from '../security.js';
|
|||||||
import logger from '../../services/logger.js';
|
import logger from '../../services/logger.js';
|
||||||
import { nullOrEmpty } from '../../utils.js';
|
import { nullOrEmpty } from '../../utils.js';
|
||||||
import { getJobs } from '../../services/storage/jobStorage.js';
|
import { getJobs } from '../../services/storage/jobStorage.js';
|
||||||
|
import { getSettings } from '../../services/storage/settingsStorage.js';
|
||||||
|
|
||||||
const service = restana();
|
const service = restana();
|
||||||
|
|
||||||
@@ -63,6 +64,29 @@ listingsRouter.get('/table', async (req, res) => {
|
|||||||
res.send();
|
res.send();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
listingsRouter.get('/map', async (req, res) => {
|
||||||
|
const { jobId } = req.query || {};
|
||||||
|
|
||||||
|
res.body = listingStorage.getListingsForMap({
|
||||||
|
jobId: nullOrEmpty(jobId) ? null : jobId,
|
||||||
|
userId: req.session.currentUser,
|
||||||
|
isAdmin: isAdminFn(req),
|
||||||
|
});
|
||||||
|
res.send();
|
||||||
|
});
|
||||||
|
|
||||||
|
listingsRouter.get('/:listingId', async (req, res) => {
|
||||||
|
const { listingId } = req.params;
|
||||||
|
const listing = listingStorage.getListingById(listingId, req.session.currentUser, isAdminFn(req));
|
||||||
|
if (!listing) {
|
||||||
|
res.statusCode = 404;
|
||||||
|
res.body = { message: 'Listing not found' };
|
||||||
|
return res.send();
|
||||||
|
}
|
||||||
|
res.body = listing;
|
||||||
|
res.send();
|
||||||
|
});
|
||||||
|
|
||||||
// Toggle watch state for the current user on a listing
|
// Toggle watch state for the current user on a listing
|
||||||
listingsRouter.post('/watch', async (req, res) => {
|
listingsRouter.post('/watch', async (req, res) => {
|
||||||
try {
|
try {
|
||||||
@@ -83,9 +107,15 @@ listingsRouter.post('/watch', async (req, res) => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
listingsRouter.delete('/job', async (req, res) => {
|
listingsRouter.delete('/job', async (req, res) => {
|
||||||
const { jobId } = req.body;
|
const { jobId, hardDelete = false } = req.body;
|
||||||
|
const settings = await getSettings();
|
||||||
try {
|
try {
|
||||||
listingStorage.deleteListingsByJobId(jobId);
|
if (settings.demoMode) {
|
||||||
|
res.send(new Error('Sorry, but you cannot remove listings in demo mode ;)'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
listingStorage.deleteListingsByJobId(jobId, hardDelete);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
res.send(new Error(error));
|
res.send(new Error(error));
|
||||||
logger.error(error);
|
logger.error(error);
|
||||||
@@ -94,10 +124,10 @@ listingsRouter.delete('/job', async (req, res) => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
listingsRouter.delete('/', async (req, res) => {
|
listingsRouter.delete('/', async (req, res) => {
|
||||||
const { ids } = req.body;
|
const { ids, hardDelete = false } = req.body;
|
||||||
try {
|
try {
|
||||||
if (Array.isArray(ids) && ids.length > 0) {
|
if (Array.isArray(ids) && ids.length > 0) {
|
||||||
listingStorage.deleteListingsById(ids);
|
listingStorage.deleteListingsById(ids, hardDelete);
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
res.send(new Error(error));
|
res.send(new Error(error));
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import fs from 'fs';
|
import fs from 'fs';
|
||||||
import restana from 'restana';
|
import restana from 'restana';
|
||||||
|
import logger from '../../services/logger.js';
|
||||||
|
|
||||||
const service = restana();
|
const service = restana();
|
||||||
const notificationAdapterRouter = service.newRouter();
|
const notificationAdapterRouter = service.newRouter();
|
||||||
const notificationAdapterList = fs.readdirSync('./lib//notification/adapter').filter((file) => file.endsWith('.js'));
|
const notificationAdapterList = fs.readdirSync('./lib//notification/adapter').filter((file) => file.endsWith('.js'));
|
||||||
@@ -34,11 +36,14 @@ notificationAdapterRouter.post('/try', async (req, res) => {
|
|||||||
serviceName: 'TestCall',
|
serviceName: 'TestCall',
|
||||||
newListings: [
|
newListings: [
|
||||||
{
|
{
|
||||||
price: '42 €',
|
address: 'Heidestrasse 17, 51147 Köln',
|
||||||
title: 'This is a test listing',
|
description: exampleDescription,
|
||||||
address: 'some address',
|
id: '1',
|
||||||
size: '666 2m',
|
imageUrl: 'https://placehold.co/600x400/png',
|
||||||
link: 'https://www.orange-coding.net',
|
price: '1.000 €',
|
||||||
|
size: '76 m²',
|
||||||
|
title: 'Stilvolle gepflegte 3-Raum-Wohnung mit gehobener Innenausstattung',
|
||||||
|
url: 'https://www.orange-coding.net',
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
notificationConfig,
|
notificationConfig,
|
||||||
@@ -46,6 +51,7 @@ notificationAdapterRouter.post('/try', async (req, res) => {
|
|||||||
});
|
});
|
||||||
res.send();
|
res.send();
|
||||||
} catch (Exception) {
|
} catch (Exception) {
|
||||||
|
logger.error('Error during notification adapter test:', Exception);
|
||||||
res.send(new Error(Exception));
|
res.send(new Error(Exception));
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -54,3 +60,51 @@ notificationAdapterRouter.get('/', async (req, res) => {
|
|||||||
res.send();
|
res.send();
|
||||||
});
|
});
|
||||||
export { notificationAdapterRouter };
|
export { notificationAdapterRouter };
|
||||||
|
|
||||||
|
const exampleDescription = `
|
||||||
|
Wohnungstyp: Etagenwohnung
|
||||||
|
Nutzfläche: 76 m²
|
||||||
|
Etage: 2 von 3
|
||||||
|
Schlafzimmer: 1
|
||||||
|
Badezimmer: 1
|
||||||
|
Bezugsfrei ab: 1.4.2026
|
||||||
|
Haustiere: Nein
|
||||||
|
Garage/Stellplatz: Tiefgarage
|
||||||
|
Anzahl Garage/Stellplatz: 1
|
||||||
|
Kaltmiete (zzgl. Nebenkosten): 1.000 €
|
||||||
|
Preis/m²: 13,16 €/m²
|
||||||
|
Nebenkosten: 230 €
|
||||||
|
Heizkosten in Nebenkosten enthalten: Ja
|
||||||
|
Gesamtmiete: 1.230 €
|
||||||
|
Kaution: 3.000,00
|
||||||
|
Preis pro Parkfläche: 60 €
|
||||||
|
Baujahr: 2000
|
||||||
|
Objektzustand: Modernisiert
|
||||||
|
Qualität der Ausstattung: Gehoben
|
||||||
|
Heizungsart: Fernwärme
|
||||||
|
Energieausweistyp: Verbrauchsausweis
|
||||||
|
Energieausweis: liegt vor
|
||||||
|
Endenergieverbrauch: 72 kWh/(m²∙a)
|
||||||
|
Baujahr laut Energieausweis: 2000
|
||||||
|
|
||||||
|
Diese moderne 3-Zimmer-Wohnung liegt direkt neben einem Park und nur wenige Minuten von der S-Bahn-Haltestelle entfernt. Das Stadtzentrum sowie Freizeiteinrichtungen sind 1,5 km entfernt.
|
||||||
|
|
||||||
|
Die Wohnung ist ideal für Paare oder kleine Familien geeignet.
|
||||||
|
|
||||||
|
Ausstattung:
|
||||||
|
- neuer hochwertiger Bodenbelag (Holzoptik) in allen Räumen außer Bad/Küche
|
||||||
|
- sonniger Balkon (Süd)
|
||||||
|
- Tiefgaragenstellplatz
|
||||||
|
- Kellerabteil
|
||||||
|
- gepflegtes Mehrfamilienhaus
|
||||||
|
|
||||||
|
Die Küche ist vom Mieter nach eigenen Wünschen einzurichten.
|
||||||
|
|
||||||
|
Vermietung direkt vom Eigentümer - provisionsfrei!
|
||||||
|
|
||||||
|
Lage:
|
||||||
|
• Park: 1 Minute zu Fuß
|
||||||
|
• S-Bahn Station: 2 Minuten zu Fuß
|
||||||
|
• Supermärkte, Restaurants, täglicher Bedarf in der Nähe
|
||||||
|
• Gute Anbindung Richtung Großstadt und Flughafen
|
||||||
|
`;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
37
lib/api/routes/trackingRoute.js
Normal file
37
lib/api/routes/trackingRoute.js
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import restana from 'restana';
|
||||||
|
import { trackPoi } from '../../services/tracking/Tracker.js';
|
||||||
|
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
|
||||||
|
import logger from '../../services/logger.js';
|
||||||
|
|
||||||
|
const service = restana();
|
||||||
|
const trackingRouter = service.newRouter();
|
||||||
|
|
||||||
|
trackingRouter.get('/trackingPois', async (req, res) => {
|
||||||
|
res.body = TRACKING_POIS;
|
||||||
|
res.send();
|
||||||
|
});
|
||||||
|
|
||||||
|
trackingRouter.post('/poi', async (req, res) => {
|
||||||
|
const { poi } = req.body;
|
||||||
|
if (!poi) {
|
||||||
|
res.statusCode = 400;
|
||||||
|
res.send({ error: 'Feature name is required' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
await trackPoi(poi);
|
||||||
|
res.send({ success: true });
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error tracking feature', error);
|
||||||
|
res.statusCode = 500;
|
||||||
|
res.send({ error: error.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
export { trackingRouter };
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
121
lib/api/routes/userSettingsRoute.js
Normal file
121
lib/api/routes/userSettingsRoute.js
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import restana from 'restana';
|
||||||
|
import SqliteConnection from '../../services/storage/SqliteConnection.js';
|
||||||
|
import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
|
||||||
|
import { resetGeocoordinatesAndDistanceForUser } from '../../services/storage/listingsStorage.js';
|
||||||
|
import { geocodeAddress } from '../../services/geocoding/geoCodingService.js';
|
||||||
|
import { autocompleteAddress } from '../../services/geocoding/autocompleteService.js';
|
||||||
|
import { fromJson } from '../../utils.js';
|
||||||
|
import { trackPoi } from '../../services/tracking/Tracker.js';
|
||||||
|
import { TRACKING_POIS } from '../../TRACKING_POIS.js';
|
||||||
|
import logger from '../../services/logger.js';
|
||||||
|
import { runGeoCordTask } from '../../services/crons/geocoding-cron.js';
|
||||||
|
|
||||||
|
const service = restana();
|
||||||
|
const userSettingsRouter = service.newRouter();
|
||||||
|
|
||||||
|
userSettingsRouter.get('/', async (req, res) => {
|
||||||
|
const userId = req.session.currentUser;
|
||||||
|
const rows = SqliteConnection.query('SELECT name, value FROM settings WHERE user_id = @userId', { userId });
|
||||||
|
const settings = {};
|
||||||
|
for (const r of rows) {
|
||||||
|
settings[r.name] = fromJson(r.value, null);
|
||||||
|
}
|
||||||
|
res.body = settings;
|
||||||
|
res.send();
|
||||||
|
});
|
||||||
|
|
||||||
|
userSettingsRouter.get('/autocomplete', async (req, res) => {
|
||||||
|
const { q } = req.query;
|
||||||
|
try {
|
||||||
|
const results = await autocompleteAddress(q);
|
||||||
|
res.body = results;
|
||||||
|
res.send();
|
||||||
|
} catch (error) {
|
||||||
|
res.statusCode = 500;
|
||||||
|
res.send({ error: error.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
userSettingsRouter.post('/home-address', async (req, res) => {
|
||||||
|
const userId = req.session.currentUser;
|
||||||
|
const { home_address } = req.body;
|
||||||
|
const settings = await getSettings();
|
||||||
|
|
||||||
|
if (settings.demoMode) {
|
||||||
|
res.send(new Error('In demo mode, it is not allowed to change the home address.'));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (home_address) {
|
||||||
|
await trackPoi(TRACKING_POIS.DISTANCE_ADDRESS_ENTERED);
|
||||||
|
const coords = await geocodeAddress(home_address);
|
||||||
|
if (coords && coords.lat !== -1) {
|
||||||
|
upsertSettings({ home_address: { address: home_address, coords } }, userId);
|
||||||
|
resetGeocoordinatesAndDistanceForUser(userId);
|
||||||
|
//we do NOT wait for this to finish, as we don't want to block the response
|
||||||
|
runGeoCordTask();
|
||||||
|
res.send({ success: true, coords });
|
||||||
|
} else {
|
||||||
|
res.statusCode = 400;
|
||||||
|
res.send({ error: 'Could not geocode address' });
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
upsertSettings({ home_address: null }, userId);
|
||||||
|
res.send({ success: true });
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error updating home address settings', error);
|
||||||
|
res.statusCode = 500;
|
||||||
|
res.send({ error: error.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
userSettingsRouter.post('/news-hash', async (req, res) => {
|
||||||
|
const userId = req.session.currentUser;
|
||||||
|
const { news_hash } = req.body;
|
||||||
|
|
||||||
|
const globalSettings = await getSettings();
|
||||||
|
if (globalSettings.demoMode) {
|
||||||
|
res.statusCode = 403;
|
||||||
|
res.send({ error: 'In demo mode, it is not allowed to change settings.' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
upsertSettings({ news_hash }, userId);
|
||||||
|
res.send({ success: true });
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error updating news hash', error);
|
||||||
|
res.statusCode = 500;
|
||||||
|
res.send({ error: error.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
userSettingsRouter.post('/immoscout-details', async (req, res) => {
|
||||||
|
const userId = req.session.currentUser;
|
||||||
|
const { immoscout_details } = req.body;
|
||||||
|
|
||||||
|
const globalSettings = await getSettings();
|
||||||
|
if (globalSettings.demoMode) {
|
||||||
|
res.statusCode = 403;
|
||||||
|
res.send({ error: 'In demo mode, it is not allowed to change settings.' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
upsertSettings({ immoscout_details: !!immoscout_details }, userId);
|
||||||
|
res.send({ success: true });
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error updating immoscout details setting', error);
|
||||||
|
res.statusCode = 500;
|
||||||
|
res.send({ error: error.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
export { userSettingsRouter };
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,14 +0,0 @@
|
|||||||
/*
|
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
|
||||||
*/
|
|
||||||
|
|
||||||
const FEATURES = {
|
|
||||||
WATCHLIST_MANAGEMENT: false,
|
|
||||||
};
|
|
||||||
|
|
||||||
export default function getFeatures() {
|
|
||||||
return {
|
|
||||||
...FEATURES,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -16,8 +16,8 @@ const mapListing = (listing) => ({
|
|||||||
url: listing.link,
|
url: listing.link,
|
||||||
});
|
});
|
||||||
|
|
||||||
export const send = ({ serviceName, newListings, notificationConfig, jobKey }) => {
|
export const send = async ({ serviceName, newListings, notificationConfig, jobKey }) => {
|
||||||
const { authToken, endpointUrl } = notificationConfig.find((a) => a.id === config.id).fields;
|
const { authToken, endpointUrl, selfSignedCerts } = notificationConfig.find((a) => a.id === config.id).fields;
|
||||||
|
|
||||||
const listings = newListings.map(mapListing);
|
const listings = newListings.map(mapListing);
|
||||||
const body = {
|
const body = {
|
||||||
@@ -34,11 +34,20 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey }) =
|
|||||||
headers['Authorization'] = `Bearer ${authToken}`;
|
headers['Authorization'] = `Bearer ${authToken}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
return fetch(endpointUrl, {
|
let fetchOptions = {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
headers: headers,
|
headers,
|
||||||
|
timeout: 10000,
|
||||||
body: JSON.stringify(body),
|
body: JSON.stringify(body),
|
||||||
});
|
};
|
||||||
|
|
||||||
|
if (selfSignedCerts === true) {
|
||||||
|
fetchOptions.dispatcher = new (await import('undici')).Agent({
|
||||||
|
connect: { rejectUnauthorized: false },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return fetch(endpointUrl, fetchOptions);
|
||||||
};
|
};
|
||||||
|
|
||||||
export const config = {
|
export const config = {
|
||||||
@@ -52,6 +61,10 @@ export const config = {
|
|||||||
label: 'Endpoint URL',
|
label: 'Endpoint URL',
|
||||||
type: 'text',
|
type: 'text',
|
||||||
},
|
},
|
||||||
|
selfSignedCerts: {
|
||||||
|
label: 'Self-signed certificates',
|
||||||
|
type: 'boolean',
|
||||||
|
},
|
||||||
authToken: {
|
authToken: {
|
||||||
description: "Your application's auth token, if required by your endpoint.",
|
description: "Your application's auth token, if required by your endpoint.",
|
||||||
label: 'Auth token (optional)',
|
label: 'Auth token (optional)',
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
88
lib/notification/adapter/resend.js
Executable file
88
lib/notification/adapter/resend.js
Executable file
@@ -0,0 +1,88 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Resend } from 'resend';
|
||||||
|
import path from 'path';
|
||||||
|
import fs from 'fs';
|
||||||
|
import Handlebars from 'handlebars';
|
||||||
|
import { markdown2Html } from '../../services/markdown.js';
|
||||||
|
import { getDirName, normalizeImageUrl } from '../../utils.js';
|
||||||
|
|
||||||
|
const __dirname = getDirName();
|
||||||
|
const template = fs.readFileSync(path.resolve(__dirname + '/notification/emailTemplate/template.hbs'), 'utf8');
|
||||||
|
const emailTemplate = Handlebars.compile(template);
|
||||||
|
|
||||||
|
const mapListings = (serviceName, jobKey, listings) =>
|
||||||
|
listings.map((l) => {
|
||||||
|
const image = normalizeImageUrl(l.image);
|
||||||
|
return {
|
||||||
|
title: l.title || '',
|
||||||
|
link: l.link || '',
|
||||||
|
address: l.address || '',
|
||||||
|
size: l.size || '',
|
||||||
|
price: l.price || '',
|
||||||
|
image,
|
||||||
|
hasImage: Boolean(image),
|
||||||
|
serviceName,
|
||||||
|
jobKey,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
export const send = async ({ serviceName, newListings, notificationConfig, jobKey }) => {
|
||||||
|
const { apiKey, receiver, from } = notificationConfig.find((adapter) => adapter.id === config.id).fields;
|
||||||
|
|
||||||
|
const to = receiver
|
||||||
|
.trim()
|
||||||
|
.split(',')
|
||||||
|
.map((r) => r.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
|
||||||
|
const resend = new Resend(apiKey);
|
||||||
|
|
||||||
|
const listings = mapListings(serviceName, jobKey, newListings);
|
||||||
|
|
||||||
|
const html = emailTemplate({
|
||||||
|
serviceName: `Job: (${jobKey}) | Service: ${serviceName}`,
|
||||||
|
numberOfListings: listings.length,
|
||||||
|
listings,
|
||||||
|
});
|
||||||
|
|
||||||
|
const { error } = await resend.emails.send({
|
||||||
|
from,
|
||||||
|
to,
|
||||||
|
subject: `Fredy found ${listings.length} new listing(s) for ${serviceName}`,
|
||||||
|
html,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!error) {
|
||||||
|
return Promise.resolve();
|
||||||
|
} else {
|
||||||
|
return Promise.reject(error.message);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
export const config = {
|
||||||
|
id: 'resend',
|
||||||
|
name: 'Resend',
|
||||||
|
description: 'Resend is being used to send new listings via mail.',
|
||||||
|
readme: markdown2Html('lib/notification/adapter/resend.md'),
|
||||||
|
fields: {
|
||||||
|
apiKey: {
|
||||||
|
type: 'text',
|
||||||
|
label: 'Api Key',
|
||||||
|
description: 'The Resend API key used to send emails.',
|
||||||
|
},
|
||||||
|
receiver: {
|
||||||
|
type: 'email',
|
||||||
|
label: 'Receiver Email',
|
||||||
|
description: 'Comma-separated email addresses Fredy will send notifications to.',
|
||||||
|
},
|
||||||
|
from: {
|
||||||
|
type: 'email',
|
||||||
|
label: 'Sender Email',
|
||||||
|
description: 'The verified email address or domain you send from in Resend.',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
17
lib/notification/adapter/resend.md
Normal file
17
lib/notification/adapter/resend.md
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
### Resend Adapter
|
||||||
|
|
||||||
|
Resend is a modern email delivery service that Fredy can use to send notifications.
|
||||||
|
|
||||||
|
Setup:
|
||||||
|
- Create a Resend account: https://resend.com/
|
||||||
|
- Create an API key and add it to Fredy's configuration.
|
||||||
|
- Choose the sender address (e.g., you@yourdomain.com). Verify the domain (https://resend.com/domains/) in Resend before using it.
|
||||||
|
- Optional for local testing: you can use `onboarding@resend.dev`, but Resend may restrict who you can send to when using test domains.
|
||||||
|
|
||||||
|
Multiple recipients:
|
||||||
|
- Separate email addresses with commas (e.g., some@email.com, someOther@email.com).
|
||||||
|
|
||||||
|
Notes & Troubleshooting:
|
||||||
|
- Ensure the `from` address is verified or belongs to a verified domain in Resend.
|
||||||
|
- If emails don't arrive, check your spam folder and Resend dashboard logs.
|
||||||
|
- The template displays listing images via their public URLs; make sure images are reachable.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -9,7 +9,9 @@ import checkIfListingIsActive from '../services/listings/listingActiveTester.js'
|
|||||||
let appliedBlackList = [];
|
let appliedBlackList = [];
|
||||||
|
|
||||||
function shortenLink(link) {
|
function shortenLink(link) {
|
||||||
return link.substring(0, link.indexOf('?'));
|
if (!link) return '';
|
||||||
|
const index = link.indexOf('?');
|
||||||
|
return index === -1 ? link : link.substring(0, index);
|
||||||
}
|
}
|
||||||
|
|
||||||
function parseId(shortenedLink) {
|
function parseId(shortenedLink) {
|
||||||
@@ -23,7 +25,7 @@ function normalize(o) {
|
|||||||
const title = o.title || 'No title available';
|
const title = o.title || 'No title available';
|
||||||
const address = o.address || null;
|
const address = o.address || null;
|
||||||
const shortLink = shortenLink(o.link);
|
const shortLink = shortenLink(o.link);
|
||||||
const link = `${baseUrl}/${shortLink}`;
|
const link = baseUrl + shortLink;
|
||||||
const image = baseUrl + o.image;
|
const image = baseUrl + o.image;
|
||||||
const id = buildHash(parseId(shortLink), o.price);
|
const id = buildHash(parseId(shortLink), o.price);
|
||||||
return Object.assign(o, { id, price, size, title, address, link, image });
|
return Object.assign(o, { id, price, size, title, address, link, image });
|
||||||
@@ -37,18 +39,18 @@ function applyBlacklist(o) {
|
|||||||
|
|
||||||
const config = {
|
const config = {
|
||||||
url: null,
|
url: null,
|
||||||
crawlContainer: '._ref',
|
crawlContainer: 'a:has(div.list_entry)',
|
||||||
sortByDateParam: 'sort_col=*created_ts&sort_dir=desc',
|
sortByDateParam: 'sort_col=*created_ts&sort_dir=desc',
|
||||||
waitForSelector: 'body',
|
waitForSelector: 'body',
|
||||||
crawlFields: {
|
crawlFields: {
|
||||||
id: '@href', //will be transformed later
|
id: '@href', //will be transformed later
|
||||||
price: '.list_entry .immo_preis .label_info',
|
price: '.immo_preis .label_info',
|
||||||
size: '.list_entry .flaeche .label_info | removeNewline | trim',
|
size: '.flaeche .label_info | removeNewline | trim',
|
||||||
title: '.list_entry .part_text h3 span',
|
title: 'h3 span',
|
||||||
description: '.list_entry .description | trim',
|
description: '.description | trim',
|
||||||
link: '@href',
|
link: '@href',
|
||||||
address: '.list_entry .place',
|
address: '.place',
|
||||||
image: '.list_entry img@src',
|
image: 'img@src',
|
||||||
},
|
},
|
||||||
normalize: normalize,
|
normalize: normalize,
|
||||||
filter: applyBlacklist,
|
filter: applyBlacklist,
|
||||||
|
|||||||
@@ -1,53 +0,0 @@
|
|||||||
/*
|
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { isOneOf, buildHash } from '../utils.js';
|
|
||||||
import checkIfListingIsActive from '../services/listings/listingActiveTester.js';
|
|
||||||
let appliedBlackList = [];
|
|
||||||
|
|
||||||
function normalize(o) {
|
|
||||||
const size = o.size != null ? o.size.replace('Wohnfläche ', '') : 'N/A m²';
|
|
||||||
const price = o.price.replace('Kaufpreis ', '');
|
|
||||||
const address = o.address?.split(' • ')?.pop() ?? null;
|
|
||||||
const title = o.title || 'No title available';
|
|
||||||
const link = o.link != null ? decodeURIComponent(o.link) : config.url;
|
|
||||||
const id = buildHash(title, price);
|
|
||||||
return Object.assign(o, { id, address, price, size, title, link });
|
|
||||||
}
|
|
||||||
function applyBlacklist(o) {
|
|
||||||
const titleNotBlacklisted = !isOneOf(o.title, appliedBlackList);
|
|
||||||
const descNotBlacklisted = !isOneOf(o.description, appliedBlackList);
|
|
||||||
return titleNotBlacklisted && descNotBlacklisted;
|
|
||||||
}
|
|
||||||
const config = {
|
|
||||||
url: null,
|
|
||||||
crawlContainer: 'div[data-testid="serp-core-classified-card-testid"]',
|
|
||||||
sortByDateParam: 'sortby=19',
|
|
||||||
waitForSelector: 'div[data-testid="serp-gridcontainer-testid"]',
|
|
||||||
crawlFields: {
|
|
||||||
id: 'button@title |trim',
|
|
||||||
title: 'button@title |trim',
|
|
||||||
price: 'div[data-testid="cardmfe-price-testid"] | trim',
|
|
||||||
size: 'div[data-testid="cardmfe-keyfacts-testid"] | trim',
|
|
||||||
address: 'div[data-testid="cardmfe-description-box-address"] | trim',
|
|
||||||
image: 'div[data-testid="cardmfe-picture-box-test-id"] img@src',
|
|
||||||
link: 'button@data-base',
|
|
||||||
description: 'div[data-testid="cardmfe-description-text-test-id"] | trim',
|
|
||||||
},
|
|
||||||
normalize: normalize,
|
|
||||||
filter: applyBlacklist,
|
|
||||||
activeTester: checkIfListingIsActive,
|
|
||||||
};
|
|
||||||
export const init = (sourceConfig, blacklist) => {
|
|
||||||
config.enabled = sourceConfig.enabled;
|
|
||||||
config.url = sourceConfig.url;
|
|
||||||
appliedBlackList = blacklist || [];
|
|
||||||
};
|
|
||||||
export const metaInformation = {
|
|
||||||
name: 'Immonet',
|
|
||||||
baseUrl: 'https://www.immonet.de/',
|
|
||||||
id: 'immonet',
|
|
||||||
};
|
|
||||||
export { config };
|
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -8,7 +8,7 @@
|
|||||||
*
|
*
|
||||||
* The mobile API provides the following endpoints:
|
* The mobile API provides the following endpoints:
|
||||||
* - GET /search/total?{search parameters}: Returns the total number of listings for the given query
|
* - GET /search/total?{search parameters}: Returns the total number of listings for the given query
|
||||||
* Example: `curl -H "User-Agent: ImmoScout_27.3_26.0_._" https://api.mobile.immobilienscout24.de/search/total?searchType=region&realestatetype=apartmentrent&pricetype=calculatedtotalrent&geocodes=%2Fde%2Fberlin%2Fberlin `
|
* Example: `curl -H "User-Agent: ImmoScout_27.12_26.2_._" https://api.mobile.immobilienscout24.de/search/total?searchType=region&realestatetype=apartmentrent&pricetype=calculatedtotalrent&geocodes=%2Fde%2Fberlin%2Fberlin `
|
||||||
*
|
*
|
||||||
* - POST /search/list?{search parameters}: Actually retrieves the listings. Body is json encoded and contains
|
* - POST /search/list?{search parameters}: Actually retrieves the listings. Body is json encoded and contains
|
||||||
* data specifying additional results (advertisements) to return. The format is as follows:
|
* data specifying additional results (advertisements) to return. The format is as follows:
|
||||||
@@ -20,12 +20,12 @@
|
|||||||
* ```
|
* ```
|
||||||
* It is not necessary to provide data for the specified keys.
|
* It is not necessary to provide data for the specified keys.
|
||||||
*
|
*
|
||||||
* Example: `curl -X POST 'https://api.mobile.immobilienscout24.de/search/list?pricetype=calculatedtotalrent&realestatetype=apartmentrent&searchType=region&geocodes=%2Fde%2Fberlin%2Fberlin&pagenumber=1' -H "Connection: keep-alive" -H "User-Agent: ImmoScout_27.3_26.0_._" -H "Accept: application/json" -H "Content-Type: application/json" -d '{"supportedResultListType": [], "userData": {}}'`
|
* Example: `curl -X POST 'https://api.mobile.immobilienscout24.de/search/list?pricetype=calculatedtotalrent&realestatetype=apartmentrent&searchType=region&geocodes=%2Fde%2Fberlin%2Fberlin&pagenumber=1' -H "Connection: keep-alive" -H "User-Agent: ImmoScout_27.12_26.2_._" -H "Accept: application/json" -H "Content-Type: application/json" -d '{"supportedResultListType": [], "userData": {}}'`
|
||||||
|
|
||||||
* - GET /expose/{id} - Returns the details of a listing. The response contains additional details not included in the
|
* - GET /expose/{id} - Returns the details of a listing. The response contains additional details not included in the
|
||||||
* listing response.
|
* listing response.
|
||||||
*
|
*
|
||||||
* Example: `curl -H "User-Agent: ImmoScout_27.3_26.0_._" "https://api.mobile.immobilienscout24.de/expose/158382494"`
|
* Example: `curl -H "User-Agent: ImmoScout_27.12_26.2_._" "https://api.mobile.immobilienscout24.de/expose/158382494"`
|
||||||
*
|
*
|
||||||
*
|
*
|
||||||
* It is necessary to set the correct User Agent (see `getListings`) in the request header.
|
* It is necessary to set the correct User Agent (see `getListings`) in the request header.
|
||||||
@@ -46,13 +46,15 @@ import {
|
|||||||
convertWebToMobile,
|
convertWebToMobile,
|
||||||
} from '../services/immoscout/immoscout-web-translator.js';
|
} from '../services/immoscout/immoscout-web-translator.js';
|
||||||
import logger from '../services/logger.js';
|
import logger from '../services/logger.js';
|
||||||
|
import { getUserSettings } from '../services/storage/settingsStorage.js';
|
||||||
let appliedBlackList = [];
|
let appliedBlackList = [];
|
||||||
|
let currentUserId = null;
|
||||||
|
|
||||||
async function getListings(url) {
|
async function getListings(url) {
|
||||||
const response = await fetch(url, {
|
const response = await fetch(url, {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
headers: {
|
headers: {
|
||||||
'User-Agent': 'ImmoScout_27.3_26.0_._',
|
'User-Agent': 'ImmoScout_27.12_26.2_._',
|
||||||
'Content-Type': 'application/json',
|
'Content-Type': 'application/json',
|
||||||
},
|
},
|
||||||
body: JSON.stringify({
|
body: JSON.stringify({
|
||||||
@@ -66,29 +68,92 @@ async function getListings(url) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const responseBody = await response.json();
|
const responseBody = await response.json();
|
||||||
return responseBody.resultListItems
|
return Promise.all(
|
||||||
.filter((item) => item.type === 'EXPOSE_RESULT')
|
responseBody.resultListItems
|
||||||
.map((expose) => {
|
.filter((item) => item.type === 'EXPOSE_RESULT')
|
||||||
const item = expose.item;
|
.map(async (expose) => {
|
||||||
const [price, size] = item.attributes;
|
const item = expose.item;
|
||||||
const image = item?.titlePicture?.preview ?? null;
|
const [price, size] = item.attributes;
|
||||||
return {
|
const image = item?.titlePicture?.full ?? item?.titlePicture?.preview ?? null;
|
||||||
id: item.id,
|
let listing = {
|
||||||
price: price?.value,
|
id: item.id,
|
||||||
size: size?.value,
|
price: price?.value,
|
||||||
title: item.title,
|
size: size?.value,
|
||||||
description: item.description,
|
title: item.title,
|
||||||
link: `${metaInformation.baseUrl}expose/${item.id}`,
|
link: `${metaInformation.baseUrl}expose/${item.id}`,
|
||||||
address: item.address?.line,
|
address: item.address?.line,
|
||||||
image,
|
image,
|
||||||
};
|
};
|
||||||
});
|
if (currentUserId) {
|
||||||
|
const userSettings = getUserSettings(currentUserId);
|
||||||
|
if (userSettings.immoscout_details) {
|
||||||
|
return await pushDetails(listing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return listing;
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function pushDetails(listing) {
|
||||||
|
const detailed = await fetch(`https://api.mobile.immobilienscout24.de/expose/${listing.id}`, {
|
||||||
|
headers: {
|
||||||
|
'User-Agent': 'ImmoScout_27.3_26.0_._',
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
if (!detailed.ok) {
|
||||||
|
logger.error('Error fetching listing details from ImmoScout Mobile API:', detailed.statusText);
|
||||||
|
return listing;
|
||||||
|
}
|
||||||
|
const detailBody = await detailed.json();
|
||||||
|
|
||||||
|
listing.description = buildDescription(detailBody);
|
||||||
|
|
||||||
|
return listing;
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildDescription(detailBody) {
|
||||||
|
const sections = detailBody.sections || [];
|
||||||
|
const contact = detailBody.contact || {};
|
||||||
|
const cData = contact?.contactData || {};
|
||||||
|
const agentName = cData?.agent?.name || '';
|
||||||
|
const agentCompany = cData?.agent?.company || '';
|
||||||
|
const stars = cData?.agent?.rating?.numberOfStars || '';
|
||||||
|
const phoneNumbers = contact?.phoneNumbers || [];
|
||||||
|
const phoneNumbersMapped = phoneNumbers
|
||||||
|
.map((p) => `${p.label}: ${p.text}`)
|
||||||
|
.join('\n')
|
||||||
|
.trim();
|
||||||
|
|
||||||
|
const attributes = sections
|
||||||
|
.filter((s) => s.type === 'ATTRIBUTE_LIST')
|
||||||
|
.flatMap((s) => s.attributes)
|
||||||
|
.filter((attr) => attr.label && attr.text)
|
||||||
|
.map((attr) => `${attr.label} ${attr.text}`)
|
||||||
|
.join('\n');
|
||||||
|
|
||||||
|
const freeText = sections
|
||||||
|
.filter((s) => s.type === 'TEXT_AREA')
|
||||||
|
.map((s) => {
|
||||||
|
return `${s.title}\n${s.text}`;
|
||||||
|
})
|
||||||
|
.join('\n\n');
|
||||||
|
|
||||||
|
return (
|
||||||
|
`Agent: ${agentName ? agentName : 'Unbekannt'} ${agentCompany ? `(${agentCompany}) ` : ''}${stars ? `- ${stars} stars` : ''}\n` +
|
||||||
|
(phoneNumbersMapped ? `Phone Numbers:\n${phoneNumbersMapped}` : '') +
|
||||||
|
'\n\n' +
|
||||||
|
attributes.trim() +
|
||||||
|
'\n\n' +
|
||||||
|
freeText.trim()
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
async function isListingActive(link) {
|
async function isListingActive(link) {
|
||||||
const result = await fetch(convertImmoscoutListingToMobileListing(link), {
|
const result = await fetch(convertImmoscoutListingToMobileListing(link), {
|
||||||
headers: {
|
headers: {
|
||||||
'User-Agent': 'ImmoScout_27.3_26.0_._',
|
'User-Agent': 'ImmoScout_27.12_26.2_._',
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -137,6 +202,7 @@ export const init = (sourceConfig, blacklist) => {
|
|||||||
config.enabled = sourceConfig.enabled;
|
config.enabled = sourceConfig.enabled;
|
||||||
config.url = convertWebToMobile(sourceConfig.url);
|
config.url = convertWebToMobile(sourceConfig.url);
|
||||||
appliedBlackList = blacklist || [];
|
appliedBlackList = blacklist || [];
|
||||||
|
currentUserId = sourceConfig.userId || null;
|
||||||
};
|
};
|
||||||
export const metaInformation = {
|
export const metaInformation = {
|
||||||
name: 'Immoscout',
|
name: 'Immoscout',
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -36,7 +36,7 @@ const config = {
|
|||||||
},
|
},
|
||||||
normalize: normalize,
|
normalize: normalize,
|
||||||
filter: applyBlacklist,
|
filter: applyBlacklist,
|
||||||
activeTester: checkIfListingIsActive,
|
activeTester: (url) => checkIfListingIsActive(url, 'Angebot nicht gefunden'),
|
||||||
};
|
};
|
||||||
export const init = (sourceConfig, blacklist) => {
|
export const init = (sourceConfig, blacklist) => {
|
||||||
config.enabled = sourceConfig.enabled;
|
config.enabled = sourceConfig.enabled;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
58
lib/provider/wohnungsboerse.js
Normal file
58
lib/provider/wohnungsboerse.js
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import * as utils from '../utils.js';
|
||||||
|
import checkIfListingIsActive from '../services/listings/listingActiveTester.js';
|
||||||
|
|
||||||
|
let appliedBlackList = [];
|
||||||
|
|
||||||
|
function normalize(o) {
|
||||||
|
const id = o.link.split('/').pop();
|
||||||
|
const price = o.price;
|
||||||
|
const size = o.size;
|
||||||
|
const rooms = o.rooms;
|
||||||
|
const [city = '', part = ''] = (o.description || '').split('-').map((v) => v.trim());
|
||||||
|
const address = `${part}, ${city}`;
|
||||||
|
return Object.assign(o, { id, price, size, rooms, address });
|
||||||
|
}
|
||||||
|
|
||||||
|
function applyBlacklist(o) {
|
||||||
|
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
|
||||||
|
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
|
||||||
|
return o.id != null && o.title != null && titleNotBlacklisted && descNotBlacklisted && o.link.startsWith(o.link);
|
||||||
|
}
|
||||||
|
|
||||||
|
const config = {
|
||||||
|
url: null,
|
||||||
|
sortByDateParam: null,
|
||||||
|
waitForSelector: 'body',
|
||||||
|
crawlContainer: '.search_result_container > a',
|
||||||
|
crawlFields: {
|
||||||
|
id: '*',
|
||||||
|
title: 'h3 | trim',
|
||||||
|
price: 'dl:nth-of-type(1) dd | removeNewline | trim',
|
||||||
|
rooms: 'dl:nth-of-type(2) dd | removeNewline | trim',
|
||||||
|
size: 'dl:nth-of-type(3) dd | removeNewline | trim',
|
||||||
|
description: 'div.before\\:icon-location_marker | trim',
|
||||||
|
link: '@href',
|
||||||
|
imageUrl: 'img@src',
|
||||||
|
},
|
||||||
|
normalize: normalize,
|
||||||
|
filter: applyBlacklist,
|
||||||
|
activeTester: checkIfListingIsActive,
|
||||||
|
};
|
||||||
|
|
||||||
|
export const init = (sourceConfig, blacklistTerms) => {
|
||||||
|
config.url = sourceConfig.url;
|
||||||
|
appliedBlackList = blacklistTerms || [];
|
||||||
|
};
|
||||||
|
|
||||||
|
export const metaInformation = {
|
||||||
|
name: 'Wohnungsboerse',
|
||||||
|
baseUrl: 'https://www.wohnungsboerse.net',
|
||||||
|
id: 'wohnungsboerse',
|
||||||
|
};
|
||||||
|
|
||||||
|
export { config };
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
/*
|
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { removeJobsByUserId } from '../storage/jobStorage.js';
|
|
||||||
import { getUsers } from '../storage/userStorage.js';
|
|
||||||
import logger from '../logger.js';
|
|
||||||
import cron from 'node-cron';
|
|
||||||
import { getSettings } from '../storage/settingsStorage.js';
|
|
||||||
|
|
||||||
/**
|
|
||||||
* if we are running in demo environment, we have to cleanup the db files (specifically the jobs table)
|
|
||||||
*/
|
|
||||||
export function cleanupDemoAtMidnight() {
|
|
||||||
cron.schedule('0 0 * * *', cleanup);
|
|
||||||
}
|
|
||||||
|
|
||||||
async function cleanup() {
|
|
||||||
const settings = await getSettings();
|
|
||||||
if (settings.demoMode) {
|
|
||||||
const demoUser = getUsers(false).find((user) => user.username === 'demo');
|
|
||||||
if (demoUser == null) {
|
|
||||||
logger.error('Demo user not found, cannot remove Jobs');
|
|
||||||
return Promise.resolve();
|
|
||||||
}
|
|
||||||
removeJobsByUserId(demoUser.id);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
46
lib/services/crons/geocoding-cron.js
Normal file
46
lib/services/crons/geocoding-cron.js
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import cron from 'node-cron';
|
||||||
|
import { getListingsToGeocode, updateListingGeocoordinates } from '../storage/listingsStorage.js';
|
||||||
|
import { geocodeAddress, isGeocodingPaused } from '../geocoding/geoCodingService.js';
|
||||||
|
import { getJobs } from '../storage/jobStorage.js';
|
||||||
|
import { calculateDistanceForJob } from '../geocoding/distanceService.js';
|
||||||
|
import { getSettings } from '../storage/settingsStorage.js';
|
||||||
|
import logger from '../logger.js';
|
||||||
|
|
||||||
|
export async function runGeoCordTask() {
|
||||||
|
const listings = getListingsToGeocode();
|
||||||
|
if (listings.length > 0) {
|
||||||
|
for (const listing of listings) {
|
||||||
|
if (isGeocodingPaused()) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
const coords = await geocodeAddress(listing.address);
|
||||||
|
if (coords) {
|
||||||
|
updateListingGeocoordinates(listing.id, coords.lat, coords.lng);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
//additional run
|
||||||
|
const jobs = getJobs();
|
||||||
|
for (const job of jobs) {
|
||||||
|
calculateDistanceForJob(job.id, job.userId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function initGeocodingCron() {
|
||||||
|
const settings = await getSettings();
|
||||||
|
if (settings.demoMode) {
|
||||||
|
logger.info('Do not start geo service as we are in demo mode');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// run directly on start
|
||||||
|
await runGeoCordTask();
|
||||||
|
// then every 6 hours
|
||||||
|
cron.schedule('0 */6 * * *', runGeoCordTask);
|
||||||
|
}
|
||||||
@@ -1,16 +1,23 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import cron from 'node-cron';
|
import cron from 'node-cron';
|
||||||
import runActiveChecker from '../listings/listingActiveService.js';
|
import runActiveChecker from '../listings/listingActiveService.js';
|
||||||
|
import logger from '../logger.js';
|
||||||
|
import { getSettings } from '../storage/settingsStorage.js';
|
||||||
|
|
||||||
async function runTask() {
|
async function runTask() {
|
||||||
await runActiveChecker();
|
await runActiveChecker();
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function initActiveCheckerCron() {
|
export async function initActiveCheckerCron() {
|
||||||
|
const settings = await getSettings();
|
||||||
|
if (settings.demoMode) {
|
||||||
|
logger.info('Do not start listing active checker as we are in demo mode');
|
||||||
|
return;
|
||||||
|
}
|
||||||
//run directly on start
|
//run directly on start
|
||||||
await runTask();
|
await runTask();
|
||||||
// then every day at 1 am
|
// then every day at 1 am
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -19,52 +19,80 @@ import path from 'path';
|
|||||||
|
|
||||||
puppeteer.use(StealthPlugin());
|
puppeteer.use(StealthPlugin());
|
||||||
|
|
||||||
export default async function execute(url, waitForSelector, options) {
|
export async function launchBrowser(url, options) {
|
||||||
let browser;
|
const preCfg = getPreLaunchConfig(url, options || {});
|
||||||
let page;
|
const launchArgs = [
|
||||||
let result = null;
|
'--no-sandbox',
|
||||||
|
'--disable-gpu',
|
||||||
|
'--disable-setuid-sandbox',
|
||||||
|
'--disable-dev-shm-usage',
|
||||||
|
'--disable-crash-reporter',
|
||||||
|
'--no-first-run',
|
||||||
|
'--no-default-browser-check',
|
||||||
|
preCfg.langArg,
|
||||||
|
preCfg.windowSizeArg,
|
||||||
|
...preCfg.extraArgs,
|
||||||
|
];
|
||||||
|
if (options?.proxyUrl) {
|
||||||
|
launchArgs.push(`--proxy-server=${options.proxyUrl}`);
|
||||||
|
}
|
||||||
|
|
||||||
let userDataDir;
|
let userDataDir;
|
||||||
let removeUserDataDir = false;
|
let removeUserDataDir = false;
|
||||||
|
if (options && options.userDataDir) {
|
||||||
|
userDataDir = options.userDataDir;
|
||||||
|
} else {
|
||||||
|
const prefix = path.join(os.tmpdir(), 'puppeteer-fredy-');
|
||||||
|
userDataDir = fs.mkdtempSync(prefix);
|
||||||
|
removeUserDataDir = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
const browser = await puppeteer.launch({
|
||||||
|
headless: options?.puppeteerHeadless ?? true,
|
||||||
|
args: launchArgs,
|
||||||
|
timeout: options?.puppeteerTimeout || 45_000,
|
||||||
|
userDataDir,
|
||||||
|
executablePath: options?.executablePath,
|
||||||
|
});
|
||||||
|
|
||||||
|
browser.__fredy_userDataDir = userDataDir;
|
||||||
|
browser.__fredy_removeUserDataDir = removeUserDataDir;
|
||||||
|
|
||||||
|
return browser;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function closeBrowser(browser) {
|
||||||
|
if (!browser) return;
|
||||||
|
const userDataDir = browser.__fredy_userDataDir;
|
||||||
|
const removeUserDataDir = browser.__fredy_removeUserDataDir;
|
||||||
|
try {
|
||||||
|
await browser.close();
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
if (removeUserDataDir && userDataDir) {
|
||||||
|
try {
|
||||||
|
await fs.promises.rm(userDataDir, { recursive: true, force: true });
|
||||||
|
} catch {
|
||||||
|
// ignore
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export default async function execute(url, waitForSelector, options) {
|
||||||
|
let browser = options?.browser;
|
||||||
|
let isExternalBrowser = !!browser;
|
||||||
|
let page;
|
||||||
|
let result;
|
||||||
try {
|
try {
|
||||||
debug(`Sending request to ${url} using Puppeteer.`);
|
debug(`Sending request to ${url} using Puppeteer.`);
|
||||||
|
|
||||||
// Prepare a dedicated temporary userDataDir to avoid leaking /tmp/.org.chromium.* dirs
|
if (!isExternalBrowser) {
|
||||||
if (options && options.userDataDir) {
|
browser = await launchBrowser(url, options);
|
||||||
userDataDir = options.userDataDir;
|
|
||||||
removeUserDataDir = !!options.cleanupUserDataDir;
|
|
||||||
} else {
|
|
||||||
const prefix = path.join(os.tmpdir(), 'puppeteer-fredy-');
|
|
||||||
userDataDir = fs.mkdtempSync(prefix);
|
|
||||||
removeUserDataDir = true;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const launchArgs = [
|
|
||||||
'--no-sandbox',
|
|
||||||
'--disable-gpu',
|
|
||||||
'--disable-setuid-sandbox',
|
|
||||||
'--disable-dev-shm-usage',
|
|
||||||
'--disable-crash-reporter',
|
|
||||||
'--no-first-run',
|
|
||||||
'--no-default-browser-check',
|
|
||||||
];
|
|
||||||
if (options?.proxyUrl) {
|
|
||||||
launchArgs.push(`--proxy-server=${options.proxyUrl}`);
|
|
||||||
}
|
|
||||||
// Prepare bot prevention pre-launch config
|
|
||||||
const preCfg = getPreLaunchConfig(url, options || {});
|
|
||||||
launchArgs.push(preCfg.langArg);
|
|
||||||
launchArgs.push(preCfg.windowSizeArg);
|
|
||||||
launchArgs.push(...preCfg.extraArgs);
|
|
||||||
|
|
||||||
browser = await puppeteer.launch({
|
|
||||||
headless: options?.puppeteerHeadless ?? true,
|
|
||||||
args: launchArgs,
|
|
||||||
timeout: options?.puppeteerTimeout || 30_000,
|
|
||||||
userDataDir,
|
|
||||||
executablePath: options?.executablePath, // allow using system Chrome
|
|
||||||
});
|
|
||||||
|
|
||||||
page = await browser.newPage();
|
page = await browser.newPage();
|
||||||
|
const preCfg = getPreLaunchConfig(url, options || {});
|
||||||
await applyBotPreventionToPage(page, preCfg);
|
await applyBotPreventionToPage(page, preCfg);
|
||||||
// Provide languages value before navigation
|
// Provide languages value before navigation
|
||||||
await applyLanguagePersistence(page, preCfg);
|
await applyLanguagePersistence(page, preCfg);
|
||||||
@@ -104,7 +132,7 @@ export default async function execute(url, waitForSelector, options) {
|
|||||||
result = pageSource || (await page.content());
|
result = pageSource || (await page.content());
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
if (error?.message?.includes('Timeout')) {
|
if (error?.name?.includes('Timeout')) {
|
||||||
logger.debug('Error executing with puppeteer executor', error);
|
logger.debug('Error executing with puppeteer executor', error);
|
||||||
} else {
|
} else {
|
||||||
logger.warn('Error executing with puppeteer executor', error);
|
logger.warn('Error executing with puppeteer executor', error);
|
||||||
@@ -118,19 +146,8 @@ export default async function execute(url, waitForSelector, options) {
|
|||||||
} catch {
|
} catch {
|
||||||
// ignore
|
// ignore
|
||||||
}
|
}
|
||||||
try {
|
if (browser != null && !isExternalBrowser) {
|
||||||
if (browser != null) {
|
await closeBrowser(browser);
|
||||||
await browser.close();
|
|
||||||
}
|
|
||||||
} catch {
|
|
||||||
// ignore
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
if (removeUserDataDir && userDataDir) {
|
|
||||||
await fs.promises.rm(userDataDir, { recursive: true, force: true });
|
|
||||||
}
|
|
||||||
} catch {
|
|
||||||
// ignore
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return result;
|
return result;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
26
lib/services/geocoding/autocompleteService.js
Normal file
26
lib/services/geocoding/autocompleteService.js
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { autocomplete as nominatimAutocomplete } from './client/nominatimClient.js';
|
||||||
|
import logger from '../logger.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Autocompletes an address using Nominatim.
|
||||||
|
*
|
||||||
|
* @param {string} query - The search query.
|
||||||
|
* @returns {Promise<string[]>} List of matching addresses.
|
||||||
|
*/
|
||||||
|
export async function autocompleteAddress(query) {
|
||||||
|
if (!query) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
return await nominatimAutocomplete(query);
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error during address autocomplete:', error);
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
153
lib/services/geocoding/client/nominatimClient.js
Normal file
153
lib/services/geocoding/client/nominatimClient.js
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import os from 'os';
|
||||||
|
import crypto from 'crypto';
|
||||||
|
import https from 'https';
|
||||||
|
import fetch from 'node-fetch';
|
||||||
|
import pThrottle from 'p-throttle';
|
||||||
|
import logger from '../../logger.js';
|
||||||
|
|
||||||
|
const API_URL = 'https://nominatim.openstreetmap.org/search';
|
||||||
|
|
||||||
|
const agent = new https.Agent({
|
||||||
|
keepAlive: true,
|
||||||
|
keepAliveMsecs: 1000,
|
||||||
|
});
|
||||||
|
|
||||||
|
const throttle = pThrottle({
|
||||||
|
limit: 1,
|
||||||
|
interval: 1000,
|
||||||
|
});
|
||||||
|
|
||||||
|
function computeMachineId() {
|
||||||
|
const hostname = os.hostname() || 'unknown-host';
|
||||||
|
const nets = os.networkInterfaces?.() || {};
|
||||||
|
const macs = [];
|
||||||
|
|
||||||
|
for (const ifname of Object.keys(nets)) {
|
||||||
|
for (const addr of nets[ifname] || []) {
|
||||||
|
if (!addr) continue;
|
||||||
|
if (addr.internal) continue;
|
||||||
|
if (addr.mac && addr.mac !== '00:00:00:00:00:00') macs.push(addr.mac);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
macs.sort();
|
||||||
|
|
||||||
|
const raw = [hostname, os.platform(), os.arch(), ...macs].join('|');
|
||||||
|
|
||||||
|
return crypto.createHash('sha256').update(raw).digest('hex').slice(0, 20);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Nominatim requires a specific User-Agent.
|
||||||
|
* Since Fredy is self-hosted, we use a unique machine ID to make it specific.
|
||||||
|
*/
|
||||||
|
const userAgent = `Fredy-Self-Hosted (${computeMachineId()}; https://github.com/orangecoding/fredy)`;
|
||||||
|
|
||||||
|
let last429 = 0;
|
||||||
|
const PAUSE_DURATION = 3600000; // 1 hour
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Geocodes an address using Nominatim.
|
||||||
|
*
|
||||||
|
* @param {string} address - The address to geocode.
|
||||||
|
* @returns {Promise<{lat: number, lng: number}|null>} The geocoordinates or null if error. {lat: -1, lng: -1} if not found.
|
||||||
|
*/
|
||||||
|
async function doGeocode(address) {
|
||||||
|
if (Date.now() - last429 < PAUSE_DURATION) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const url = `${API_URL}?q=${encodeURIComponent(address)}&format=json&countrycodes=de`;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const response = await fetch(url, {
|
||||||
|
agent,
|
||||||
|
timeout: 60000,
|
||||||
|
headers: {
|
||||||
|
'User-Agent': userAgent,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
if (response.status === 429) {
|
||||||
|
logger.warn('Nominatim rate limit hit. Pausing for 1 hour.');
|
||||||
|
last429 = Date.now();
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
logger.error(`Nominatim API error: ${response.status} ${response.statusText}`);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = await response.json();
|
||||||
|
|
||||||
|
if (Array.isArray(data) && data.length > 0) {
|
||||||
|
const result = data[0];
|
||||||
|
return {
|
||||||
|
lat: parseFloat(result.lat),
|
||||||
|
lng: parseFloat(result.lon),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return { lat: -1, lng: -1 };
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error during Nominatim geocoding:', error);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Autocompletes an address using Nominatim.
|
||||||
|
*
|
||||||
|
* @param {string} query - The search query.
|
||||||
|
* @returns {Promise<string[]>} List of matching addresses.
|
||||||
|
*/
|
||||||
|
async function doAutocomplete(query) {
|
||||||
|
if (Date.now() - last429 < PAUSE_DURATION) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
const url = `${API_URL}?q=${encodeURIComponent(query)}&format=json&addressdetails=1&limit=5&countrycodes=de`;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const response = await fetch(url, {
|
||||||
|
agent,
|
||||||
|
headers: {
|
||||||
|
'User-Agent': userAgent,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
if (response.status === 429) {
|
||||||
|
logger.warn('Nominatim rate limit hit. Pausing for 1 hour.');
|
||||||
|
last429 = Date.now();
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
logger.error(`Nominatim API error: ${response.status} ${response.statusText}`);
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = await response.json();
|
||||||
|
|
||||||
|
if (Array.isArray(data)) {
|
||||||
|
return data.map((item) => item.display_name);
|
||||||
|
}
|
||||||
|
|
||||||
|
return [];
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error during Nominatim autocomplete:', error);
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export const geocode = throttle(doGeocode);
|
||||||
|
|
||||||
|
export const autocomplete = throttle(doAutocomplete);
|
||||||
|
|
||||||
|
export const isPaused = () => Date.now() - last429 < PAUSE_DURATION;
|
||||||
61
lib/services/geocoding/distanceService.js
Normal file
61
lib/services/geocoding/distanceService.js
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { distanceMeters } from '../listings/distanceCalculator.js';
|
||||||
|
import {
|
||||||
|
getListingsToCalculateDistance,
|
||||||
|
getListingsForUserToCalculateDistance,
|
||||||
|
updateListingDistance,
|
||||||
|
} from '../storage/listingsStorage.js';
|
||||||
|
import { getUserSettings } from '../storage/settingsStorage.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Calculates and updates distances for listings of a specific job.
|
||||||
|
* Only processes listings where distance_to_destination is null.
|
||||||
|
*
|
||||||
|
* @param {string} jobId
|
||||||
|
* @param {string} userId
|
||||||
|
* @returns {void}
|
||||||
|
*/
|
||||||
|
export function calculateDistanceForJob(jobId, userId) {
|
||||||
|
const userSettings = getUserSettings(userId);
|
||||||
|
const homeAddress = userSettings.home_address;
|
||||||
|
|
||||||
|
if (!homeAddress || !homeAddress.coords) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const listings = getListingsToCalculateDistance(jobId);
|
||||||
|
const { lat, lng } = homeAddress.coords;
|
||||||
|
|
||||||
|
for (const listing of listings) {
|
||||||
|
const dist = distanceMeters(lat, lng, listing.latitude, listing.longitude);
|
||||||
|
updateListingDistance(listing.id, dist);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Calculates and updates distances for all active listings of a user.
|
||||||
|
* Usually called when the user updates their home address.
|
||||||
|
*
|
||||||
|
* @param {string} userId
|
||||||
|
* @returns {void}
|
||||||
|
*/
|
||||||
|
export function calculateDistanceForUser(userId) {
|
||||||
|
const userSettings = getUserSettings(userId);
|
||||||
|
const homeAddress = userSettings.home_address;
|
||||||
|
|
||||||
|
if (!homeAddress || !homeAddress.coords) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const listings = getListingsForUserToCalculateDistance(userId);
|
||||||
|
const { lat, lng } = homeAddress.coords;
|
||||||
|
|
||||||
|
for (const listing of listings) {
|
||||||
|
const dist = distanceMeters(lat, lng, listing.latitude, listing.longitude);
|
||||||
|
updateListingDistance(listing.id, dist);
|
||||||
|
}
|
||||||
|
}
|
||||||
43
lib/services/geocoding/geoCodingService.js
Normal file
43
lib/services/geocoding/geoCodingService.js
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { getGeocoordinatesByAddress } from '../storage/listingsStorage.js';
|
||||||
|
import { geocode as nominatimGeocode, isPaused as isNominatimPaused } from './client/nominatimClient.js';
|
||||||
|
import logger from '../logger.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Geocodes an address using Nominatim or cached results from the database.
|
||||||
|
*
|
||||||
|
* @param {string} address - The address to geocode.
|
||||||
|
* @returns {Promise<{lat: number, lng: number}|null>} The geocoordinates or null if error. {lat: -1, lng: -1} if not found.
|
||||||
|
*/
|
||||||
|
export async function geocodeAddress(address) {
|
||||||
|
if (!address) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
// 1. Check if we already have this address geocoded in our database
|
||||||
|
const cachedCoordinates = getGeocoordinatesByAddress(address);
|
||||||
|
if (cachedCoordinates) {
|
||||||
|
logger.debug(`Found cached geocoordinates for address: ${address}`);
|
||||||
|
return cachedCoordinates;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. If not, use Nominatim
|
||||||
|
return await nominatimGeocode(address);
|
||||||
|
} catch (error) {
|
||||||
|
logger.error('Error during geocoding:', error);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Checks if we are currently in a rate limit pause.
|
||||||
|
* @returns {boolean}
|
||||||
|
*/
|
||||||
|
export function isGeocodingPaused() {
|
||||||
|
return isNominatimPaused();
|
||||||
|
}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -86,6 +86,7 @@ const PARAM_NAME_MAP = {
|
|||||||
shape: 'shape',
|
shape: 'shape',
|
||||||
sorting: 'sorting',
|
sorting: 'sorting',
|
||||||
newbuilding: 'newbuilding',
|
newbuilding: 'newbuilding',
|
||||||
|
fulltext: 'fulltext',
|
||||||
};
|
};
|
||||||
|
|
||||||
const EQUIPMENT_MAP = {
|
const EQUIPMENT_MAP = {
|
||||||
@@ -103,13 +104,17 @@ const REAL_ESTATE_TYPE = {
|
|||||||
'haus-mieten': 'houserent',
|
'haus-mieten': 'houserent',
|
||||||
'wohnung-mieten': 'apartmentrent',
|
'wohnung-mieten': 'apartmentrent',
|
||||||
'wohnung-kaufen': 'apartmentbuy',
|
'wohnung-kaufen': 'apartmentbuy',
|
||||||
|
'wohnung-kaufen-mit-balkon': 'apartmentbuy',
|
||||||
|
'eigentumswohnung-mit-garten': 'apartmentbuy',
|
||||||
'haus-kaufen': 'housebuy',
|
'haus-kaufen': 'housebuy',
|
||||||
};
|
};
|
||||||
|
|
||||||
const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
|
const WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP = {
|
||||||
// Category "Balkon/Terrasse"
|
// Category "Balkon/Terrasse"
|
||||||
'wohnung-mit-balkon-mieten': { equipment: ['balcony'] },
|
'wohnung-mit-balkon-mieten': { equipment: ['balcony'] },
|
||||||
|
'wohnung-kaufen-mit-balkon': { equipment: ['balcony'] },
|
||||||
'wohnung-mit-garten-mieten': { equipment: ['garden'] },
|
'wohnung-mit-garten-mieten': { equipment: ['garden'] },
|
||||||
|
'eigentumswohnung-mit-garten': { equipment: ['garden'] },
|
||||||
// Category "Wohnungstyp"
|
// Category "Wohnungstyp"
|
||||||
'souterrainwohnung-mieten': { apartmenttypes: ['halfbasement'] },
|
'souterrainwohnung-mieten': { apartmenttypes: ['halfbasement'] },
|
||||||
'erdgeschosswohnung-mieten': { apartmenttypes: ['groundfloor'] },
|
'erdgeschosswohnung-mieten': { apartmenttypes: ['groundfloor'] },
|
||||||
@@ -144,7 +149,7 @@ export function convertWebToMobile(webUrl) {
|
|||||||
|
|
||||||
const realTypeKey = segments.at(-1);
|
const realTypeKey = segments.at(-1);
|
||||||
let realType = REAL_ESTATE_TYPE[realTypeKey];
|
let realType = REAL_ESTATE_TYPE[realTypeKey];
|
||||||
let additionalParamsFromWebPath;
|
let additionalParamsFromWebPath = WEB_PATH_TO_APARTMENT_EQUIPMENT_MAP[realTypeKey] || null;
|
||||||
|
|
||||||
if (!realType) {
|
if (!realType) {
|
||||||
// Test for seo optimized apartment path (only used on the ImmoScout web app)
|
// Test for seo optimized apartment path (only used on the ImmoScout web app)
|
||||||
@@ -165,7 +170,7 @@ export function convertWebToMobile(webUrl) {
|
|||||||
Object.entries(rawParams).filter(([key]) => key !== 'enteredFrom' && PARAM_NAME_MAP[key]),
|
Object.entries(rawParams).filter(([key]) => key !== 'enteredFrom' && PARAM_NAME_MAP[key]),
|
||||||
);
|
);
|
||||||
|
|
||||||
const geocodes = `/${segments.slice(2, 5).join('/')}`;
|
const geocodes = `/${segments.slice(2, segments.length - 1).join('/')}`;
|
||||||
const isRadius = segments.includes('radius');
|
const isRadius = segments.includes('radius');
|
||||||
const mobileParams = {
|
const mobileParams = {
|
||||||
searchType: isRadius ? 'radius' : 'region',
|
searchType: isRadius ? 'radius' : 'region',
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -13,6 +13,7 @@ import FredyPipelineExecutioner from '../../FredyPipelineExecutioner.js';
|
|||||||
import * as similarityCache from '../similarity-check/similarityCache.js';
|
import * as similarityCache from '../similarity-check/similarityCache.js';
|
||||||
import { isRunning, markFinished, markRunning } from './run-state.js';
|
import { isRunning, markFinished, markRunning } from './run-state.js';
|
||||||
import { sendToUsers } from '../sse/sse-broker.js';
|
import { sendToUsers } from '../sse/sse-broker.js';
|
||||||
|
import * as puppeteerExtractor from '../extractor/puppeteerExtractor.js';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initializes the job execution service.
|
* Initializes the job execution service.
|
||||||
@@ -94,7 +95,7 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
|
|||||||
* @param {{userId?: string, isAdmin?: boolean}} [context] - Who requested the run; determines job filtering.
|
* @param {{userId?: string, isAdmin?: boolean}} [context] - Who requested the run; determines job filtering.
|
||||||
* @returns {void}
|
* @returns {void}
|
||||||
*/
|
*/
|
||||||
function runAll(respectWorkingHours = true, context = undefined) {
|
async function runAll(respectWorkingHours = true, context = undefined) {
|
||||||
if (settings.demoMode) return;
|
if (settings.demoMode) return;
|
||||||
const now = Date.now();
|
const now = Date.now();
|
||||||
const withinHours = duringWorkingHoursOrNotSet(settings, now);
|
const withinHours = duringWorkingHoursOrNotSet(settings, now);
|
||||||
@@ -103,15 +104,18 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
settings.lastRun = now;
|
settings.lastRun = now;
|
||||||
jobStorage
|
const jobs = jobStorage
|
||||||
.getJobs()
|
.getJobs()
|
||||||
.filter((job) => job.enabled)
|
.filter((job) => job.enabled)
|
||||||
.filter((job) => {
|
.filter((job) => {
|
||||||
if (!context) return true; // startup/cron → all
|
if (!context) return true; // startup/cron → all
|
||||||
if (context.isAdmin) return true; // admin → all
|
if (context.isAdmin) return true; // admin → all
|
||||||
return context.userId ? job.userId === context.userId : false; // user → own
|
return context.userId ? job.userId === context.userId : false; // user → own
|
||||||
})
|
});
|
||||||
.forEach((job) => executeJob(job));
|
|
||||||
|
for (const job of jobs) {
|
||||||
|
await executeJob(job);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -154,28 +158,43 @@ export function initJobExecutionService({ providers, settings, intervalMs }) {
|
|||||||
} catch (err) {
|
} catch (err) {
|
||||||
logger.warn('Failed to emit start status for job', job.id, err);
|
logger.warn('Failed to emit start status for job', job.id, err);
|
||||||
}
|
}
|
||||||
|
let browser;
|
||||||
try {
|
try {
|
||||||
const jobProviders = job.provider.filter(
|
const jobProviders = job.provider.filter(
|
||||||
(p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
|
(p) => providers.find((loaded) => loaded.metaInformation.id === p.id) != null,
|
||||||
);
|
);
|
||||||
const executions = jobProviders.map(async (prov) => {
|
for (const prov of jobProviders) {
|
||||||
const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
|
try {
|
||||||
matchedProvider.init(prov, job.blacklist);
|
const matchedProvider = providers.find((loaded) => loaded.metaInformation.id === prov.id);
|
||||||
await new FredyPipelineExecutioner(
|
matchedProvider.init({ ...prov, userId: job.userId }, job.blacklist);
|
||||||
matchedProvider.config,
|
|
||||||
job.notificationAdapter,
|
if (browser && !browser.isConnected()) {
|
||||||
prov.id,
|
logger.debug('Browser is disconnected, nullifying to launch a new one.');
|
||||||
job.id,
|
await puppeteerExtractor.closeBrowser(browser);
|
||||||
similarityCache,
|
browser = null;
|
||||||
).execute();
|
}
|
||||||
});
|
|
||||||
const results = await Promise.allSettled(executions);
|
if (!browser && matchedProvider.config.getListings == null) {
|
||||||
for (const r of results) {
|
browser = await puppeteerExtractor.launchBrowser(matchedProvider.config.url, {});
|
||||||
if (r.status === 'rejected') {
|
}
|
||||||
logger.error(r.reason);
|
|
||||||
|
await new FredyPipelineExecutioner(
|
||||||
|
matchedProvider.config,
|
||||||
|
job.notificationAdapter,
|
||||||
|
job.spatialFilter,
|
||||||
|
prov.id,
|
||||||
|
job.id,
|
||||||
|
similarityCache,
|
||||||
|
browser,
|
||||||
|
).execute();
|
||||||
|
} catch (err) {
|
||||||
|
logger.error(err);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} finally {
|
} finally {
|
||||||
|
if (browser) {
|
||||||
|
await puppeteerExtractor.closeBrowser(browser);
|
||||||
|
}
|
||||||
markFinished(job.id);
|
markFinished(job.id);
|
||||||
try {
|
try {
|
||||||
bus.emit('jobs:status', { jobId: job.id, running: false });
|
bus.emit('jobs:status', { jobId: job.id, running: false });
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
35
lib/services/listings/distanceCalculator.js
Normal file
35
lib/services/listings/distanceCalculator.js
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
const R = 6371000; // Earth radius in meters
|
||||||
|
/**
|
||||||
|
* Calculate the great-circle distance between two points on Earth using the Haversine formula.
|
||||||
|
* This is to calculate the distance between the listing address & the address provided by the user. I know, it is only
|
||||||
|
* a rough estimation as this calculates the distance as a straight line, but it's more convenient than using an external
|
||||||
|
* service and still gives a good approximation for sorting purposes.
|
||||||
|
* Returns distance in meters.
|
||||||
|
*
|
||||||
|
* @param {number} lat1
|
||||||
|
* @param {number} lon1
|
||||||
|
* @param {number} lat2
|
||||||
|
* @param {number} lon2
|
||||||
|
* @returns {number}
|
||||||
|
*/
|
||||||
|
export function distanceMeters(lat1, lon1, lat2, lon2) {
|
||||||
|
const toRad = (deg) => (deg * Math.PI) / 180;
|
||||||
|
|
||||||
|
const phi1 = toRad(lat1);
|
||||||
|
const phi2 = toRad(lat2);
|
||||||
|
const dPhi = toRad(lat2 - lat1);
|
||||||
|
const dLambda = toRad(lon2 - lon1);
|
||||||
|
|
||||||
|
const a =
|
||||||
|
Math.sin(dPhi / 2) * Math.sin(dPhi / 2) +
|
||||||
|
Math.cos(phi1) * Math.cos(phi2) * Math.sin(dLambda / 2) * Math.sin(dLambda / 2);
|
||||||
|
|
||||||
|
const c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
|
||||||
|
|
||||||
|
return Math.round(R * c * 10) / 10;
|
||||||
|
}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -8,37 +8,71 @@ import { randomBetween, sleep } from '../../utils.js';
|
|||||||
|
|
||||||
const maxAttempts = 3;
|
const maxAttempts = 3;
|
||||||
|
|
||||||
|
const userAgents = [
|
||||||
|
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
|
||||||
|
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
|
||||||
|
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
|
||||||
|
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.2 Safari/605.1.15',
|
||||||
|
'Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1',
|
||||||
|
];
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Check if a listing is still active with up to 3 attempts and exponential backoff.
|
* Check if a listing is still active with up to 5 attempts and exponential backoff.
|
||||||
* Backoff waits are capped and the last wait is at most 2000 ms.
|
* Backoff waits are randomized and capped.
|
||||||
*
|
*
|
||||||
* Rules:
|
* Rules:
|
||||||
* - HTTP 200 => return 1
|
* - HTTP 200 => return 1 (if checkForText is provided and found, returns 0)
|
||||||
* - HTTP 401/403 => return -1 (most certainly detected as a bot)
|
* - HTTP 401/403 => return -1 (most certainly detected as a bot)
|
||||||
* - HTTP 404 => return 0
|
* - HTTP 404 => return 0
|
||||||
* - Other statuses or network errors => retry until attempts are exhausted
|
* - Other statuses or network errors => retry until attempts are exhausted
|
||||||
*
|
*
|
||||||
* @returns {Promise<Integer>} 1 if active, o if not active and -1 if detected as bot
|
* @returns {Promise<Integer>} 1 if active, 0 if not active and -1 if detected as bot
|
||||||
*/
|
*/
|
||||||
export default async function checkIfListingIsActive(link) {
|
export default async function checkIfListingIsActive(link, checkForText = null) {
|
||||||
await sleep(randomBetween(50, 100));
|
await sleep(randomBetween(50, 100));
|
||||||
|
|
||||||
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
|
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
|
||||||
try {
|
try {
|
||||||
|
const userAgent = userAgents[Math.floor(Math.random() * userAgents.length)];
|
||||||
const res = await fetch(link, {
|
const res = await fetch(link, {
|
||||||
|
redirect: 'manual',
|
||||||
headers: {
|
headers: {
|
||||||
'User-Agent':
|
'User-Agent': userAgent,
|
||||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
|
Accept:
|
||||||
'Accept-Language': 'de-DE,de;q=0.9,en;q=0.8',
|
'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
|
||||||
|
'Accept-Language': 'de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7',
|
||||||
|
'Accept-Encoding': 'gzip, deflate, br',
|
||||||
|
'Cache-Control': 'max-age=0',
|
||||||
|
'Sec-Ch-Ua': '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"',
|
||||||
|
'Sec-Ch-Ua-Mobile': '?0',
|
||||||
|
'Sec-Ch-Ua-Platform': '"macOS"',
|
||||||
|
'Sec-Fetch-Dest': 'document',
|
||||||
|
'Sec-Fetch-Mode': 'navigate',
|
||||||
|
'Sec-Fetch-Site': 'none',
|
||||||
|
'Sec-Fetch-User': '?1',
|
||||||
|
'Upgrade-Insecure-Requests': '1',
|
||||||
|
Referer: 'https://www.google.com/',
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
if (res.status === 200) {
|
if (res.status === 200) {
|
||||||
|
if (checkForText) {
|
||||||
|
const htmText = await res.text();
|
||||||
|
if (htmText.includes(checkForText)) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
if (res.status === 401) return -1;
|
if (res.status === 401 || res.status === 403) {
|
||||||
if (res.status === 403) return -1;
|
if (attempt < maxAttempts) {
|
||||||
if (res.status === 404) return 0;
|
await sleep(backoffDelay(attempt));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (res.status === 404 || res.status === 410) return 0;
|
||||||
|
|
||||||
// For any other status, only retry if attempts remain
|
// For any other status, only retry if attempts remain
|
||||||
if (attempt < maxAttempts) {
|
if (attempt < maxAttempts) {
|
||||||
@@ -61,13 +95,13 @@ export default async function checkIfListingIsActive(link) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Exponential backoff delay with cap.
|
* Exponential backoff delay with cap and jitter.
|
||||||
* attempt: 1 -> 500ms, 2 -> 1000ms, 3 -> 2000ms (cap)
|
|
||||||
* @param {number} attempt 1-based attempt index
|
* @param {number} attempt 1-based attempt index
|
||||||
* @returns {number} delay in ms
|
* @returns {number} delay in ms
|
||||||
*/
|
*/
|
||||||
function backoffDelay(attempt) {
|
function backoffDelay(attempt) {
|
||||||
const base = 500;
|
const base = 500;
|
||||||
const cap = 2000;
|
const cap = 2000;
|
||||||
return Math.min(base * 2 ** (attempt - 1), cap);
|
const delay = Math.min(base * 2 ** (attempt - 1), cap);
|
||||||
|
return delay + randomBetween(0, 1000);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -34,9 +34,8 @@ class SqliteConnection {
|
|||||||
|
|
||||||
static async init() {
|
static async init() {
|
||||||
if (this.#sqlLiteCfg == null) {
|
if (this.#sqlLiteCfg == null) {
|
||||||
readConfigFromStorage().then((c) => {
|
const c = await readConfigFromStorage();
|
||||||
this.#sqlLiteCfg = c.sqlitepath;
|
this.#sqlLiteCfg = c.sqlitepath;
|
||||||
});
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -30,6 +30,7 @@ export const upsertJob = ({
|
|||||||
notificationAdapter,
|
notificationAdapter,
|
||||||
userId,
|
userId,
|
||||||
shareWithUsers = [],
|
shareWithUsers = [],
|
||||||
|
spatialFilter = null,
|
||||||
}) => {
|
}) => {
|
||||||
const id = jobId || nanoid();
|
const id = jobId || nanoid();
|
||||||
const existing = SqliteConnection.query(`SELECT id, user_id FROM jobs WHERE id = @id LIMIT 1`, { id })[0];
|
const existing = SqliteConnection.query(`SELECT id, user_id FROM jobs WHERE id = @id LIMIT 1`, { id })[0];
|
||||||
@@ -37,12 +38,13 @@ export const upsertJob = ({
|
|||||||
if (existing) {
|
if (existing) {
|
||||||
SqliteConnection.execute(
|
SqliteConnection.execute(
|
||||||
`UPDATE jobs
|
`UPDATE jobs
|
||||||
SET enabled = @enabled,
|
SET enabled = @enabled,
|
||||||
name = @name,
|
name = @name,
|
||||||
blacklist = @blacklist,
|
blacklist = @blacklist,
|
||||||
provider = @provider,
|
provider = @provider,
|
||||||
notification_adapter = @notification_adapter,
|
notification_adapter = @notification_adapter,
|
||||||
shared_with_user = @shareWithUsers
|
shared_with_user = @shareWithUsers,
|
||||||
|
spatial_filter = @spatialFilter
|
||||||
WHERE id = @id`,
|
WHERE id = @id`,
|
||||||
{
|
{
|
||||||
id,
|
id,
|
||||||
@@ -52,12 +54,13 @@ export const upsertJob = ({
|
|||||||
shareWithUsers: toJson(shareWithUsers ?? []),
|
shareWithUsers: toJson(shareWithUsers ?? []),
|
||||||
provider: toJson(provider ?? []),
|
provider: toJson(provider ?? []),
|
||||||
notification_adapter: toJson(notificationAdapter ?? []),
|
notification_adapter: toJson(notificationAdapter ?? []),
|
||||||
|
spatialFilter: spatialFilter ? toJson(spatialFilter) : null,
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
} else {
|
} else {
|
||||||
SqliteConnection.execute(
|
SqliteConnection.execute(
|
||||||
`INSERT INTO jobs (id, user_id, enabled, name, blacklist, provider, notification_adapter, shared_with_user)
|
`INSERT INTO jobs (id, user_id, enabled, name, blacklist, provider, notification_adapter, shared_with_user, spatial_filter)
|
||||||
VALUES (@id, @user_id, @enabled, @name, @blacklist, @provider, @notification_adapter, @shareWithUsers)`,
|
VALUES (@id, @user_id, @enabled, @name, @blacklist, @provider, @notification_adapter, @shareWithUsers, @spatialFilter)`,
|
||||||
{
|
{
|
||||||
id,
|
id,
|
||||||
user_id: ownerId,
|
user_id: ownerId,
|
||||||
@@ -67,6 +70,7 @@ export const upsertJob = ({
|
|||||||
provider: toJson(provider ?? []),
|
provider: toJson(provider ?? []),
|
||||||
shareWithUsers: toJson(shareWithUsers ?? []),
|
shareWithUsers: toJson(shareWithUsers ?? []),
|
||||||
notification_adapter: toJson(notificationAdapter ?? []),
|
notification_adapter: toJson(notificationAdapter ?? []),
|
||||||
|
spatialFilter: spatialFilter ? toJson(spatialFilter) : null,
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -87,10 +91,11 @@ export const getJob = (jobId) => {
|
|||||||
j.provider,
|
j.provider,
|
||||||
j.shared_with_user,
|
j.shared_with_user,
|
||||||
j.notification_adapter AS notificationAdapter,
|
j.notification_adapter AS notificationAdapter,
|
||||||
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id) AS numberOfFoundListings
|
j.spatial_filter AS spatialFilter,
|
||||||
FROM jobs j
|
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id AND l.is_active = 1 AND l.manually_deleted = 0) AS numberOfFoundListings
|
||||||
WHERE j.id = @id
|
FROM jobs j
|
||||||
LIMIT 1`,
|
WHERE j.id = @id
|
||||||
|
LIMIT 1`,
|
||||||
{ id: jobId },
|
{ id: jobId },
|
||||||
)[0];
|
)[0];
|
||||||
if (!row) return null;
|
if (!row) return null;
|
||||||
@@ -101,6 +106,7 @@ export const getJob = (jobId) => {
|
|||||||
provider: fromJson(row.provider, []),
|
provider: fromJson(row.provider, []),
|
||||||
shared_with_user: fromJson(row.shared_with_user, []),
|
shared_with_user: fromJson(row.shared_with_user, []),
|
||||||
notificationAdapter: fromJson(row.notificationAdapter, []),
|
notificationAdapter: fromJson(row.notificationAdapter, []),
|
||||||
|
spatialFilter: fromJson(row.spatialFilter, null),
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -150,9 +156,11 @@ export const getJobs = () => {
|
|||||||
j.provider,
|
j.provider,
|
||||||
j.shared_with_user,
|
j.shared_with_user,
|
||||||
j.notification_adapter AS notificationAdapter,
|
j.notification_adapter AS notificationAdapter,
|
||||||
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id) AS numberOfFoundListings
|
j.spatial_filter AS spatialFilter,
|
||||||
FROM jobs j
|
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id AND l.is_active = 1 AND l.manually_deleted = 0) AS numberOfFoundListings
|
||||||
ORDER BY j.name IS NULL, j.name`,
|
FROM jobs j
|
||||||
|
WHERE j.enabled = 1
|
||||||
|
ORDER BY j.name IS NULL, j.name`,
|
||||||
);
|
);
|
||||||
return rows.map((row) => ({
|
return rows.map((row) => ({
|
||||||
...row,
|
...row,
|
||||||
@@ -161,6 +169,7 @@ export const getJobs = () => {
|
|||||||
provider: fromJson(row.provider, []),
|
provider: fromJson(row.provider, []),
|
||||||
shared_with_user: fromJson(row.shared_with_user, []),
|
shared_with_user: fromJson(row.shared_with_user, []),
|
||||||
notificationAdapter: fromJson(row.notificationAdapter, []),
|
notificationAdapter: fromJson(row.notificationAdapter, []),
|
||||||
|
spatialFilter: fromJson(row.spatialFilter, null),
|
||||||
}));
|
}));
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -189,7 +198,7 @@ export const queryJobs = ({
|
|||||||
isAdmin = false,
|
isAdmin = false,
|
||||||
} = {}) => {
|
} = {}) => {
|
||||||
// sanitize inputs
|
// sanitize inputs
|
||||||
const safePageSize = Number.isFinite(pageSize) && pageSize > 0 ? Math.min(500, Math.floor(pageSize)) : 50;
|
const safePageSize = Number.isFinite(pageSize) && pageSize > 0 ? Math.min(1000, Math.floor(pageSize)) : 50;
|
||||||
const safePage = Number.isFinite(page) && page > 0 ? Math.floor(page) : 1;
|
const safePage = Number.isFinite(page) && page > 0 ? Math.floor(page) : 1;
|
||||||
const offset = (safePage - 1) * safePageSize;
|
const offset = (safePage - 1) * safePageSize;
|
||||||
|
|
||||||
@@ -250,11 +259,12 @@ export const queryJobs = ({
|
|||||||
j.provider,
|
j.provider,
|
||||||
j.shared_with_user,
|
j.shared_with_user,
|
||||||
j.notification_adapter AS notificationAdapter,
|
j.notification_adapter AS notificationAdapter,
|
||||||
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id) AS numberOfFoundListings
|
j.spatial_filter AS spatialFilter,
|
||||||
|
(SELECT COUNT(1) FROM listings l WHERE l.job_id = j.id AND l.is_active = 1 AND l.manually_deleted = 0) AS numberOfFoundListings
|
||||||
FROM jobs j
|
FROM jobs j
|
||||||
${whereSql}
|
${whereSql}
|
||||||
${orderSql}
|
${orderSql}
|
||||||
LIMIT @limit OFFSET @offset`,
|
LIMIT @limit OFFSET @offset`,
|
||||||
params,
|
params,
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -265,6 +275,7 @@ export const queryJobs = ({
|
|||||||
provider: fromJson(row.provider, []),
|
provider: fromJson(row.provider, []),
|
||||||
shared_with_user: fromJson(row.shared_with_user, []),
|
shared_with_user: fromJson(row.shared_with_user, []),
|
||||||
notificationAdapter: fromJson(row.notificationAdapter, []),
|
notificationAdapter: fromJson(row.notificationAdapter, []),
|
||||||
|
spatialFilter: fromJson(row.spatialFilter, null),
|
||||||
}));
|
}));
|
||||||
|
|
||||||
return { totalNumber, page: safePage, result };
|
return { totalNumber, page: safePage, result };
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -48,7 +48,8 @@ export const getListingsKpisForJobIds = (jobIds = []) => {
|
|||||||
SUM(CASE WHEN is_active = 1 THEN 1 ELSE 0 END) AS activeCount,
|
SUM(CASE WHEN is_active = 1 THEN 1 ELSE 0 END) AS activeCount,
|
||||||
AVG(price) AS avgPrice
|
AVG(price) AS avgPrice
|
||||||
FROM listings
|
FROM listings
|
||||||
WHERE job_id IN (${placeholders})`,
|
WHERE job_id IN (${placeholders})
|
||||||
|
AND manually_deleted = 0`,
|
||||||
jobIds,
|
jobIds,
|
||||||
)[0] || {};
|
)[0] || {};
|
||||||
|
|
||||||
@@ -80,6 +81,7 @@ export const getProviderDistributionForJobIds = (jobIds = []) => {
|
|||||||
`SELECT provider, COUNT(*) AS cnt
|
`SELECT provider, COUNT(*) AS cnt
|
||||||
FROM listings
|
FROM listings
|
||||||
WHERE job_id IN (${placeholders})
|
WHERE job_id IN (${placeholders})
|
||||||
|
AND manually_deleted = 0
|
||||||
GROUP BY provider
|
GROUP BY provider
|
||||||
ORDER BY cnt DESC`,
|
ORDER BY cnt DESC`,
|
||||||
jobIds,
|
jobIds,
|
||||||
@@ -118,8 +120,8 @@ export const getActiveOrUnknownListings = () => {
|
|||||||
return SqliteConnection.query(
|
return SqliteConnection.query(
|
||||||
`SELECT *
|
`SELECT *
|
||||||
FROM listings
|
FROM listings
|
||||||
WHERE is_active is null
|
WHERE (is_active is null OR is_active = 1)
|
||||||
OR is_active = 1
|
AND manually_deleted = 0
|
||||||
ORDER BY provider`,
|
ORDER BY provider`,
|
||||||
);
|
);
|
||||||
};
|
};
|
||||||
@@ -173,9 +175,9 @@ export const storeListings = (jobId, providerId, listings) => {
|
|||||||
SqliteConnection.withTransaction((db) => {
|
SqliteConnection.withTransaction((db) => {
|
||||||
const stmt = db.prepare(
|
const stmt = db.prepare(
|
||||||
`INSERT INTO listings (id, hash, provider, job_id, price, size, title, image_url, description, address,
|
`INSERT INTO listings (id, hash, provider, job_id, price, size, title, image_url, description, address,
|
||||||
link, created_at, is_active)
|
link, created_at, is_active, latitude, longitude)
|
||||||
VALUES (@id, @hash, @provider, @job_id, @price, @size, @title, @image_url, @description, @address, @link,
|
VALUES (@id, @hash, @provider, @job_id, @price, @size, @title, @image_url, @description, @address, @link,
|
||||||
@created_at, 1)
|
@created_at, 1, @latitude, @longitude)
|
||||||
ON CONFLICT(job_id, hash) DO NOTHING`,
|
ON CONFLICT(job_id, hash) DO NOTHING`,
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -193,6 +195,8 @@ export const storeListings = (jobId, providerId, listings) => {
|
|||||||
address: removeParentheses(item.address),
|
address: removeParentheses(item.address),
|
||||||
link: item.link,
|
link: item.link,
|
||||||
created_at: Date.now(),
|
created_at: Date.now(),
|
||||||
|
latitude: item.latitude || null,
|
||||||
|
longitude: item.longitude || null,
|
||||||
};
|
};
|
||||||
stmt.run(params);
|
stmt.run(params);
|
||||||
}
|
}
|
||||||
@@ -238,6 +242,8 @@ export const storeListings = (jobId, providerId, listings) => {
|
|||||||
* @param {object} [params.watchListFilter]
|
* @param {object} [params.watchListFilter]
|
||||||
* @param {string|null} [params.sortField=null] - One of: 'created_at','price','size','provider','title'.
|
* @param {string|null} [params.sortField=null] - One of: 'created_at','price','size','provider','title'.
|
||||||
* @param {('asc'|'desc')} [params.sortDir='asc']
|
* @param {('asc'|'desc')} [params.sortDir='asc']
|
||||||
|
* @param {number} [params.createdAfter] - Only include listings created at or after this unix timestamp (ms).
|
||||||
|
* @param {number} [params.createdBefore] - Only include listings created at or before this unix timestamp (ms).
|
||||||
* @param {string} [params.userId] - Current user id used to scope listings (ignored for admins).
|
* @param {string} [params.userId] - Current user id used to scope listings (ignored for admins).
|
||||||
* @param {boolean} [params.isAdmin=false] - When true, returns all listings.
|
* @param {boolean} [params.isAdmin=false] - When true, returns all listings.
|
||||||
* @returns {{ totalNumber:number, page:number, result:Object[] }}
|
* @returns {{ totalNumber:number, page:number, result:Object[] }}
|
||||||
@@ -253,11 +259,15 @@ export const queryListings = ({
|
|||||||
freeTextFilter,
|
freeTextFilter,
|
||||||
sortField = null,
|
sortField = null,
|
||||||
sortDir = 'asc',
|
sortDir = 'asc',
|
||||||
|
createdAfter = null,
|
||||||
|
createdBefore = null,
|
||||||
|
minPrice = null,
|
||||||
|
maxPrice = null,
|
||||||
userId = null,
|
userId = null,
|
||||||
isAdmin = false,
|
isAdmin = false,
|
||||||
} = {}) => {
|
} = {}) => {
|
||||||
// sanitize inputs
|
// sanitize inputs
|
||||||
const safePageSize = Number.isFinite(pageSize) && pageSize > 0 ? Math.min(500, Math.floor(pageSize)) : 50;
|
const safePageSize = Number.isFinite(pageSize) && pageSize > 0 ? Math.min(1000, Math.floor(pageSize)) : 50;
|
||||||
const safePage = Number.isFinite(page) && page > 0 ? Math.floor(page) : 1;
|
const safePage = Number.isFinite(page) && page > 0 ? Math.floor(page) : 1;
|
||||||
const offset = (safePage - 1) * safePageSize;
|
const offset = (safePage - 1) * safePageSize;
|
||||||
|
|
||||||
@@ -303,6 +313,27 @@ export const queryListings = ({
|
|||||||
} else if (watchListFilter === false) {
|
} else if (watchListFilter === false) {
|
||||||
whereParts.push('(wl.id IS NULL)');
|
whereParts.push('(wl.id IS NULL)');
|
||||||
}
|
}
|
||||||
|
// Time range filters (unix timestamps in milliseconds)
|
||||||
|
if (Number.isFinite(createdAfter) && createdAfter > 0) {
|
||||||
|
params.createdAfter = createdAfter;
|
||||||
|
whereParts.push('(created_at >= @createdAfter)');
|
||||||
|
}
|
||||||
|
if (Number.isFinite(createdBefore) && createdBefore > 0) {
|
||||||
|
params.createdBefore = createdBefore;
|
||||||
|
whereParts.push('(created_at <= @createdBefore)');
|
||||||
|
}
|
||||||
|
// Price range filters
|
||||||
|
if (Number.isFinite(minPrice) && minPrice >= 0) {
|
||||||
|
params.minPrice = minPrice;
|
||||||
|
whereParts.push('(l.price >= @minPrice)');
|
||||||
|
}
|
||||||
|
if (Number.isFinite(maxPrice) && maxPrice >= 0) {
|
||||||
|
params.maxPrice = maxPrice;
|
||||||
|
whereParts.push('(l.price <= @maxPrice)');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build whereSql (filtering by manually_deleted = 0)
|
||||||
|
whereParts.push('(l.manually_deleted = 0)');
|
||||||
|
|
||||||
const whereSql = whereParts.length ? `WHERE ${whereParts.join(' AND ')}` : '';
|
const whereSql = whereParts.length ? `WHERE ${whereParts.join(' AND ')}` : '';
|
||||||
const whereSqlWithAlias = whereSql
|
const whereSqlWithAlias = whereSql
|
||||||
@@ -363,13 +394,21 @@ export const queryListings = ({
|
|||||||
* Delete all listings for a given job id.
|
* Delete all listings for a given job id.
|
||||||
*
|
*
|
||||||
* @param {string} jobId - The job identifier whose listings should be removed.
|
* @param {string} jobId - The job identifier whose listings should be removed.
|
||||||
* @returns {any} The result from SqliteConnection.execute (may contain changes count).
|
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
|
||||||
|
* @returns {any} The result from SqliteConnection.execute.
|
||||||
*/
|
*/
|
||||||
export const deleteListingsByJobId = (jobId) => {
|
export const deleteListingsByJobId = (jobId, hardDelete = false) => {
|
||||||
if (!jobId) return;
|
if (!jobId) return;
|
||||||
|
if (hardDelete) {
|
||||||
|
return SqliteConnection.execute(
|
||||||
|
`DELETE FROM listings
|
||||||
|
WHERE job_id = @jobId`,
|
||||||
|
{ jobId },
|
||||||
|
);
|
||||||
|
}
|
||||||
return SqliteConnection.execute(
|
return SqliteConnection.execute(
|
||||||
`DELETE
|
`UPDATE listings
|
||||||
FROM listings
|
SET manually_deleted = 1
|
||||||
WHERE job_id = @jobId`,
|
WHERE job_id = @jobId`,
|
||||||
{ jobId },
|
{ jobId },
|
||||||
);
|
);
|
||||||
@@ -379,19 +418,107 @@ export const deleteListingsByJobId = (jobId) => {
|
|||||||
* Delete listings by a list of listing IDs.
|
* Delete listings by a list of listing IDs.
|
||||||
*
|
*
|
||||||
* @param {string[]} ids - Array of listing IDs to delete.
|
* @param {string[]} ids - Array of listing IDs to delete.
|
||||||
|
* @param {boolean} [hardDelete=false] - Whether to hard delete from DB or just mark as deleted.
|
||||||
* @returns {any} The result from SqliteConnection.execute.
|
* @returns {any} The result from SqliteConnection.execute.
|
||||||
*/
|
*/
|
||||||
export const deleteListingsById = (ids) => {
|
export const deleteListingsById = (ids, hardDelete = false) => {
|
||||||
if (!Array.isArray(ids) || ids.length === 0) return;
|
if (!Array.isArray(ids) || ids.length === 0) return;
|
||||||
const placeholders = ids.map(() => '?').join(',');
|
const placeholders = ids.map(() => '?').join(',');
|
||||||
|
if (hardDelete) {
|
||||||
|
return SqliteConnection.execute(
|
||||||
|
`DELETE FROM listings
|
||||||
|
WHERE id IN (${placeholders})`,
|
||||||
|
ids,
|
||||||
|
);
|
||||||
|
}
|
||||||
return SqliteConnection.execute(
|
return SqliteConnection.execute(
|
||||||
`DELETE
|
`UPDATE listings
|
||||||
FROM listings
|
SET manually_deleted = 1
|
||||||
WHERE id IN (${placeholders})`,
|
WHERE id IN (${placeholders})`,
|
||||||
ids,
|
ids,
|
||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return all listings that are active, have an address, and do not yet have geocoordinates.
|
||||||
|
*
|
||||||
|
* @returns {Object[]} Array of listing objects {id, address}.
|
||||||
|
*/
|
||||||
|
export const getListingsToGeocode = () => {
|
||||||
|
return SqliteConnection.query(
|
||||||
|
`SELECT id, address
|
||||||
|
FROM listings
|
||||||
|
WHERE is_active = 1
|
||||||
|
AND manually_deleted = 0
|
||||||
|
AND address IS NOT NULL
|
||||||
|
AND (latitude IS NULL OR longitude IS NULL)`,
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Update the geocoordinates for a listing.
|
||||||
|
*
|
||||||
|
* @param {string} id - The listing ID.
|
||||||
|
* @param {number} latitude
|
||||||
|
* @param {number} longitude
|
||||||
|
* @returns {void}
|
||||||
|
*/
|
||||||
|
export const updateListingGeocoordinates = (id, latitude, longitude) => {
|
||||||
|
SqliteConnection.execute(
|
||||||
|
`UPDATE listings
|
||||||
|
SET latitude = @latitude,
|
||||||
|
longitude = @longitude
|
||||||
|
WHERE id = @id`,
|
||||||
|
{ id, latitude, longitude },
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return listings with geocoordinates for the map view, with optional filtering.
|
||||||
|
*
|
||||||
|
* @param {Object} params
|
||||||
|
* @param {string} [params.jobId]
|
||||||
|
* @param {string} [params.userId]
|
||||||
|
* @param {boolean} [params.isAdmin=false]
|
||||||
|
* @returns {{listings: Object[], maxPrice: number}} Object containing listings and maxPrice.
|
||||||
|
*/
|
||||||
|
export const getListingsForMap = ({ jobId, userId = null, isAdmin = false } = {}) => {
|
||||||
|
const baseWhereParts = [
|
||||||
|
'l.latitude IS NOT NULL',
|
||||||
|
'l.longitude IS NOT NULL',
|
||||||
|
'l.latitude != -1',
|
||||||
|
'l.longitude != -1',
|
||||||
|
'l.is_active = 1',
|
||||||
|
'l.manually_deleted = 0',
|
||||||
|
];
|
||||||
|
const params = { userId: userId || '__NO_USER__' };
|
||||||
|
|
||||||
|
if (!isAdmin) {
|
||||||
|
baseWhereParts.push(
|
||||||
|
`(j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (jobId) {
|
||||||
|
params.jobId = jobId;
|
||||||
|
baseWhereParts.push('l.job_id = @jobId');
|
||||||
|
}
|
||||||
|
|
||||||
|
const wherePartsForListings = [...baseWhereParts];
|
||||||
|
|
||||||
|
const listings = SqliteConnection.query(
|
||||||
|
`SELECT l.*, j.name AS job_name
|
||||||
|
FROM listings l
|
||||||
|
LEFT JOIN jobs j ON j.id = l.job_id
|
||||||
|
WHERE ${wherePartsForListings.join(' AND ')}`,
|
||||||
|
params,
|
||||||
|
);
|
||||||
|
|
||||||
|
return {
|
||||||
|
listings,
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Return all listings with only the fields: title, address, and price.
|
* Return all listings with only the fields: title, address, and price.
|
||||||
* This is the single helper requested for simple consumers.
|
* This is the single helper requested for simple consumers.
|
||||||
@@ -399,5 +526,129 @@ export const deleteListingsById = (ids) => {
|
|||||||
* @returns {{title: string|null, address: string|null, price: number|null}[]}
|
* @returns {{title: string|null, address: string|null, price: number|null}[]}
|
||||||
*/
|
*/
|
||||||
export const getAllEntriesFromListings = () => {
|
export const getAllEntriesFromListings = () => {
|
||||||
return SqliteConnection.query(`SELECT title, address, price FROM listings`);
|
return SqliteConnection.query(`SELECT title, address, price FROM listings WHERE manually_deleted = 0`);
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return geocoordinates for a given address if it has been geocoded before.
|
||||||
|
*
|
||||||
|
* @param {string} address
|
||||||
|
* @returns {{lat: number, lng: number}|null}
|
||||||
|
*/
|
||||||
|
export const getGeocoordinatesByAddress = (address) => {
|
||||||
|
const row = SqliteConnection.query(
|
||||||
|
`SELECT latitude, longitude
|
||||||
|
FROM listings
|
||||||
|
WHERE address = @address
|
||||||
|
AND manually_deleted = 0
|
||||||
|
AND latitude IS NOT NULL
|
||||||
|
AND longitude IS NOT NULL
|
||||||
|
AND latitude != -1
|
||||||
|
AND longitude != -1
|
||||||
|
LIMIT 1`,
|
||||||
|
{ address },
|
||||||
|
)[0];
|
||||||
|
return row ? { lat: row.latitude, lng: row.longitude } : null;
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return all active listings for a given job that have geocoordinates but no distance set.
|
||||||
|
*
|
||||||
|
* @param {string} jobId
|
||||||
|
* @returns {Object[]}
|
||||||
|
*/
|
||||||
|
export const getListingsToCalculateDistance = (jobId) => {
|
||||||
|
return SqliteConnection.query(
|
||||||
|
`SELECT id, latitude, longitude
|
||||||
|
FROM listings
|
||||||
|
WHERE job_id = @jobId
|
||||||
|
AND is_active = 1
|
||||||
|
AND manually_deleted = 0
|
||||||
|
AND latitude IS NOT NULL
|
||||||
|
AND longitude IS NOT NULL
|
||||||
|
AND distance_to_destination IS NULL`,
|
||||||
|
{ jobId },
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return all active listings for a given user (across all jobs) that have geocoordinates.
|
||||||
|
*
|
||||||
|
* @param {string} userId
|
||||||
|
* @returns {Object[]}
|
||||||
|
*/
|
||||||
|
export const getListingsForUserToCalculateDistance = (userId) => {
|
||||||
|
return SqliteConnection.query(
|
||||||
|
`SELECT l.id, l.latitude, l.longitude
|
||||||
|
FROM listings l
|
||||||
|
JOIN jobs j ON l.job_id = j.id
|
||||||
|
WHERE j.user_id = @userId
|
||||||
|
AND l.is_active = 1
|
||||||
|
AND l.manually_deleted = 0
|
||||||
|
AND l.latitude IS NOT NULL
|
||||||
|
AND l.longitude IS NOT NULL`,
|
||||||
|
{ userId },
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Update the distance to destination for a listing.
|
||||||
|
*
|
||||||
|
* @param {string} id
|
||||||
|
* @param {number} distance
|
||||||
|
* @returns {void}
|
||||||
|
*/
|
||||||
|
export const updateListingDistance = (id, distance) => {
|
||||||
|
SqliteConnection.execute(
|
||||||
|
`UPDATE listings
|
||||||
|
SET distance_to_destination = @distance
|
||||||
|
WHERE id = @id`,
|
||||||
|
{ id, distance },
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return a single listing by id.
|
||||||
|
*
|
||||||
|
* @param {string} id
|
||||||
|
* @param {string} userId
|
||||||
|
* @param {boolean} isAdmin
|
||||||
|
* @returns {Object|null}
|
||||||
|
*/
|
||||||
|
export const getListingById = (id, userId = null, isAdmin = false) => {
|
||||||
|
const params = { id, userId: userId || '__NO_USER__' };
|
||||||
|
let whereScoping = '';
|
||||||
|
if (!isAdmin) {
|
||||||
|
whereScoping = `AND (j.user_id = @userId OR EXISTS (SELECT 1 FROM json_each(j.shared_with_user) AS sw WHERE sw.value = @userId))`;
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
SqliteConnection.query(
|
||||||
|
`SELECT l.*, j.name AS job_name, CASE WHEN wl.id IS NOT NULL THEN 1 ELSE 0 END AS isWatched
|
||||||
|
FROM listings l
|
||||||
|
LEFT JOIN jobs j ON j.id = l.job_id
|
||||||
|
LEFT JOIN watch_list wl ON wl.listing_id = l.id AND wl.user_id = @userId
|
||||||
|
WHERE l.id = @id AND l.manually_deleted = 0 ${whereScoping}`,
|
||||||
|
params,
|
||||||
|
)[0] || null
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Resets geocoordinates and distance for all listings related to a user.
|
||||||
|
*
|
||||||
|
* @param {string} userId
|
||||||
|
* @returns {void}
|
||||||
|
*/
|
||||||
|
export const resetGeocoordinatesAndDistanceForUser = (userId) => {
|
||||||
|
SqliteConnection.execute(
|
||||||
|
`UPDATE listings
|
||||||
|
SET latitude = NULL,
|
||||||
|
longitude = NULL,
|
||||||
|
distance_to_destination = NULL
|
||||||
|
WHERE job_id IN (
|
||||||
|
SELECT id FROM jobs j
|
||||||
|
WHERE j.user_id = @userId
|
||||||
|
)`,
|
||||||
|
{ userId },
|
||||||
|
);
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -88,7 +88,7 @@ export function up(db) {
|
|||||||
}
|
}
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
// If parsing fails, let it throw to rollback the migration
|
// If parsing fails, let it throw to rollback the migration
|
||||||
throw new Error(`Failed to import users from ${usersJsonPath}: ${e.message}`);
|
throw new Error(`Failed to import users from ${usersJsonPath}: ${e.message}`, { cause: e });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -116,7 +116,7 @@ export function up(db) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
throw new Error(`Failed to import jobs from ${jobsJsonPath}: ${e.message}`);
|
throw new Error(`Failed to import jobs from ${jobsJsonPath}: ${e.message}`, { cause: e });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,16 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
export function up(db) {
|
||||||
|
// 1. Add manually_deleted column
|
||||||
|
db.exec(`ALTER TABLE listings ADD COLUMN manually_deleted INTEGER NOT NULL DEFAULT 0;`);
|
||||||
|
|
||||||
|
// 2. Remove change_set column
|
||||||
|
try {
|
||||||
|
db.exec(`ALTER TABLE listings DROP COLUMN change_set;`);
|
||||||
|
} catch {
|
||||||
|
// if column does not exists for whatever reason
|
||||||
|
}
|
||||||
|
}
|
||||||
11
lib/services/storage/migrations/sql/11.add-spatial-filter.js
Normal file
11
lib/services/storage/migrations/sql/11.add-spatial-filter.js
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
// Migration: Add spatial_filter column to jobs table for storing GeoJSON-based spatial filters
|
||||||
|
export function up(db) {
|
||||||
|
db.exec(`
|
||||||
|
ALTER TABLE jobs ADD COLUMN spatial_filter JSONB DEFAULT NULL;
|
||||||
|
`);
|
||||||
|
}
|
||||||
25
lib/services/storage/migrations/sql/11.mcp-tokens.js
Normal file
25
lib/services/storage/migrations/sql/11.mcp-tokens.js
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
|
*/
|
||||||
|
import crypto from 'crypto';
|
||||||
|
|
||||||
|
// Migration: Add mcp_token column to users table.
|
||||||
|
// Each user gets a permanent, non-expiring secret token used for MCP API authentication.
|
||||||
|
// Tokens are auto-generated for all existing users during this migration.
|
||||||
|
export function up(db) {
|
||||||
|
db.exec(`ALTER TABLE users ADD COLUMN mcp_token TEXT`);
|
||||||
|
|
||||||
|
// Backfill all existing users that don't have a token yet
|
||||||
|
const users = db.prepare(`SELECT id FROM users WHERE mcp_token IS NULL`).all();
|
||||||
|
const update = db.prepare(`UPDATE users SET mcp_token = @token WHERE id = @id`);
|
||||||
|
for (const user of users) {
|
||||||
|
const token = `fredy_${crypto.randomBytes(32).toString('hex')}`;
|
||||||
|
update.run({ id: user.id, token });
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
/*
|
/*
|
||||||
* Copyright (c) 2025 by Christian Kellner.
|
* Copyright (c) 2026 by Christian Kellner.
|
||||||
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user