Compare commits

..

25 Commits

Author SHA1 Message Date
Christian Kellner
206f768b41 next version 2025-07-25 13:21:12 +02:00
Alexander Roidl
2302f69ff3 Rename NPM startup scripts (#144)
* feat: rename npm start scripts
2025-07-25 13:13:04 +02:00
Alexander Roidl
9bb33e723a Workflow to check sourcecode's linting and formatting (#146)
* ci: workflow to check sourcecode

* fix: make workflow to check source fail for incorrect linting/formatting

* ci: change step name for workflow to check sourcecode
2025-07-23 08:58:43 +02:00
Alexander Roidl
cca1463a68 chore: run formatter (#145) 2025-07-23 08:47:26 +02:00
Alexander Roidl
314b1818d7 Formatting and linting pre-commit hook (#143) 2025-07-22 21:39:52 +02:00
Christian Kellner
25cc7fb650 next release version 2025-07-22 20:01:01 +02:00
Alexander Roidl
78df4b21a6 Remove leading commas from listings in Telegram messages (#142) 2025-07-22 19:58:16 +02:00
weakmap@gmail.com
d89b078237 lol 2025-07-19 22:41:30 +02:00
weakmap@gmail.com
395199a4a2 fixing duplicate provider removal / ugrade dependencies 2025-07-19 20:10:19 +02:00
weakmap@gmail.com
c2680fe49f next release version 2025-06-14 19:26:17 +02:00
weakmap@gmail.com
2b862b2d98 fixing blacklist 2025-06-14 19:25:52 +02:00
weakmap@gmail.com
9065448b6b upgrade dependencies 2025-06-14 19:12:55 +02:00
weakmap@gmail.com
b9f49cb5b2 upgrade dependencies 2025-06-14 19:06:27 +02:00
weakmap@gmail.com
53121742c2 improving error message 2025-06-14 19:03:23 +02:00
Christian Kellner
1a3eae0390 next version 2025-06-04 09:47:42 +02:00
Christian Kellner
a42905d63f fixing docker ignore issue 2025-06-04 09:46:07 +02:00
Christian Kellner
9917491728 Merge branch 'master' of github.com:orangecoding/fredy 2025-06-04 09:29:50 +02:00
Christian Kellner
f032e6a724 test: verify unrelated text yields no similarity (#130) 2025-06-04 09:15:53 +02:00
Christian Kellner
111c154ae3 Fix job ownership verification (#132) 2025-06-04 09:15:36 +02:00
Christian Kellner
2194ffe0f4 Fix typo in README (#133) 2025-06-04 09:15:15 +02:00
Christian Kellner
cfa25fc0e0 docs: fix adapter sentence (#131) 2025-06-04 09:14:57 +02:00
Christian Kellner
d50dd61f3e Merge branch 'master' of github.com:orangecoding/fredy 2025-06-04 09:12:00 +02:00
Christian Kellner
31e7f77bde uprade restana & vite 2025-05-27 12:01:26 +02:00
Christian Kellner
a418d64f1a uprade dependencies 2025-05-27 11:51:57 +02:00
Christian Kellner
d099872950 Update README.md 2025-05-26 13:23:36 +02:00
43 changed files with 1672 additions and 1338 deletions

View File

@@ -2,5 +2,6 @@ node_modules/
npm-debug.log
test/
db/
conf/
.git/
.github/

26
.github/workflows/check_source.yml vendored Normal file
View File

@@ -0,0 +1,26 @@
name: Check the source code
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
check_source_code:
name: Check the source code
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: 'yarn'
- name: Install dependencies
run: yarn install
- name: Check formatting
run: yarn format:check
- name: Lint
run: yarn lint

View File

@@ -1,4 +1,4 @@
name: "Close stale issues and PRs"
name: Close stale issues and PRs
on:
schedule:

View File

@@ -13,8 +13,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: 'yarn'

4
.prettierrc Normal file
View File

@@ -0,0 +1,4 @@
{
"singleQuote": true,
"printWidth": 120
}

View File

@@ -106,14 +106,14 @@ exports.config = {
```
#### Running Tests
If you've written a new provider you are an awesome person. If you now write tests for it, you are even more awesome. And who doesn't want to be more awesome right?
If you've written a new provider you are an awesome person. If you now write tests for it, you are even more awesome. And who doesn't want to be more awesome, right?
#### Codestyle
I'm using Eslint to maintain quote style and quality. Do not skip it...
I'm using ESLint to maintain quote style and quality. Do not skip it...
##### To do before merging:
##### To-do before merging:
- executed tests? (`pnpm test`)
- sure the changes are useful for everybody? Or is it maybe a custom modification just for your case?
- Have you executed the tests? (`yarn test`)
- Are you sure the changes are useful for everybody? Or is it maybe a custom modification just for your case?
_Thanks!_ :heart:

View File

@@ -11,16 +11,16 @@ ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
# Copy lockfiles first to leverage cache for dependencies
COPY package.json yarn.lock ./
COPY package.json yarn.lock .
# Set Yarn timeout, install dependencies and PM2 globally
RUN yarn config set network-timeout 600000 \
&& yarn install --frozen-lockfile \
&& yarn --frozen-lockfile \
&& yarn global add pm2
# Copy application source and build production assets
COPY . ./
RUN yarn run prod
COPY . .
RUN yarn build:frontend
# Prepare runtime directories and symlinks for data and config
RUN mkdir -p /db /conf \

View File

@@ -1,6 +1,6 @@
<img src="https://github.com/orangecoding/fredy/blob/master/doc/logo.png" width="400">
![Build Status](https://github.com/orangecoding/fredy/actions/workflows/test.yml/badge.svg) [![Create and publish Docker image](https://github.com/orangecoding/fredy/actions/workflows/docker.yml/badge.svg)](https://github.com/orangecoding/fredy/actions/workflows/docker.yml)
![Test](https://github.com/orangecoding/fredy/actions/workflows/test.yml/badge.svg) [![Create and publish Docker image](https://github.com/orangecoding/fredy/actions/workflows/docker.yml/badge.svg)](https://github.com/orangecoding/fredy/actions/workflows/docker.yml) ![Check the sourcecode](https://github.com/orangecoding/fredy/actions/workflows/check_source.yml/badge.svg)
Searching an apartment in Germany can be a frustrating task. Not any longer though, as _Fredy_ will take over and will only notify you once new listings have been found that match your requirements.
@@ -8,8 +8,6 @@ _Fredy_ scrapes multiple services (Immonet, Immowelt etc.) and send new listings
If _Fredy_ finds matching results, it will send them to you via Slack, Email, Telegram etc. (More adapters can be configured.) As _Fredy_ stores the listings it has found, new results will not be sent to you twice (and as a side-effect, _Fredy_ can show some statistics). Furthermore, _Fredy_ checks duplicates per scraping so that the same listings are not being sent twice or more when posted on various platforms (which happens more often than one might think).
<a href="https://www.producthunt.com/posts/fredy-find-real-estates-damn-easy?embed=true&utm_source=badge-featured&utm_medium=badge&utm_source=badge-fredy&#0045;find&#0045;real&#0045;estates&#0045;damn&#0045;easy" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=965690&theme=light&t=1747292331626" alt="Fredy&#0032;&#0045;&#0032;Find&#0032;Real&#0032;Estates&#0032;Damn&#0032;EasY&#0032; - Your&#0032;personal&#0032;real&#0032;estate&#0032;search&#0032;bot | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
# Sponsorship [![](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=%23fe8e86)](https://github.com/sponsors/orangecoding)
If you like my work, consider becoming a sponsor. I'm not expecting anybody to pay for _Fredy_ or any other Open Source Project I'm maintaining, however keep in mind, I'm doing all of this in my spare time :) Thanks.
@@ -25,9 +23,9 @@ If you want to try out _Fredy_, you can access the demo version [here](https://f
- Make sure to use Node.js 20 or above
- Run the following commands:
```ssh
yarn (or npm install)
yarn run prod
yarn run start
yarn
yarn run start:backend
yarn run start:frontend
```
_Fredy_ will start with the default port, set to `9998`. You can access _Fredy_ by opening your browser at `http://localhost:9998`. The default login is `admin`, both for username and password. You should change the password as soon as possible when you plan to run Fredy on a server.
@@ -48,7 +46,7 @@ A provider contains the URL that points to the search results for the respective
**It is important that you order the search results by date, so that _Fredy_ always picks the latest results first!**
#### Adapter
_Fredy_ supports multiple adapters, such as Slack, SendGrid, Telegram etc. A search job can have as many adapters as supported by _Fredy_. Each adapter needs different configuration values, which you have to provide when using them. A adapter dictactes how the frontend renders by telling the frontend what information it needs in order to send listings to the user.
_Fredy_ supports multiple adapters, such as Slack, SendGrid, Telegram etc. A search job can have as many adapters as supported by _Fredy_. Each adapter needs different configuration values, which you have to provide when using them. An adapter dictates how the frontend renders by telling the frontend what information it needs in order to send listings to the user.
#### Jobs
A Job wraps adapters and providers. _Fredy_ runs the configured jobs in a specific interval (can be configured in `/conf/config.json`).
@@ -63,14 +61,13 @@ As an administrator, you can create, edit and remove users from _Fredy_. Be care
# Development
### Running Fredy in development mode
To run _Fredy_ in development mode, you need to run the backend & frontend separately.
Start the backend with:
```shell
yarn run start
yarn run start:backend:dev
```
For the frontend, run:
```shell
yarn run dev
yarn run start:frontend:dev
```
You should now be able to access _Fredy_ from your browser. Check your Terminal to see what port the frontend is running on.
@@ -84,7 +81,7 @@ yarn run test
![Architecture](/doc/architecture.jpg "Architecture")
### Immoscout
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
# Analytics
Fredy is completely free (and will always remain free). However, it would be a huge help if youd allow me to collect some analytical data.

View File

@@ -1,14 +1,14 @@
import fs from 'fs';
import {config} from './lib/utils.js';
import { config } from './lib/utils.js';
import * as similarityCache from './lib/services/similarity-check/similarityCache.js';
import { setLastJobExecution } from './lib/services/storage/listingsStorage.js';
import * as jobStorage from './lib/services/storage/jobStorage.js';
import FredyRuntime from './lib/FredyRuntime.js';
import { duringWorkingHoursOrNotSet } from './lib/utils.js';
import './lib/api/api.js';
import {track} from './lib/services/tracking/Tracker.js';
import {handleDemoUser} from './lib/services/storage/userStorage.js';
import {cleanupDemoAtMidnight} from './lib/services/demoCleanup.js';
import { track } from './lib/services/tracking/Tracker.js';
import { handleDemoUser } from './lib/services/storage/userStorage.js';
import { cleanupDemoAtMidnight } from './lib/services/demoCleanup.js';
//if db folder does not exist, ensure to create it before loading anything else
if (!fs.existsSync('./db')) {
fs.mkdirSync('./db');
@@ -19,13 +19,13 @@ const provider = fs.readdirSync(path).filter((file) => file.endsWith('.js'));
const INTERVAL = config.interval * 60 * 1000;
/* eslint-disable no-console */
console.log(`Started Fredy successfully. Ui can be accessed via http://localhost:${config.port}`);
if(config.demoMode){
console.info('Running in demo mode');
cleanupDemoAtMidnight();
if (config.demoMode) {
console.info('Running in demo mode');
cleanupDemoAtMidnight();
}
/* eslint-enable no-console */
const fetchedProvider = await Promise.all(
provider.filter((provider) => provider.endsWith('.js')).map(async (pro) => import(`${path}/${pro}`))
provider.filter((provider) => provider.endsWith('.js')).map(async (pro) => import(`${path}/${pro}`)),
);
handleDemoUser();
@@ -33,30 +33,30 @@ handleDemoUser();
setInterval(
(function exec() {
const isDuringWorkingHoursOrNotSet = duringWorkingHoursOrNotSet(config, Date.now());
if(!config.demoMode) {
if (isDuringWorkingHoursOrNotSet) {
track();
config.lastRun = Date.now();
jobStorage
.getJobs()
.filter((job) => job.enabled)
.forEach((job) => {
job.provider
.filter((p) => fetchedProvider.find((fp) => fp.metaInformation.id === p.id) != null)
.forEach(async (prov) => {
const pro = fetchedProvider.find((fp) => fp.metaInformation.id === prov.id);
pro.init(prov, job.blacklist);
await new FredyRuntime(pro.config, job.notificationAdapter, prov.id, job.id, similarityCache).execute();
setLastJobExecution(job.id);
});
});
} else {
/* eslint-disable no-console */
console.debug('Working hours set. Skipping as outside of working hours.');
/* eslint-enable no-console */
}
}
if (!config.demoMode) {
if (isDuringWorkingHoursOrNotSet) {
track();
config.lastRun = Date.now();
jobStorage
.getJobs()
.filter((job) => job.enabled)
.forEach((job) => {
job.provider
.filter((p) => fetchedProvider.find((fp) => fp.metaInformation.id === p.id) != null)
.forEach(async (prov) => {
const pro = fetchedProvider.find((fp) => fp.metaInformation.id === prov.id);
pro.init(prov, job.blacklist);
await new FredyRuntime(pro.config, job.notificationAdapter, prov.id, job.id, similarityCache).execute();
setLastJobExecution(job.id);
});
});
} else {
/* eslint-disable no-console */
console.debug('Working hours set. Skipping as outside of working hours.');
/* eslint-enable no-console */
}
}
return exec;
})(),
INTERVAL
INTERVAL,
);

View File

@@ -12,7 +12,7 @@ import restana from 'restana';
import files from 'serve-static';
import path from 'path';
import { getDirName } from '../utils.js';
import {demoRouter} from './routes/demoRouter.js';
import { demoRouter } from './routes/demoRouter.js';
const service = restana();
const staticService = files(path.join(getDirName(), '../ui/public'));
const PORT = config.port || 9998;

View File

@@ -15,7 +15,7 @@ function doesJobBelongsToUser(job, req) {
if (user == null) {
return false;
}
return user.isAdmin || job.userId === job.userId;
return user.isAdmin || job.userId === user.id;
}
jobRouter.get('/', async (req, res) => {
const isUserAdmin = isAdmin(req);

View File

@@ -22,16 +22,18 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey }) =
const { token, chatId } = notificationConfig.find((adapter) => adapter.id === config.id).fields;
const job = getJob(jobKey);
const jobName = job == null ? jobKey : job.name;
//we have to split messages into chunk, because otherwise messages are going to become too big and will fail
// we have to split messages into chunks, because otherwise messages are going to become too big and will fail
const chunks = arrayChunks(newListings, MAX_ENTITIES_PER_CHUNK);
const promises = chunks.map((chunk) => {
let message = `<i>${jobName}</i> (${serviceName}) found <b>${newListings.length}</b> new listings:\n\n`;
message += chunk.map(
const messageParagraphs = [];
messageParagraphs.push(`<i>${jobName}</i> (${serviceName}) found <b>${newListings.length}</b> new listings:`);
messageParagraphs.push(...chunk.map(
(o) =>
`<a href='${o.link}'><b>${shorten(o.title.replace(/\*/g, ''), 45).trim()}</b></a>\n` +
[o.address, o.price, o.size].join(' | ') +
'\n\n',
);
[o.address, o.price, o.size].join(' | ')
));
/**
* This is to not break the rate limit. It is to only send 1 message per second
*/
@@ -41,7 +43,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey }) =
method: 'post',
body: JSON.stringify({
chat_id: chatId,
text: message,
text: messageParagraphs.join('\n\n'),
parse_mode: 'HTML',
disable_web_page_preview: true,
}),

View File

@@ -6,7 +6,7 @@ const adapter = await Promise.all(
fs
.readdirSync('./lib/notification/adapter')
.filter((file) => file.endsWith('.js'))
.map(async (integPath) => await import(`${path}/${integPath}`))
.map(async (integPath) => await import(`${path}/${integPath}`)),
);
if (adapter.length === 0) {

View File

@@ -1,4 +1,4 @@
import utils, {buildHash} from '../utils.js';
import utils, { buildHash } from '../utils.js';
let appliedBlackList = [];
function shortenLink(link) {
return link.substring(0, link.indexOf('?'));

View File

@@ -36,7 +36,7 @@
*/
import utils, { buildHash } from '../utils.js';
import { convertWebToMobile } from '../services/immoscout/immoscout-web-translater.js';
import { convertWebToMobile } from '../services/immoscout/immoscout-web-translator.js';
let appliedBlackList = [];
async function getListings(url) {

View File

@@ -1,50 +1,50 @@
import utils, {buildHash} from '../utils.js';
import utils, { buildHash } from '../utils.js';
let appliedBlackList = [];
let appliedBlacklistedDistricts = [];
function normalize(o) {
const size = o.size || '--- m²';
const id = buildHash(o.id, o.price);
const link = `https://www.kleinanzeigen.de${o.link}`;
return Object.assign(o, {id, size, link});
const size = o.size || '--- m²';
const id = buildHash(o.id, o.price);
const link = `https://www.kleinanzeigen.de${o.link}`;
return Object.assign(o, { id, size, link });
}
function applyBlacklist(o) {
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
const isBlacklistedDistrict =
appliedBlacklistedDistricts.length === 0 ? false : utils.isOneOf(o.description, appliedBlacklistedDistricts);
return o.title != null && !isBlacklistedDistrict && titleNotBlacklisted && descNotBlacklisted;
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
const isBlacklistedDistrict =
appliedBlacklistedDistricts.length === 0 ? false : utils.isOneOf(o.description, appliedBlacklistedDistricts);
return o.title != null && !isBlacklistedDistrict && titleNotBlacklisted && descNotBlacklisted;
}
const config = {
url: null,
crawlContainer: '#srchrslt-adtable .ad-listitem ',
//sort by date is standard oO
sortByDateParam: null,
waitForSelector: 'body',
crawlFields: {
id: '.aditem@data-adid | int',
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
size: '.aditem-main .text-module-end | removeNewline | trim',
title: '.aditem-main .text-module-begin a | removeNewline | trim',
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
address: '.aditem-main--top--left | trim | removeNewline',
},
normalize: normalize,
filter: applyBlacklist,
url: null,
crawlContainer: '#srchrslt-adtable .ad-listitem ',
//sort by date is standard oO
sortByDateParam: null,
waitForSelector: 'body',
crawlFields: {
id: '.aditem@data-adid | int',
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
size: '.aditem-main .text-module-end | removeNewline | trim',
title: '.aditem-main .text-module-begin a | removeNewline | trim',
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
address: '.aditem-main--top--left | trim | removeNewline',
},
normalize: normalize,
filter: applyBlacklist,
};
export const metaInformation = {
name: 'Ebay Kleinanzeigen',
baseUrl: 'https://www.kleinanzeigen.de/',
id: 'kleinanzeigen',
name: 'Ebay Kleinanzeigen',
baseUrl: 'https://www.kleinanzeigen.de/',
id: 'kleinanzeigen',
};
export const init = (sourceConfig, blacklist, blacklistedDistricts) => {
config.enabled = sourceConfig.enabled;
config.url = sourceConfig.url;
appliedBlacklistedDistricts = blacklistedDistricts || [];
appliedBlackList = blacklist || [];
config.enabled = sourceConfig.enabled;
config.url = sourceConfig.url;
appliedBlacklistedDistricts = blacklistedDistricts || [];
appliedBlackList = blacklist || [];
};
export {config};
export { config };

View File

@@ -1,44 +1,46 @@
import utils, {buildHash} from '../utils.js';
import utils, { buildHash } from '../utils.js';
let appliedBlackList = [];
function nullOrEmpty(val) {
return val == null || val.length === 0;
return val == null || val.length === 0;
}
function normalize(o) {
const link = nullOrEmpty(o.link) ? 'NO LINK' : `https://www.neubaukompass.de${o.link.substring(o.link.indexOf('/neubau'))}`;
const id = buildHash(o.link, o.price);
return Object.assign(o, {id, link});
const link = nullOrEmpty(o.link)
? 'NO LINK'
: `https://www.neubaukompass.de${o.link.substring(o.link.indexOf('/neubau'))}`;
const id = buildHash(o.link, o.price);
return Object.assign(o, { id, link });
}
function applyBlacklist(o) {
return !utils.isOneOf(o.title, appliedBlackList);
return !utils.isOneOf(o.title, appliedBlackList);
}
const config = {
url: null,
crawlContainer: '.col-12.mb-4',
sortByDateParam: 'Sortierung=Id&Richtung=DESC',
waitForSelector: '.nbk-section',
crawlFields: {
id: 'a@href',
title: 'a@title | removeNewline | trim',
link: 'a@href',
address: '.nbk-project-card__description | removeNewline | trim',
price: '.nbk-project-card__spec-item .nbk-project-card__spec-value | removeNewline | trim',
},
normalize: normalize,
filter: applyBlacklist,
url: null,
crawlContainer: '.col-12.mb-4',
sortByDateParam: 'Sortierung=Id&Richtung=DESC',
waitForSelector: '.nbk-section',
crawlFields: {
id: 'a@href',
title: 'a@title | removeNewline | trim',
link: 'a@href',
address: '.nbk-project-card__description | removeNewline | trim',
price: '.nbk-project-card__spec-item .nbk-project-card__spec-value | removeNewline | trim',
},
normalize: normalize,
filter: applyBlacklist,
};
export const init = (sourceConfig, blacklist) => {
config.enabled = sourceConfig.enabled;
config.url = sourceConfig.url;
appliedBlackList = blacklist || [];
config.enabled = sourceConfig.enabled;
config.url = sourceConfig.url;
appliedBlackList = blacklist || [];
};
export const metaInformation = {
name: 'Neubau Kompass',
baseUrl: 'https://www.neubaukompass.de/',
id: 'neubauKompass',
name: 'Neubau Kompass',
baseUrl: 'https://www.neubaukompass.de/',
id: 'neubauKompass',
};
export {config};
export { config };

View File

@@ -1,43 +1,43 @@
import utils, {buildHash} from '../utils.js';
import utils, { buildHash } from '../utils.js';
let appliedBlackList = [];
function normalize(o) {
const id = buildHash(o.id, o.price);
const link = `https://www.wg-gesucht.de${o.link}`;
return Object.assign(o, { id, link });
const id = buildHash(o.id, o.price);
const link = `https://www.wg-gesucht.de${o.link}`;
return Object.assign(o, { id, link });
}
function applyBlacklist(o) {
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
return o.id != null && titleNotBlacklisted && descNotBlacklisted;
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
return o.id != null && titleNotBlacklisted && descNotBlacklisted;
}
const config = {
url: null,
crawlContainer: '#main_column .wgg_card',
sortByDateParam: 'sort_column=0&sort_order=0',
waitForSelector: 'body',
crawlFields: {
id: '@data-id',
details: '.row .noprint .col-xs-11 |removeNewline |trim',
price: '.middle .col-xs-3 |removeNewline |trim',
size: '.middle .text-right |removeNewline |trim',
title: '.truncate_title a |removeNewline |trim',
link: '.truncate_title a@href',
},
normalize: normalize,
filter: applyBlacklist,
url: null,
crawlContainer: '#main_column .wgg_card',
sortByDateParam: 'sort_column=0&sort_order=0',
waitForSelector: 'body',
crawlFields: {
id: '@data-id',
details: '.row .noprint .col-xs-11 |removeNewline |trim',
price: '.middle .col-xs-3 |removeNewline |trim',
size: '.middle .text-right |removeNewline |trim',
title: '.truncate_title a |removeNewline |trim',
link: '.truncate_title a@href',
},
normalize: normalize,
filter: applyBlacklist,
};
export const init = (sourceConfig, blacklist) => {
config.enabled = sourceConfig.enabled;
config.url = sourceConfig.url;
appliedBlackList = blacklist || [];
config.enabled = sourceConfig.enabled;
config.url = sourceConfig.url;
appliedBlackList = blacklist || [];
};
export const metaInformation = {
name: 'Wg gesucht',
baseUrl: 'https://www.wg-gesucht.de/',
id: 'wgGesucht',
name: 'Wg gesucht',
baseUrl: 'https://www.wg-gesucht.de/',
id: 'wgGesucht',
};
export {config};
export { config };

View File

@@ -8,7 +8,7 @@ export function loadParser(text) {
export function parse(crawlContainer, crawlFields, text, url) {
if (!text) {
console.warn('Cannot parse, text was empty for url ', url);
console.warn('No content found for ', url);
return null;
}

View File

@@ -1,5 +1,5 @@
import { JSONFileSync } from 'lowdb/node';
import {config, getDirName} from '../../utils.js';
import { config, getDirName } from '../../utils.js';
import * as hasher from '../security/hash.js';
import { nanoid } from 'nanoid';
import * as jobStorage from './jobStorage.js';
@@ -7,23 +7,23 @@ import path from 'path';
import LowdashAdapter from './LowDashAdapter.js';
const defaultData = {
user: [
//you probably want to change the default password ;)
{
id: nanoid(),
lastLogin: Date.now(),
username: 'admin',
password: hasher.hash('admin'),
isAdmin: true,
},
{
id: nanoid(),
lastLogin: Date.now(),
username: 'demo',
password: hasher.hash('demo'),
isAdmin: true,
},
],
user: [
//you probably want to change the default password ;)
{
id: nanoid(),
lastLogin: Date.now(),
username: 'admin',
password: hasher.hash('admin'),
isAdmin: true,
},
{
id: nanoid(),
lastLogin: Date.now(),
username: 'demo',
password: hasher.hash('demo'),
isAdmin: true,
},
],
};
const file = path.join(getDirName(), '../', 'db/users.json');
@@ -86,34 +86,38 @@ export const removeUser = (userId) => {
db.chain
.set(
'user',
user.filter((u) => u.id !== userId)
user.filter((u) => u.id !== userId),
)
.value();
db.write();
};
export const handleDemoUser = () => {
if(!config.demoMode){
const user = db.chain.get('user').value();
db.chain.get('user').value();
db.chain.set('user', user.filter((u) => u.username !== 'demo')).value();
db.write();
}else {
const demoUser = db.chain
.get('user')
.filter((u) => u.username === 'demo')
.value();
if (demoUser == null || demoUser.length === 0) {
db.chain.get('user')
.value()
.push({
id: nanoid(),
username: 'demo',
password: hasher.hash('demo'),
isAdmin: true,
});
db.write();
}
if (!config.demoMode) {
const user = db.chain.get('user').value();
db.chain
.set(
'user',
user.filter((u) => u.username !== 'demo'),
)
.value();
db.write();
} else {
const demoUser = db.chain
.get('user')
.filter((u) => u.username === 'demo')
.value();
if (demoUser == null || demoUser.length === 0) {
db.chain
.get('user')
.value()
.push({
id: nanoid(),
username: 'demo',
password: hasher.hash('demo'),
isAdmin: true,
});
db.write();
}
}
};

View File

@@ -9,12 +9,9 @@ function inDevMode(){
}
function isOneOf(word, arr) {
if (arr == null || arr.length === 0) {
return false;
}
const expression = String.raw`\b(${arr.join('|')})\b`;
const blacklist = new RegExp(expression, 'ig');
return blacklist.test(word);
if (!arr || arr.length === 0 || word == null) return false;
const lowerWord = word.toLowerCase();
return arr.some(item => lowerWord.indexOf(item.toLowerCase()) !== -1);
}
function nullOrEmpty(val) {

View File

@@ -1,21 +1,25 @@
{
"name": "fredy",
"version": "11.2.2",
"version": "11.3.0",
"description": "[F]ind [R]eal [E]states [d]amn eas[y].",
"scripts": {
"start": "node prod.js",
"dev": "yarn && rm -rf ./ui/public/* && vite",
"ui": "rm -rf ./ui/public/* && vite",
"prod": "yarn && vite build --emptyOutDir",
"format": "prettier --write lib/**/*.js ui/src/**/*.jsx test/**/*.js *.js --single-quote --print-width 120",
"prepare": "husky",
"start:backend": "x-var NODE_ENV=production node index.js",
"start:backend:dev": "nodemon --watch index.js --watch lib",
"start:frontend": "vite -m production",
"start:frontend:dev": "vite",
"build:frontend": "vite build",
"format": "prettier --write lib/**/*.js ui/src/**/*.jsx test/**/*.js *.js",
"format:check": "prettier --check lib/**/*.js ui/src/**/*.jsx test/**/*.js *.js",
"test": "mocha --loader=esmock --timeout 3000000 test/**/*.test.js",
"lint": "eslint ./index.js ./lib/**/*.js ./test/**/*.js ./ui/src/**/*.jsx"
"lint": "eslint index.js lib/**/*.js test/**/*.js ui/src/**/*.jsx",
"lint:fix": "yarn lint --fix"
},
"type": "module",
"lint-staged": {
"*.js": [
"eslint ./index.js ./lib/**/*.js ./test/**/*.js",
"prettier --single-quote --print-width 120 --write"
"*.{js,jsx}": [
"yarn lint",
"yarn format"
]
},
"main": "index.js",
@@ -50,17 +54,17 @@
"Firefox ESR"
],
"dependencies": {
"@douyinfe/semi-ui": "2.79.0",
"@douyinfe/semi-ui": "2.83.0",
"@rematch/core": "2.2.0",
"@rematch/loading": "2.1.2",
"@sendgrid/mail": "8.1.5",
"@vitejs/plugin-react": "4.4.1",
"@vitejs/plugin-react": "4.7.0",
"better-sqlite3": "^11.10.0",
"body-parser": "2.2.0",
"cheerio": "^1.0.0",
"cookie-session": "2.1.0",
"cheerio": "^1.1.0",
"cookie-session": "2.1.1",
"handlebars": "4.7.8",
"highcharts": "12.2.0",
"highcharts": "12.3.0",
"highcharts-react-official": "3.2.2",
"lodash": "4.17.21",
"lowdb": "6.0.1",
@@ -70,10 +74,10 @@
"node-fetch": "3.3.2",
"node-mailjet": "6.0.8",
"package-up": "^5.0.0",
"puppeteer": "^24.8.2",
"puppeteer": "^24.14.0",
"puppeteer-extra": "^3.3.6",
"puppeteer-extra-plugin-stealth": "^2.11.2",
"query-string": "9.1.2",
"query-string": "9.2.2",
"react": "18.3.1",
"react-dom": "18.3.1",
"react-redux": "9.2.0",
@@ -81,28 +85,30 @@
"react-router-dom": "5.3.0",
"redux": "5.0.1",
"redux-thunk": "3.1.0",
"restana": "4.9.9",
"serve-static": "1.16.2",
"restana": "5.0.0",
"serve-static": "2.2.0",
"slack": "11.0.2",
"string-similarity": "^4.0.4",
"vite": "5.4.11"
"vite": "7.0.5",
"x-var": "^2.1.0"
},
"devDependencies": {
"@babel/core": "7.27.1",
"@babel/eslint-parser": "7.27.1",
"@babel/core": "7.27.3",
"@babel/eslint-parser": "7.27.5",
"@babel/preset-env": "7.27.2",
"@babel/preset-react": "7.27.1",
"chai": "5.2.0",
"chai": "5.2.1",
"eslint": "8.56.0",
"eslint-config-prettier": "8.8.0",
"eslint-plugin-react": "7.37.4",
"esmock": "2.7.0",
"eslint-plugin-react": "7.37.5",
"esmock": "2.7.1",
"history": "5.3.0",
"husky": "9.1.7",
"less": "4.3.0",
"less": "4.4.0",
"lint-staged": "15.5.2",
"mocha": "10.8.2",
"prettier": "3.5.3",
"nodemon": "^3.1.10",
"prettier": "3.6.2",
"redux-logger": "3.0.6"
}
}

View File

@@ -1,2 +0,0 @@
process.env.NODE_ENV = 'production';
import('./index.js');

View File

@@ -77,4 +77,4 @@ curl -H "User-Agent: ImmoScout24_1410_30_._" \
## Parameters
The parameters between web and mobile are very different which is why we have to translate them. Please see `immoscout-web-translator.js`.
The parameters between web and mobile are very different which is why we have to translate them. Please see [/lib/services/immoscout/immoscout-web-translator.js](https://github.com/orangecoding/fredy/blob/master/lib/services/immoscout/immoscout-web-translator.js).

View File

@@ -1,38 +1,38 @@
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
import {get} from '../mocks/mockNotification.js';
import {mockFredy, providerConfig} from '../utils.js';
import {expect} from 'chai';
import { get } from '../mocks/mockNotification.js';
import { mockFredy, providerConfig } from '../utils.js';
import { expect } from 'chai';
import * as provider from '../../lib/provider/immonet.js';
describe('#immonet testsuite()', () => {
after(() => {
similarityCache.stopCacheCleanup();
});
provider.init(providerConfig.immonet, [], []);
it('should test immonet provider', async () => {
const Fredy = await mockFredy();
return await new Promise((resolve) => {
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immonet', similarityCache);
fredy.execute().then((listing) => {
expect(listing).to.be.a('array');
const notificationObj = get();
expect(notificationObj).to.be.a('object');
expect(notificationObj.serviceName).to.equal('immonet');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).to.be.a('string');
expect(notify.price).to.be.a('string');
expect(notify.size).to.be.a('string');
expect(notify.title).to.be.a('string');
expect(notify.link).to.be.a('string');
expect(notify.address).to.be.a('string');
after(() => {
similarityCache.stopCacheCleanup();
});
provider.init(providerConfig.immonet, [], []);
it('should test immonet provider', async () => {
const Fredy = await mockFredy();
return await new Promise((resolve) => {
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immonet', similarityCache);
fredy.execute().then((listing) => {
expect(listing).to.be.a('array');
const notificationObj = get();
expect(notificationObj).to.be.a('object');
expect(notificationObj.serviceName).to.equal('immonet');
notificationObj.payload.forEach((notify) => {
/** check the actual structure **/
expect(notify.id).to.be.a('string');
expect(notify.price).to.be.a('string');
expect(notify.size).to.be.a('string');
expect(notify.title).to.be.a('string');
expect(notify.link).to.be.a('string');
expect(notify.address).to.be.a('string');
expect(notify.size).that.does.include('m²');
expect(notify.title).to.be.not.empty;
expect(notify.address).to.be.not.empty;
});
resolve();
});
expect(notify.size).that.does.include('m²');
expect(notify.title).to.be.not.empty;
expect(notify.address).to.be.not.empty;
});
resolve();
});
});
});
});

View File

@@ -1,36 +1,36 @@
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
import {get} from '../mocks/mockNotification.js';
import {mockFredy, providerConfig} from '../utils.js';
import {expect} from 'chai';
import { get } from '../mocks/mockNotification.js';
import { mockFredy, providerConfig } from '../utils.js';
import { expect } from 'chai';
import * as provider from '../../lib/provider/neubauKompass.js';
describe('#neubauKompass testsuite()', () => {
after(() => {
similarityCache.stopCacheCleanup();
});
provider.init(providerConfig.neubauKompass, [], []);
it('should test neubauKompass provider', async () => {
const Fredy = await mockFredy();
return await new Promise((resolve) => {
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'neubauKompass', similarityCache);
fredy.execute().then((listing) => {
expect(listing).to.be.a('array');
const notificationObj = get();
expect(notificationObj.serviceName).to.equal('neubauKompass');
notificationObj.payload.forEach((notify) => {
expect(notify).to.be.a('object');
/** check the actual structure **/
expect(notify.id).to.be.a('string');
expect(notify.title).to.be.a('string');
expect(notify.link).to.be.a('string');
expect(notify.address).to.be.a('string');
/** check the values if possible **/
expect(notify.title).to.be.not.empty;
expect(notify.link).that.does.include('https://www.neubaukompass.de');
expect(notify.address).to.be.not.empty;
});
resolve();
});
after(() => {
similarityCache.stopCacheCleanup();
});
provider.init(providerConfig.neubauKompass, [], []);
it('should test neubauKompass provider', async () => {
const Fredy = await mockFredy();
return await new Promise((resolve) => {
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'neubauKompass', similarityCache);
fredy.execute().then((listing) => {
expect(listing).to.be.a('array');
const notificationObj = get();
expect(notificationObj.serviceName).to.equal('neubauKompass');
notificationObj.payload.forEach((notify) => {
expect(notify).to.be.a('object');
/** check the actual structure **/
expect(notify.id).to.be.a('string');
expect(notify.title).to.be.a('string');
expect(notify.link).to.be.a('string');
expect(notify.address).to.be.a('string');
/** check the values if possible **/
expect(notify.title).to.be.not.empty;
expect(notify.link).that.does.include('https://www.neubaukompass.de');
expect(notify.address).to.be.not.empty;
});
resolve();
});
});
});
});

View File

@@ -34,6 +34,7 @@ describe('similarityCheck', () => {
check.setCacheEntry(
'where |X| and |Y| are the cardinalities of the two sets (i.e. the number of elements in each set). The Sørensen index equals twice the number of elements common to both sets divided by the sum of the number of elements in each set.',
);
expect(check.hasSimilarEntries('unrelated text')).to.be.false;
});
});
});

View File

@@ -1,16 +1,16 @@
import { expect } from 'chai';
import {buildHash} from '../../lib/utils.js';
import { buildHash } from '../../lib/utils.js';
describe('utilsCheck', () => {
describe('#utilsCheck()', () => {
it('should be null when null input', () => {
expect(buildHash(null)).to.be.null;
});
it('should be null when null empty', () => {
expect(buildHash('')).to.be.null;
});
it('should return a value', () => {
expect(buildHash('bla', '', null)).to.be.a.string;
});
it('should be null when null input', () => {
expect(buildHash(null)).to.be.null;
});
it('should be null when null empty', () => {
expect(buildHash('')).to.be.null;
});
it('should return a value', () => {
expect(buildHash('bla', '', null)).to.be.a.string;
});
});
});

View File

@@ -1,4 +1,4 @@
import React, {useEffect} from 'react';
import React, { useEffect } from 'react';
import InsufficientPermission from './components/permission/InsufficientPermission';
import PermissionAwareRoute from './components/permission/PermissionAwareRoute';
@@ -6,106 +6,108 @@ import GeneralSettings from './views/generalSettings/GeneralSettings';
import JobMutation from './views/jobs/mutation/JobMutation';
import UserMutator from './views/user/mutation/UserMutator';
import JobInsight from './views/jobs/insights/JobInsight.jsx';
import {useDispatch, useSelector} from 'react-redux';
import {Switch, Redirect} from 'react-router-dom';
import { useDispatch, useSelector } from 'react-redux';
import { Switch, Redirect } from 'react-router-dom';
import Logout from './components/logout/Logout';
import Logo from './components/logo/Logo';
import Menu from './components/menu/Menu';
import Login from './views/login/Login';
import Users from './views/user/Users';
import Jobs from './views/jobs/Jobs';
import {Route} from 'react-router';
import { Route } from 'react-router';
import './App.less';
import TrackingModal from './components/tracking/TrackingModal.jsx';
import {Banner} from '@douyinfe/semi-ui';
import { Banner } from '@douyinfe/semi-ui';
export default function FredyApp() {
const dispatch = useDispatch();
const [loading, setLoading] = React.useState(true);
const currentUser = useSelector((state) => state.user.currentUser);
const settings = useSelector((state) => state.generalSettings.settings);
const dispatch = useDispatch();
const [loading, setLoading] = React.useState(true);
const currentUser = useSelector((state) => state.user.currentUser);
const settings = useSelector((state) => state.generalSettings.settings);
useEffect(() => {
async function init() {
await dispatch.user.getCurrentUser();
if (!needsLogin()) {
await dispatch.provider.getProvider();
await dispatch.jobs.getJobs();
await dispatch.jobs.getProcessingTimes();
await dispatch.notificationAdapter.getAdapter();
await dispatch.generalSettings.getGeneralSettings();
}
setLoading(false);
}
useEffect(() => {
async function init() {
await dispatch.user.getCurrentUser();
if (!needsLogin()) {
await dispatch.provider.getProvider();
await dispatch.jobs.getJobs();
await dispatch.jobs.getProcessingTimes();
await dispatch.notificationAdapter.getAdapter();
await dispatch.generalSettings.getGeneralSettings();
}
setLoading(false);
}
init();
}, [currentUser?.userId]);
init();
}, [currentUser?.userId]);
const needsLogin = () => {
return currentUser == null || Object.keys(currentUser).length === 0;
};
const needsLogin = () => {
return currentUser == null || Object.keys(currentUser).length === 0;
};
const isAdmin = () => currentUser != null && currentUser.isAdmin;
const isAdmin = () => currentUser != null && currentUser.isAdmin;
const login = () => (
const login = () => (
<Switch>
<Route name="Login" path={'/login'} component={Login} />
<Redirect from="*" to={'/login'} />
</Switch>
);
return loading ? null : needsLogin() ? (
login()
) : (
<div className="app">
<div className="app__container">
<Logout />
<Logo width={190} white />
<Menu isAdmin={isAdmin()} />
{settings.demoMode && (
<>
<Banner
fullMode={true}
type="info"
bordered
closeIcon={null}
description="You're currently viewing the demo version of Fredy. Jobs won't scrape websites, and any changes you make will be reverted at midnight."
/>
<br />
</>
)}
{settings.analyticsEnabled === null && !settings.demoMode && <TrackingModal />}
<Switch>
<Route name="Login" path={'/login'} component={Login}/>
<Redirect from="*" to={'/login'}/>
<Route name="Insufficient Permission" path={'/403'} component={InsufficientPermission} />
<Route name="Create new Job" path={'/jobs/new'} component={JobMutation} />
<Route name="Edit a Job" path={'/jobs/edit/:jobId'} component={JobMutation} />
<Route name="Insights into a Job" path={'/jobs/insights/:jobId'} component={JobInsight} />
<Route name="Job overview" path={'/jobs'} component={Jobs} />
<PermissionAwareRoute
name="Create new User"
path="/users/new"
component={<UserMutator />}
currentUser={currentUser}
/>
<PermissionAwareRoute
name="Edit a user"
path="/users/edit/:userId"
component={<UserMutator />}
currentUser={currentUser}
/>
<PermissionAwareRoute name="Users" path="/users" component={<Users />} currentUser={currentUser} />
<PermissionAwareRoute
name="General Settings"
path="/generalSettings"
component={<GeneralSettings />}
currentUser={currentUser}
/>
<Redirect from="/" to={'/jobs'} />
</Switch>
);
return loading ? null : needsLogin() ? (
login()
) : (
<div className="app">
<div className="app__container">
<Logout/>
<Logo width={190} white/>
<Menu isAdmin={isAdmin()}/>
{settings.demoMode && (
<>
<Banner fullMode={true}
type="info"
bordered
closeIcon={null}
description="You're currently viewing the demo version of Fredy. Jobs won't scrape websites, and any changes you make will be reverted at midnight."
/>
<br/>
</>)}
{(settings.analyticsEnabled === null && !settings.demoMode) && <TrackingModal/>}
<Switch>
<Route name="Insufficient Permission" path={'/403'} component={InsufficientPermission}/>
<Route name="Create new Job" path={'/jobs/new'} component={JobMutation}/>
<Route name="Edit a Job" path={'/jobs/edit/:jobId'} component={JobMutation}/>
<Route name="Insights into a Job" path={'/jobs/insights/:jobId'} component={JobInsight}/>
<Route name="Job overview" path={'/jobs'} component={Jobs}/>
<PermissionAwareRoute
name="Create new User"
path="/users/new"
component={<UserMutator/>}
currentUser={currentUser}
/>
<PermissionAwareRoute
name="Edit a user"
path="/users/edit/:userId"
component={<UserMutator/>}
currentUser={currentUser}
/>
<PermissionAwareRoute name="Users" path="/users" component={<Users/>} currentUser={currentUser}/>
<PermissionAwareRoute
name="General Settings"
path="/generalSettings"
component={<GeneralSettings/>}
currentUser={currentUser}
/>
<Redirect from="/" to={'/jobs'}/>
</Switch>
</div>
</div>
);
</div>
</div>
);
}
FredyApp.displayName = 'FredyApp';

View File

@@ -40,4 +40,8 @@ a:active {
color: #54a9ff;
background-color: transparent;
text-decoration: underline;
}
}
.semi-icon:not(.semi-tabs-bar .semi-tabs-tab .semi-icon) {
vertical-align: middle;
}

View File

@@ -23,5 +23,5 @@ root.render(
<App />
</LocaleProvider>
</HashRouter>
</Provider>
</Provider>,
);

View File

@@ -30,7 +30,7 @@ export default function ProviderTable({ providerData = [], onRemove } = {}) {
render: (_, record) => {
return (
<div style={{ float: 'right' }}>
<Button type="danger" icon={<IconDelete />} onClick={() => onRemove(record.id)} />
<Button type="danger" icon={<IconDelete />} onClick={() => onRemove(record.url)} />
</div>
);
},

View File

@@ -1,48 +1,56 @@
import React from 'react';
import {Modal} from '@douyinfe/semi-ui';
import { Modal } from '@douyinfe/semi-ui';
import Logo from '../logo/Logo.jsx';
import {xhrPost} from '../../services/xhr.js';
import { xhrPost } from '../../services/xhr.js';
import './TrackingModal.less';
import inDevelopment from '../../services/developmentMode.js';
const saveResponse = async (analyticsEnabled) => {
await xhrPost('/api/admin/generalSettings', {
analyticsEnabled
});
await xhrPost('/api/admin/generalSettings', {
analyticsEnabled,
});
};
export default function TrackingModal() {
if (inDevelopment()) {
return null;
}
return <Modal
visible={true}
onOk={async () => {
await saveResponse(true);
location.reload();
}}
onCancel={async () => {
await saveResponse(false);
location.reload();
}}
maskClosable={false}
closable={false}
okText="Yes! I want to help"
cancelText="No, thanks"
return (
<Modal
visible={true}
onOk={async () => {
await saveResponse(true);
location.reload();
}}
onCancel={async () => {
await saveResponse(false);
location.reload();
}}
maskClosable={false}
closable={false}
okText="Yes! I want to help"
cancelText="No, thanks"
>
<Logo white/>
<div className="trackingModal__description">
<p>Hey 👋</p>
<p>Fed up with popups? Yeah, me too. But this ones important, and I promise it will only appear once ;)</p>
<p>Fredy is completely free (and will always remain free). If youd like, you can support me by donating
through my GitHub, but theres absolutely no obligation to do so.</p>
<p>However, it would be a huge
help if youd allow me to collect some analytical data. Wait, before you click "no", let me explain. If
you
agree, Fredy will send a ping to my Mixpanel project each time it runs.</p>
<p>The data includes: names of
active adapters/providers, OS, architecture, Node version, and language. The information is entirely
anonymous and helps me understand which adapters/providers are most frequently used.</p>
<p>Thanks🤘</p>
</div>
</Modal>;
}
<Logo white />
<div className="trackingModal__description">
<p>Hey 👋</p>
<p>Fed up with popups? Yeah, me too. But this ones important, and I promise it will only appear once ;)</p>
<p>
Fredy is completely free (and will always remain free). If youd like, you can support me by donating through
my GitHub, but theres absolutely no obligation to do so.
</p>
<p>
However, it would be a huge help if youd allow me to collect some analytical data. Wait, before you click
"no", let me explain. If you agree, Fredy will send a ping to my Mixpanel project each time it runs.
</p>
<p>
The data includes: names of active adapters/providers, OS, architecture, Node version, and language. The
information is entirely anonymous and helps me understand which adapters/providers are most frequently used.
</p>
<p>Thanks🤘</p>
</div>
</Modal>
);
}

View File

@@ -0,0 +1,4 @@
export default function isDevelopmentMode(){
const inDevMode= import.meta.env.MODE;
return inDevMode != null && inDevMode === 'development';
}

View File

@@ -1,261 +1,251 @@
import React from 'react';
import {useDispatch, useSelector} from 'react-redux';
import { useDispatch, useSelector } from 'react-redux';
import {Divider, TimePicker, Button, Checkbox} from '@douyinfe/semi-ui';
import {InputNumber} from '@douyinfe/semi-ui';
import { Divider, TimePicker, Button, Checkbox } from '@douyinfe/semi-ui';
import { InputNumber } from '@douyinfe/semi-ui';
import Headline from '../../components/headline/Headline';
import {xhrPost} from '../../services/xhr';
import {SegmentPart} from '../../components/segment/SegmentPart';
import {Banner, Toast} from '@douyinfe/semi-ui';
import {IconSave, IconCalendar, IconRefresh, IconSignal, IconLineChartStroked, IconSearch} from '@douyinfe/semi-icons';
import { xhrPost } from '../../services/xhr';
import { SegmentPart } from '../../components/segment/SegmentPart';
import { Banner, Toast } from '@douyinfe/semi-ui';
import {
IconSave,
IconCalendar,
IconRefresh,
IconSignal,
IconLineChartStroked,
IconSearch,
} from '@douyinfe/semi-icons';
import './GeneralSettings.less';
function formatFromTimestamp(ts) {
const date = new Date(ts);
return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
const date = new Date(ts);
return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
}
function formatFromTBackend(time) {
if (time == null || time.length === 0) {
return null;
}
const date = new Date();
const split = time.split(':');
date.setHours(split[0]);
date.setMinutes(split[1]);
return date.getTime();
if (time == null || time.length === 0) {
return null;
}
const date = new Date();
const split = time.split(':');
date.setHours(split[0]);
date.setMinutes(split[1]);
return date.getTime();
}
const GeneralSettings = function GeneralSettings() {
const dispatch = useDispatch();
const [loading, setLoading] = React.useState(true);
const dispatch = useDispatch();
const [loading, setLoading] = React.useState(true);
const settings = useSelector((state) => state.generalSettings.settings);
const settings = useSelector((state) => state.generalSettings.settings);
const [interval, setInterval] = React.useState('');
const [port, setPort] = React.useState('');
const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
const [workingHourTo, setWorkingHourTo] = React.useState(null);
const [demoMode, setDemoMode] = React.useState(null);
const [analyticsEnabled, setAnalyticsEnabled] = React.useState(null);
const [interval, setInterval] = React.useState('');
const [port, setPort] = React.useState('');
const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
const [workingHourTo, setWorkingHourTo] = React.useState(null);
const [demoMode, setDemoMode] = React.useState(null);
const [analyticsEnabled, setAnalyticsEnabled] = React.useState(null);
React.useEffect(() => {
async function init() {
await dispatch.generalSettings.getGeneralSettings();
setLoading(false);
}
React.useEffect(() => {
async function init() {
await dispatch.generalSettings.getGeneralSettings();
setLoading(false);
}
init();
}, []);
init();
}, []);
React.useEffect(() => {
async function init() {
setInterval(settings?.interval);
setPort(settings?.port);
setWorkingHourFrom(settings?.workingHours?.from);
setWorkingHourTo(settings?.workingHours?.to);
setAnalyticsEnabled(settings?.analyticsEnabled || false);
setDemoMode(settings?.demoMode || false);
}
React.useEffect(() => {
async function init() {
setInterval(settings?.interval);
setPort(settings?.port);
setWorkingHourFrom(settings?.workingHours?.from);
setWorkingHourTo(settings?.workingHours?.to);
setAnalyticsEnabled(settings?.analyticsEnabled || false);
setDemoMode(settings?.demoMode || false);
}
init();
}, [settings]);
init();
}, [settings]);
const nullOrEmpty = (val) => val == null || val.length === 0;
const nullOrEmpty = (val) => val == null || val.length === 0;
const throwMessage = (message, type) => {
if (type === 'error') {
Toast.error(message);
} else {
Toast.success(message);
}
};
const throwMessage = (message, type) => {
if (type === 'error') {
Toast.error(message);
} else {
Toast.success(message);
}
};
const onStore = async () => {
if (nullOrEmpty(interval)) {
throwMessage('Interval may not be empty.', 'error');
return;
}
if (nullOrEmpty(port)) {
throwMessage('Port may not be empty.', 'error');
return;
}
if (
(!nullOrEmpty(workingHourFrom) && nullOrEmpty(workingHourTo)) ||
(nullOrEmpty(workingHourFrom) && !nullOrEmpty(workingHourTo))
) {
throwMessage('Working hours to and from must be set if either to or from has been set before.', 'error');
return;
}
try {
await xhrPost('/api/admin/generalSettings', {
interval,
port,
workingHours: {
from: workingHourFrom,
to: workingHourTo,
},
demoMode,
analyticsEnabled
});
} catch (exception) {
console.error(exception);
if(exception?.json?.message != null){
throwMessage(exception.json.message, 'error');
}else {
throwMessage('Error while trying to store settings.', 'error');
}
return;
}
throwMessage('Settings stored successfully. We will reload your browser in 3 seconds.', 'success');
setTimeout(()=>{
location.reload();
}, 3000);
};
const onStore = async () => {
if (nullOrEmpty(interval)) {
throwMessage('Interval may not be empty.', 'error');
return;
}
if (nullOrEmpty(port)) {
throwMessage('Port may not be empty.', 'error');
return;
}
if (
(!nullOrEmpty(workingHourFrom) && nullOrEmpty(workingHourTo)) ||
(nullOrEmpty(workingHourFrom) && !nullOrEmpty(workingHourTo))
) {
throwMessage('Working hours to and from must be set if either to or from has been set before.', 'error');
return;
}
try {
await xhrPost('/api/admin/generalSettings', {
interval,
port,
workingHours: {
from: workingHourFrom,
to: workingHourTo,
},
demoMode,
analyticsEnabled,
});
} catch (exception) {
console.error(exception);
if (exception?.json?.message != null) {
throwMessage(exception.json.message, 'error');
} else {
throwMessage('Error while trying to store settings.', 'error');
}
return;
}
throwMessage('Settings stored successfully. We will reload your browser in 3 seconds.', 'success');
setTimeout(() => {
location.reload();
}, 3000);
};
return (
<div>
{!loading && (
<React.Fragment>
<Headline text="General Settings"/>
<div>
<SegmentPart
name="Interval"
helpText="Interval in minutes for running queries against the configured services."
Icon={IconRefresh}
>
<InputNumber
min={0}
max={1440}
placeholder="Interval in minutes"
value={interval}
formatter={(value) => `${value}`.replace(/\D/g, '')}
onChange={(value) => setInterval(value)}
suffix={'minutes'}
/>
</SegmentPart>
<Divider margin="1rem"/>
<SegmentPart name="Port" helpText="Port on which Fredy is running." Icon={IconSignal}>
<InputNumber
min={0}
max={99999}
placeholder="Port"
value={port}
formatter={(value) => `${value}`.replace(/\D/g, '')}
onChange={(value) => setPort(value)}
/>
</SegmentPart>
<Divider margin="1rem"/>
<SegmentPart
name="Working hours"
helpText="During this hours, Fredy will search for new apartments. If nothing is configured, Fredy will search around the clock."
Icon={IconCalendar}
>
<div className="generalSettings__timePickerContainer">
<TimePicker
format={'HH:mm'}
insetLabel="From"
value={formatFromTBackend(workingHourFrom)}
placeholder=""
onChange={(val) => {
setWorkingHourFrom(val == null ? null : formatFromTimestamp(val));
}}
/>
<TimePicker
format={'HH:mm'}
insetLabel="Until"
value={formatFromTBackend(workingHourTo)}
placeholder=""
onChange={(val) => {
setWorkingHourTo(val == null ? null : formatFromTimestamp(val));
}}
/>
</div>
</SegmentPart>
<Divider margin="1rem"/>
return (
<div>
{!loading && (
<React.Fragment>
<Headline text="General Settings" />
<div>
<SegmentPart
name="Interval"
helpText="Interval in minutes for running queries against the configured services."
Icon={IconRefresh}
>
<InputNumber
min={0}
max={1440}
placeholder="Interval in minutes"
value={interval}
formatter={(value) => `${value}`.replace(/\D/g, '')}
onChange={(value) => setInterval(value)}
suffix={'minutes'}
/>
</SegmentPart>
<Divider margin="1rem" />
<SegmentPart name="Port" helpText="Port on which Fredy is running." Icon={IconSignal}>
<InputNumber
min={0}
max={99999}
placeholder="Port"
value={port}
formatter={(value) => `${value}`.replace(/\D/g, '')}
onChange={(value) => setPort(value)}
/>
</SegmentPart>
<Divider margin="1rem" />
<SegmentPart
name="Working hours"
helpText="During this hours, Fredy will search for new apartments. If nothing is configured, Fredy will search around the clock."
Icon={IconCalendar}
>
<div className="generalSettings__timePickerContainer">
<TimePicker
format={'HH:mm'}
insetLabel="From"
value={formatFromTBackend(workingHourFrom)}
placeholder=""
onChange={(val) => {
setWorkingHourFrom(val == null ? null : formatFromTimestamp(val));
}}
/>
<TimePicker
format={'HH:mm'}
insetLabel="Until"
value={formatFromTBackend(workingHourTo)}
placeholder=""
onChange={(val) => {
setWorkingHourTo(val == null ? null : formatFromTimestamp(val));
}}
/>
</div>
</SegmentPart>
<Divider margin="1rem" />
<SegmentPart
name="Analytics"
helpText="Insights into the usage of Fredy."
Icon={IconLineChartStroked}
>
<Banner
fullMode={false}
type="info"
closeIcon={null}
title={
<div style={{fontWeight: 600, fontSize: '14px', lineHeight: '20px'}}>
Explanation
</div>
}
style={{marginBottom: '1rem'}}
description={
<div>
Analytics are disabled by default. If you choose to enable them, we will begin tracking the following:<br/>
<ul>
<li>Name of active provider (e.g. Immoscout)</li>
<li>Name of active adapter (e.g. Console)</li>
<li>language</li>
<li>os</li>
<li>node version</li>
<li>arch</li>
</ul>
The data is sent anonymously and helps me understand which providers or adapters are being used the most. In the end it helps me to improve fredy.
</div>
}
/>
<SegmentPart name="Analytics" helpText="Insights into the usage of Fredy." Icon={IconLineChartStroked}>
<Banner
fullMode={false}
type="info"
closeIcon={null}
title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Explanation</div>}
style={{ marginBottom: '1rem' }}
description={
<div>
Analytics are disabled by default. If you choose to enable them, we will begin tracking the
following:
<br />
<ul>
<li>Name of active provider (e.g. Immoscout)</li>
<li>Name of active adapter (e.g. Console)</li>
<li>language</li>
<li>os</li>
<li>node version</li>
<li>arch</li>
</ul>
The data is sent anonymously and helps me understand which providers or adapters are being used the
most. In the end it helps me to improve fredy.
</div>
}
/>
<Checkbox
checked={analyticsEnabled}
onChange={(e) => setAnalyticsEnabled(e.target.checked)}
> Enabled
</Checkbox>
<Checkbox checked={analyticsEnabled} onChange={(e) => setAnalyticsEnabled(e.target.checked)}>
{' '}
Enabled
</Checkbox>
</SegmentPart>
</SegmentPart>
<Divider margin="1rem" />
<Divider margin="1rem"/>
<SegmentPart name="Demo Mode" helpText="If enabled, Fredy runs in demo mode." Icon={IconSearch}>
<Banner
fullMode={false}
type="info"
closeIcon={null}
title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Explanation</div>}
style={{ marginBottom: '1rem' }}
description={
<div>
In demo mode, Fredy will not (really) search for any real estates. Fredy is in a lockdown mode. Also
all database files will be set back to the default values at midnight.
</div>
}
/>
<SegmentPart
name="Demo Mode"
helpText="If enabled, Fredy runs in demo mode."
Icon={IconSearch}
>
<Banner
fullMode={false}
type="info"
closeIcon={null}
title={
<div style={{fontWeight: 600, fontSize: '14px', lineHeight: '20px'}}>
Explanation
</div>
}
style={{marginBottom: '1rem'}}
description={
<div>
In demo mode, Fredy will not (really) search for any real estates. Fredy is in a lockdown mode. Also
all database files will be set back to the default values at midnight.
</div>
}
/>
<Checkbox checked={demoMode} onChange={(e) => setDemoMode(e.target.checked)}>
{' '}
Enabled
</Checkbox>
</SegmentPart>
<Checkbox
checked={demoMode}
onChange={(e) => setDemoMode(e.target.checked)}
> Enabled
</Checkbox>
</SegmentPart>
<Divider margin="1rem"/>
<Button type="primary" theme="solid" onClick={onStore} icon={<IconSave/>}>
Save
</Button>
</div>
</React.Fragment>
)}
</div>
);
<Divider margin="1rem" />
<Button type="primary" theme="solid" onClick={onStore} icon={<IconSave />}>
Save
</Button>
</div>
</React.Fragment>
)}
</div>
);
};
export default GeneralSettings;

View File

@@ -1,32 +1,32 @@
import React from 'react';
import {format} from '../../services/time/timeService';
import {Banner, Descriptions} from '@douyinfe/semi-ui';
import { format } from '../../services/time/timeService';
import { Descriptions } from '@douyinfe/semi-ui';
export default function ProcessingTimes({processingTimes = {}}) {
if (Object.keys(processingTimes).length === 0) {
return null;
}
return (
<>
<Descriptions
row
size="small"
style={{
backgroundColor: '#35363c',
borderRadius: '4px',
padding: '10px',
}}
>
<Descriptions.Item itemKey="Processing Interval">{processingTimes.interval} min</Descriptions.Item>
{processingTimes.lastRun && (
<>
<Descriptions.Item itemKey="Last run">{format(processingTimes.lastRun)}</Descriptions.Item>
<Descriptions.Item itemKey="Next run">
{format(processingTimes.lastRun + processingTimes.interval * 60000)}
</Descriptions.Item>
</>
)}
</Descriptions>
</>
);
export default function ProcessingTimes({ processingTimes = {} }) {
if (Object.keys(processingTimes).length === 0) {
return null;
}
return (
<>
<Descriptions
row
size="small"
style={{
backgroundColor: '#35363c',
borderRadius: '4px',
padding: '10px',
}}
>
<Descriptions.Item itemKey="Processing Interval">{processingTimes.interval} min</Descriptions.Item>
{processingTimes.lastRun && (
<>
<Descriptions.Item itemKey="Last run">{format(processingTimes.lastRun)}</Descriptions.Item>
<Descriptions.Item itemKey="Next run">
{format(processingTimes.lastRun + processingTimes.interval * 60000)}
</Descriptions.Item>
</>
)}
</Descriptions>
</>
);
}

View File

@@ -94,7 +94,7 @@ export default function JobMutator() {
<form>
<SegmentPart name="Name">
<Input
autofocus
autoFocus
type="text"
maxLength={40}
placeholder="Name"
@@ -124,8 +124,8 @@ export default function JobMutator() {
<ProviderTable
providerData={providerData}
onRemove={(providerId) => {
setProviderData(providerData.filter((provider) => provider.id !== providerId));
onRemove={(providerUrl) => {
setProviderData(providerData.filter((provider) => provider.url !== providerUrl));
}}
/>
</SegmentPart>

View File

@@ -27,7 +27,7 @@ const validate = (selectedAdapter) => {
}
if (uiElement.type === 'number') {
const numberValue = parseFloat(uiElement.value);
if(isNaN(numberValue) || numberValue < 0) {
if (isNaN(numberValue) || numberValue < 0) {
results.push('A number field cannot contain anything else and must be > 0.');
continue;
}
@@ -83,7 +83,7 @@ export default function NotificationAdapterMutator({
id: selectedAdapter.id,
name: selectedAdapter.name,
fields: selectedAdapter.fields || {},
})
}),
);
setSelectedAdapter(null);
@@ -114,7 +114,7 @@ export default function NotificationAdapterMutator({
setSuccessMessage('It seems like it worked! Please check your service.');
})
.catch((error) =>
setValidationMessage(`This did not work :-( I've received the following error: ${error.json.message}`)
setValidationMessage(`This did not work :-( I've received the following error: ${error.json.message}`),
);
};
@@ -229,7 +229,7 @@ export default function NotificationAdapterMutator({
.filter((option) =>
editNotificationAdapter != null
? true
: selected.find((selectedOption) => selectedOption.id === option.key) == null
: selected.find((selectedOption) => selectedOption.id === option.key) == null,
)
.sort(sortAdapter)}
onChange={(value) => {

View File

@@ -45,7 +45,7 @@ export default function ProviderMutator({ onVisibilityChanged, visible = false,
url: providerUrl,
id: selectedProvider.id,
name: selectedProvider.name,
})
}),
);
setProviderUrl(null);
setSelectedProvider(null);
@@ -101,7 +101,7 @@ export default function ProviderMutator({ onVisibilityChanged, visible = false,
description={
<div>
<p>
Currently, our Immoscout implementation does not drawing shapes on a map. Use a radius instead.
Currently, our Immoscout implementation does not support drawing shapes on a map. Use a radius instead.
</p>
</div>
}

View File

@@ -1,102 +1,105 @@
import React, {useEffect} from 'react';
import React, { useEffect } from 'react';
import cityBackground from '../../assets/city_background.jpg';
import Logo from '../../components/logo/Logo';
import {xhrPost} from '../../services/xhr';
import {useHistory} from 'react-router';
import {useDispatch, useSelector} from 'react-redux';
import {Input, Button, Banner} from '@douyinfe/semi-ui';
import { xhrPost } from '../../services/xhr';
import { useHistory } from 'react-router';
import { useDispatch, useSelector } from 'react-redux';
import { Input, Button, Banner } from '@douyinfe/semi-ui';
import './login.less';
import {IconUser, IconLock} from '@douyinfe/semi-icons';
import { IconUser, IconLock } from '@douyinfe/semi-icons';
export default function Login() {
const dispatch = useDispatch();
const [username, setUserName] = React.useState('');
const [password, setPassword] = React.useState('');
const [error, setError] = React.useState(null);
const demoMode = useSelector((state) => state.demoMode.demoMode || false);
const history = useHistory();
const dispatch = useDispatch();
const [username, setUserName] = React.useState('');
const [password, setPassword] = React.useState('');
const [error, setError] = React.useState(null);
const demoMode = useSelector((state) => state.demoMode.demoMode || false);
const history = useHistory();
useEffect(() => {
async function init() {
await dispatch.demoMode.getDemoMode();
}
useEffect(() => {
async function init() {
await dispatch.demoMode.getDemoMode();
}
init();
}, []);
init();
}, []);
const tryLogin = async () => {
if (username.length === 0 || password.length === 0) {
setError('Username and password are mandatory.');
return;
}
try {
await xhrPost('/api/login', {
username,
password,
});
setError(null);
} catch (Exception) {
setError('Login not successful...');
return;
}
await dispatch.user.getCurrentUser();
history.push('/jobs');
};
const tryLogin = async () => {
if (username.length === 0 || password.length === 0) {
setError('Username and password are mandatory.');
return;
}
try {
await xhrPost('/api/login', {
username,
password,
});
setError(null);
} catch (Exception) {
setError('Login not successful...');
return;
}
await dispatch.user.getCurrentUser();
history.push('/jobs');
};
return (
<div className="login">
<div className="login__bgImage" style={{background: `url("${cityBackground}")`}}/>
<Logo/>
<form>
<div className="login__loginWrapper">
{error && <Banner type="danger" closeIcon={null} description={error}/>}
<Input
size="large"
prefix={<IconUser/>}
placeholder="Username"
value={username}
showClear
style={{marginTop: error ? '1rem' : '4rem'}}
autoFocus
onChange={(value) => setUserName(value)}
onKeyPress={async (e) => {
if (e.key === 'Enter') {
await tryLogin();
}
}}
/>
return (
<div className="login">
<div className="login__bgImage" style={{ background: `url("${cityBackground}")` }} />
<Logo />
<form>
<div className="login__loginWrapper">
{error && <Banner type="danger" closeIcon={null} description={error} />}
<Input
size="large"
prefix={<IconUser />}
placeholder="Username"
value={username}
showClear
style={{ marginTop: error ? '1rem' : '4rem' }}
autoFocus
onChange={(value) => setUserName(value)}
onKeyPress={async (e) => {
if (e.key === 'Enter') {
await tryLogin();
}
}}
/>
<Input
size="large"
mode="password"
prefix={<IconLock/>}
value={password}
placeholder="Password"
style={{marginTop: '2rem'}}
onChange={(value) => setPassword(value)}
onKeyPress={async (e) => {
if (e.key === 'Enter') {
await tryLogin();
}
}}
/>
<Input
size="large"
mode="password"
prefix={<IconLock />}
value={password}
placeholder="Password"
style={{ marginTop: '2rem' }}
onChange={(value) => setPassword(value)}
onKeyPress={async (e) => {
if (e.key === 'Enter') {
await tryLogin();
}
}}
/>
<Button type="primary" onClick={tryLogin} theme="solid" style={{marginTop: '3rem'}}>
Login
</Button>
<br/>
{demoMode && <Banner fullMode={true}
type="info"
bordered
closeIcon={null}
description="This is the demo version of Fredy. Use 'demo' as both the username and password to log in."
/>}
</div>
</form>
<Button type="primary" onClick={tryLogin} theme="solid" style={{ marginTop: '3rem' }}>
Login
</Button>
<br />
{demoMode && (
<Banner
fullMode={true}
type="info"
bordered
closeIcon={null}
description="This is the demo version of Fredy. Use 'demo' as both the username and password to log in."
/>
)}
</div>
);
</form>
</div>
);
}
Login.displayName = 'Login';

View File

@@ -6,6 +6,7 @@ export default defineConfig({
build: {
chunkSizeWarningLimit: 9999999,
outDir: './ui/public',
emptyOutDir: true,
},
plugins: [react()],
server: {

1407
yarn.lock

File diff suppressed because it is too large Load Diff