mirror of
https://github.com/orangecoding/fredy.git
synced 2026-06-16 12:31:07 +00:00
Compare commits
25 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
206f768b41 | ||
|
|
2302f69ff3 | ||
|
|
9bb33e723a | ||
|
|
cca1463a68 | ||
|
|
314b1818d7 | ||
|
|
25cc7fb650 | ||
|
|
78df4b21a6 | ||
|
|
d89b078237 | ||
|
|
395199a4a2 | ||
|
|
c2680fe49f | ||
|
|
2b862b2d98 | ||
|
|
9065448b6b | ||
|
|
b9f49cb5b2 | ||
|
|
53121742c2 | ||
|
|
1a3eae0390 | ||
|
|
a42905d63f | ||
|
|
9917491728 | ||
|
|
f032e6a724 | ||
|
|
111c154ae3 | ||
|
|
2194ffe0f4 | ||
|
|
cfa25fc0e0 | ||
|
|
d50dd61f3e | ||
|
|
31e7f77bde | ||
|
|
a418d64f1a | ||
|
|
d099872950 |
@@ -2,5 +2,6 @@ node_modules/
|
||||
npm-debug.log
|
||||
test/
|
||||
db/
|
||||
conf/
|
||||
.git/
|
||||
.github/
|
||||
|
||||
26
.github/workflows/check_source.yml
vendored
Normal file
26
.github/workflows/check_source.yml
vendored
Normal file
@@ -0,0 +1,26 @@
|
||||
name: Check the source code
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
pull_request:
|
||||
branches: [master]
|
||||
jobs:
|
||||
check_source_code:
|
||||
name: Check the source code
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 20
|
||||
cache: 'yarn'
|
||||
|
||||
- name: Install dependencies
|
||||
run: yarn install
|
||||
|
||||
- name: Check formatting
|
||||
run: yarn format:check
|
||||
|
||||
- name: Lint
|
||||
run: yarn lint
|
||||
2
.github/workflows/stales.yml
vendored
2
.github/workflows/stales.yml
vendored
@@ -1,4 +1,4 @@
|
||||
name: "Close stale issues and PRs"
|
||||
name: Close stale issues and PRs
|
||||
|
||||
on:
|
||||
schedule:
|
||||
|
||||
3
.github/workflows/test.yml
vendored
3
.github/workflows/test.yml
vendored
@@ -13,8 +13,7 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 20
|
||||
cache: 'yarn'
|
||||
|
||||
4
.prettierrc
Normal file
4
.prettierrc
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"singleQuote": true,
|
||||
"printWidth": 120
|
||||
}
|
||||
@@ -106,14 +106,14 @@ exports.config = {
|
||||
```
|
||||
|
||||
#### Running Tests
|
||||
If you've written a new provider you are an awesome person. If you now write tests for it, you are even more awesome. And who doesn't want to be more awesome right?
|
||||
If you've written a new provider you are an awesome person. If you now write tests for it, you are even more awesome. And who doesn't want to be more awesome, right?
|
||||
|
||||
#### Codestyle
|
||||
I'm using Eslint to maintain quote style and quality. Do not skip it...
|
||||
I'm using ESLint to maintain quote style and quality. Do not skip it...
|
||||
|
||||
##### To do before merging:
|
||||
##### To-do before merging:
|
||||
|
||||
- executed tests? (`pnpm test`)
|
||||
- sure the changes are useful for everybody? Or is it maybe a custom modification just for your case?
|
||||
- Have you executed the tests? (`yarn test`)
|
||||
- Are you sure the changes are useful for everybody? Or is it maybe a custom modification just for your case?
|
||||
|
||||
_Thanks!_ :heart:
|
||||
|
||||
@@ -11,16 +11,16 @@ ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
|
||||
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
|
||||
|
||||
# Copy lockfiles first to leverage cache for dependencies
|
||||
COPY package.json yarn.lock ./
|
||||
COPY package.json yarn.lock .
|
||||
|
||||
# Set Yarn timeout, install dependencies and PM2 globally
|
||||
RUN yarn config set network-timeout 600000 \
|
||||
&& yarn install --frozen-lockfile \
|
||||
&& yarn --frozen-lockfile \
|
||||
&& yarn global add pm2
|
||||
|
||||
# Copy application source and build production assets
|
||||
COPY . ./
|
||||
RUN yarn run prod
|
||||
COPY . .
|
||||
RUN yarn build:frontend
|
||||
|
||||
# Prepare runtime directories and symlinks for data and config
|
||||
RUN mkdir -p /db /conf \
|
||||
|
||||
19
README.md
19
README.md
@@ -1,6 +1,6 @@
|
||||
<img src="https://github.com/orangecoding/fredy/blob/master/doc/logo.png" width="400">
|
||||
|
||||
 [](https://github.com/orangecoding/fredy/actions/workflows/docker.yml)
|
||||
 [](https://github.com/orangecoding/fredy/actions/workflows/docker.yml) 
|
||||
|
||||
Searching an apartment in Germany can be a frustrating task. Not any longer though, as _Fredy_ will take over and will only notify you once new listings have been found that match your requirements.
|
||||
|
||||
@@ -8,8 +8,6 @@ _Fredy_ scrapes multiple services (Immonet, Immowelt etc.) and send new listings
|
||||
|
||||
If _Fredy_ finds matching results, it will send them to you via Slack, Email, Telegram etc. (More adapters can be configured.) As _Fredy_ stores the listings it has found, new results will not be sent to you twice (and as a side-effect, _Fredy_ can show some statistics). Furthermore, _Fredy_ checks duplicates per scraping so that the same listings are not being sent twice or more when posted on various platforms (which happens more often than one might think).
|
||||
|
||||
<a href="https://www.producthunt.com/posts/fredy-find-real-estates-damn-easy?embed=true&utm_source=badge-featured&utm_medium=badge&utm_source=badge-fredy-find-real-estates-damn-easy" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=965690&theme=light&t=1747292331626" alt="Fredy - Find Real Estates Damn EasY  - Your personal real estate search bot | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
# Sponsorship [](https://github.com/sponsors/orangecoding)
|
||||
If you like my work, consider becoming a sponsor. I'm not expecting anybody to pay for _Fredy_ or any other Open Source Project I'm maintaining, however keep in mind, I'm doing all of this in my spare time :) Thanks.
|
||||
|
||||
@@ -25,9 +23,9 @@ If you want to try out _Fredy_, you can access the demo version [here](https://f
|
||||
- Make sure to use Node.js 20 or above
|
||||
- Run the following commands:
|
||||
```ssh
|
||||
yarn (or npm install)
|
||||
yarn run prod
|
||||
yarn run start
|
||||
yarn
|
||||
yarn run start:backend
|
||||
yarn run start:frontend
|
||||
```
|
||||
_Fredy_ will start with the default port, set to `9998`. You can access _Fredy_ by opening your browser at `http://localhost:9998`. The default login is `admin`, both for username and password. You should change the password as soon as possible when you plan to run Fredy on a server.
|
||||
|
||||
@@ -48,7 +46,7 @@ A provider contains the URL that points to the search results for the respective
|
||||
**It is important that you order the search results by date, so that _Fredy_ always picks the latest results first!**
|
||||
|
||||
#### Adapter
|
||||
_Fredy_ supports multiple adapters, such as Slack, SendGrid, Telegram etc. A search job can have as many adapters as supported by _Fredy_. Each adapter needs different configuration values, which you have to provide when using them. A adapter dictactes how the frontend renders by telling the frontend what information it needs in order to send listings to the user.
|
||||
_Fredy_ supports multiple adapters, such as Slack, SendGrid, Telegram etc. A search job can have as many adapters as supported by _Fredy_. Each adapter needs different configuration values, which you have to provide when using them. An adapter dictates how the frontend renders by telling the frontend what information it needs in order to send listings to the user.
|
||||
|
||||
#### Jobs
|
||||
A Job wraps adapters and providers. _Fredy_ runs the configured jobs in a specific interval (can be configured in `/conf/config.json`).
|
||||
@@ -63,14 +61,13 @@ As an administrator, you can create, edit and remove users from _Fredy_. Be care
|
||||
# Development
|
||||
|
||||
### Running Fredy in development mode
|
||||
To run _Fredy_ in development mode, you need to run the backend & frontend separately.
|
||||
Start the backend with:
|
||||
```shell
|
||||
yarn run start
|
||||
yarn run start:backend:dev
|
||||
```
|
||||
For the frontend, run:
|
||||
```shell
|
||||
yarn run dev
|
||||
yarn run start:frontend:dev
|
||||
```
|
||||
You should now be able to access _Fredy_ from your browser. Check your Terminal to see what port the frontend is running on.
|
||||
|
||||
@@ -84,7 +81,7 @@ yarn run test
|
||||

|
||||
|
||||
### Immoscout
|
||||
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
|
||||
Immoscout has implemented advanced bot detection. In order to work around this, we are using a reversed engineered version of their mobile api. See [Immoscout Reverse Engineering Documentation](https://github.com/orangecoding/fredy/blob/master/reverse-engineered-immoscout.md)
|
||||
|
||||
# Analytics
|
||||
Fredy is completely free (and will always remain free). However, it would be a huge help if you’d allow me to collect some analytical data.
|
||||
|
||||
64
index.js
64
index.js
@@ -1,14 +1,14 @@
|
||||
import fs from 'fs';
|
||||
import {config} from './lib/utils.js';
|
||||
import { config } from './lib/utils.js';
|
||||
import * as similarityCache from './lib/services/similarity-check/similarityCache.js';
|
||||
import { setLastJobExecution } from './lib/services/storage/listingsStorage.js';
|
||||
import * as jobStorage from './lib/services/storage/jobStorage.js';
|
||||
import FredyRuntime from './lib/FredyRuntime.js';
|
||||
import { duringWorkingHoursOrNotSet } from './lib/utils.js';
|
||||
import './lib/api/api.js';
|
||||
import {track} from './lib/services/tracking/Tracker.js';
|
||||
import {handleDemoUser} from './lib/services/storage/userStorage.js';
|
||||
import {cleanupDemoAtMidnight} from './lib/services/demoCleanup.js';
|
||||
import { track } from './lib/services/tracking/Tracker.js';
|
||||
import { handleDemoUser } from './lib/services/storage/userStorage.js';
|
||||
import { cleanupDemoAtMidnight } from './lib/services/demoCleanup.js';
|
||||
//if db folder does not exist, ensure to create it before loading anything else
|
||||
if (!fs.existsSync('./db')) {
|
||||
fs.mkdirSync('./db');
|
||||
@@ -19,13 +19,13 @@ const provider = fs.readdirSync(path).filter((file) => file.endsWith('.js'));
|
||||
const INTERVAL = config.interval * 60 * 1000;
|
||||
/* eslint-disable no-console */
|
||||
console.log(`Started Fredy successfully. Ui can be accessed via http://localhost:${config.port}`);
|
||||
if(config.demoMode){
|
||||
console.info('Running in demo mode');
|
||||
cleanupDemoAtMidnight();
|
||||
if (config.demoMode) {
|
||||
console.info('Running in demo mode');
|
||||
cleanupDemoAtMidnight();
|
||||
}
|
||||
/* eslint-enable no-console */
|
||||
const fetchedProvider = await Promise.all(
|
||||
provider.filter((provider) => provider.endsWith('.js')).map(async (pro) => import(`${path}/${pro}`))
|
||||
provider.filter((provider) => provider.endsWith('.js')).map(async (pro) => import(`${path}/${pro}`)),
|
||||
);
|
||||
|
||||
handleDemoUser();
|
||||
@@ -33,30 +33,30 @@ handleDemoUser();
|
||||
setInterval(
|
||||
(function exec() {
|
||||
const isDuringWorkingHoursOrNotSet = duringWorkingHoursOrNotSet(config, Date.now());
|
||||
if(!config.demoMode) {
|
||||
if (isDuringWorkingHoursOrNotSet) {
|
||||
track();
|
||||
config.lastRun = Date.now();
|
||||
jobStorage
|
||||
.getJobs()
|
||||
.filter((job) => job.enabled)
|
||||
.forEach((job) => {
|
||||
job.provider
|
||||
.filter((p) => fetchedProvider.find((fp) => fp.metaInformation.id === p.id) != null)
|
||||
.forEach(async (prov) => {
|
||||
const pro = fetchedProvider.find((fp) => fp.metaInformation.id === prov.id);
|
||||
pro.init(prov, job.blacklist);
|
||||
await new FredyRuntime(pro.config, job.notificationAdapter, prov.id, job.id, similarityCache).execute();
|
||||
setLastJobExecution(job.id);
|
||||
});
|
||||
});
|
||||
} else {
|
||||
/* eslint-disable no-console */
|
||||
console.debug('Working hours set. Skipping as outside of working hours.');
|
||||
/* eslint-enable no-console */
|
||||
}
|
||||
}
|
||||
if (!config.demoMode) {
|
||||
if (isDuringWorkingHoursOrNotSet) {
|
||||
track();
|
||||
config.lastRun = Date.now();
|
||||
jobStorage
|
||||
.getJobs()
|
||||
.filter((job) => job.enabled)
|
||||
.forEach((job) => {
|
||||
job.provider
|
||||
.filter((p) => fetchedProvider.find((fp) => fp.metaInformation.id === p.id) != null)
|
||||
.forEach(async (prov) => {
|
||||
const pro = fetchedProvider.find((fp) => fp.metaInformation.id === prov.id);
|
||||
pro.init(prov, job.blacklist);
|
||||
await new FredyRuntime(pro.config, job.notificationAdapter, prov.id, job.id, similarityCache).execute();
|
||||
setLastJobExecution(job.id);
|
||||
});
|
||||
});
|
||||
} else {
|
||||
/* eslint-disable no-console */
|
||||
console.debug('Working hours set. Skipping as outside of working hours.');
|
||||
/* eslint-enable no-console */
|
||||
}
|
||||
}
|
||||
return exec;
|
||||
})(),
|
||||
INTERVAL
|
||||
INTERVAL,
|
||||
);
|
||||
|
||||
@@ -12,7 +12,7 @@ import restana from 'restana';
|
||||
import files from 'serve-static';
|
||||
import path from 'path';
|
||||
import { getDirName } from '../utils.js';
|
||||
import {demoRouter} from './routes/demoRouter.js';
|
||||
import { demoRouter } from './routes/demoRouter.js';
|
||||
const service = restana();
|
||||
const staticService = files(path.join(getDirName(), '../ui/public'));
|
||||
const PORT = config.port || 9998;
|
||||
|
||||
@@ -15,7 +15,7 @@ function doesJobBelongsToUser(job, req) {
|
||||
if (user == null) {
|
||||
return false;
|
||||
}
|
||||
return user.isAdmin || job.userId === job.userId;
|
||||
return user.isAdmin || job.userId === user.id;
|
||||
}
|
||||
jobRouter.get('/', async (req, res) => {
|
||||
const isUserAdmin = isAdmin(req);
|
||||
|
||||
@@ -22,16 +22,18 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey }) =
|
||||
const { token, chatId } = notificationConfig.find((adapter) => adapter.id === config.id).fields;
|
||||
const job = getJob(jobKey);
|
||||
const jobName = job == null ? jobKey : job.name;
|
||||
//we have to split messages into chunk, because otherwise messages are going to become too big and will fail
|
||||
// we have to split messages into chunks, because otherwise messages are going to become too big and will fail
|
||||
const chunks = arrayChunks(newListings, MAX_ENTITIES_PER_CHUNK);
|
||||
const promises = chunks.map((chunk) => {
|
||||
let message = `<i>${jobName}</i> (${serviceName}) found <b>${newListings.length}</b> new listings:\n\n`;
|
||||
message += chunk.map(
|
||||
const messageParagraphs = [];
|
||||
|
||||
messageParagraphs.push(`<i>${jobName}</i> (${serviceName}) found <b>${newListings.length}</b> new listings:`);
|
||||
messageParagraphs.push(...chunk.map(
|
||||
(o) =>
|
||||
`<a href='${o.link}'><b>${shorten(o.title.replace(/\*/g, ''), 45).trim()}</b></a>\n` +
|
||||
[o.address, o.price, o.size].join(' | ') +
|
||||
'\n\n',
|
||||
);
|
||||
[o.address, o.price, o.size].join(' | ')
|
||||
));
|
||||
|
||||
/**
|
||||
* This is to not break the rate limit. It is to only send 1 message per second
|
||||
*/
|
||||
@@ -41,7 +43,7 @@ export const send = ({ serviceName, newListings, notificationConfig, jobKey }) =
|
||||
method: 'post',
|
||||
body: JSON.stringify({
|
||||
chat_id: chatId,
|
||||
text: message,
|
||||
text: messageParagraphs.join('\n\n'),
|
||||
parse_mode: 'HTML',
|
||||
disable_web_page_preview: true,
|
||||
}),
|
||||
|
||||
@@ -6,7 +6,7 @@ const adapter = await Promise.all(
|
||||
fs
|
||||
.readdirSync('./lib/notification/adapter')
|
||||
.filter((file) => file.endsWith('.js'))
|
||||
.map(async (integPath) => await import(`${path}/${integPath}`))
|
||||
.map(async (integPath) => await import(`${path}/${integPath}`)),
|
||||
);
|
||||
|
||||
if (adapter.length === 0) {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import utils, {buildHash} from '../utils.js';
|
||||
import utils, { buildHash } from '../utils.js';
|
||||
let appliedBlackList = [];
|
||||
function shortenLink(link) {
|
||||
return link.substring(0, link.indexOf('?'));
|
||||
|
||||
@@ -36,7 +36,7 @@
|
||||
*/
|
||||
|
||||
import utils, { buildHash } from '../utils.js';
|
||||
import { convertWebToMobile } from '../services/immoscout/immoscout-web-translater.js';
|
||||
import { convertWebToMobile } from '../services/immoscout/immoscout-web-translator.js';
|
||||
let appliedBlackList = [];
|
||||
|
||||
async function getListings(url) {
|
||||
|
||||
@@ -1,50 +1,50 @@
|
||||
import utils, {buildHash} from '../utils.js';
|
||||
import utils, { buildHash } from '../utils.js';
|
||||
|
||||
let appliedBlackList = [];
|
||||
let appliedBlacklistedDistricts = [];
|
||||
|
||||
function normalize(o) {
|
||||
const size = o.size || '--- m²';
|
||||
const id = buildHash(o.id, o.price);
|
||||
const link = `https://www.kleinanzeigen.de${o.link}`;
|
||||
return Object.assign(o, {id, size, link});
|
||||
const size = o.size || '--- m²';
|
||||
const id = buildHash(o.id, o.price);
|
||||
const link = `https://www.kleinanzeigen.de${o.link}`;
|
||||
return Object.assign(o, { id, size, link });
|
||||
}
|
||||
|
||||
function applyBlacklist(o) {
|
||||
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
|
||||
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
|
||||
const isBlacklistedDistrict =
|
||||
appliedBlacklistedDistricts.length === 0 ? false : utils.isOneOf(o.description, appliedBlacklistedDistricts);
|
||||
return o.title != null && !isBlacklistedDistrict && titleNotBlacklisted && descNotBlacklisted;
|
||||
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
|
||||
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
|
||||
const isBlacklistedDistrict =
|
||||
appliedBlacklistedDistricts.length === 0 ? false : utils.isOneOf(o.description, appliedBlacklistedDistricts);
|
||||
return o.title != null && !isBlacklistedDistrict && titleNotBlacklisted && descNotBlacklisted;
|
||||
}
|
||||
|
||||
const config = {
|
||||
url: null,
|
||||
crawlContainer: '#srchrslt-adtable .ad-listitem ',
|
||||
//sort by date is standard oO
|
||||
sortByDateParam: null,
|
||||
waitForSelector: 'body',
|
||||
crawlFields: {
|
||||
id: '.aditem@data-adid | int',
|
||||
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
|
||||
size: '.aditem-main .text-module-end | removeNewline | trim',
|
||||
title: '.aditem-main .text-module-begin a | removeNewline | trim',
|
||||
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
|
||||
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
|
||||
address: '.aditem-main--top--left | trim | removeNewline',
|
||||
},
|
||||
normalize: normalize,
|
||||
filter: applyBlacklist,
|
||||
url: null,
|
||||
crawlContainer: '#srchrslt-adtable .ad-listitem ',
|
||||
//sort by date is standard oO
|
||||
sortByDateParam: null,
|
||||
waitForSelector: 'body',
|
||||
crawlFields: {
|
||||
id: '.aditem@data-adid | int',
|
||||
price: '.aditem-main--middle--price-shipping--price | removeNewline | trim',
|
||||
size: '.aditem-main .text-module-end | removeNewline | trim',
|
||||
title: '.aditem-main .text-module-begin a | removeNewline | trim',
|
||||
link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
|
||||
description: '.aditem-main .aditem-main--middle--description | removeNewline | trim',
|
||||
address: '.aditem-main--top--left | trim | removeNewline',
|
||||
},
|
||||
normalize: normalize,
|
||||
filter: applyBlacklist,
|
||||
};
|
||||
export const metaInformation = {
|
||||
name: 'Ebay Kleinanzeigen',
|
||||
baseUrl: 'https://www.kleinanzeigen.de/',
|
||||
id: 'kleinanzeigen',
|
||||
name: 'Ebay Kleinanzeigen',
|
||||
baseUrl: 'https://www.kleinanzeigen.de/',
|
||||
id: 'kleinanzeigen',
|
||||
};
|
||||
export const init = (sourceConfig, blacklist, blacklistedDistricts) => {
|
||||
config.enabled = sourceConfig.enabled;
|
||||
config.url = sourceConfig.url;
|
||||
appliedBlacklistedDistricts = blacklistedDistricts || [];
|
||||
appliedBlackList = blacklist || [];
|
||||
config.enabled = sourceConfig.enabled;
|
||||
config.url = sourceConfig.url;
|
||||
appliedBlacklistedDistricts = blacklistedDistricts || [];
|
||||
appliedBlackList = blacklist || [];
|
||||
};
|
||||
export {config};
|
||||
export { config };
|
||||
|
||||
@@ -1,44 +1,46 @@
|
||||
import utils, {buildHash} from '../utils.js';
|
||||
import utils, { buildHash } from '../utils.js';
|
||||
|
||||
let appliedBlackList = [];
|
||||
|
||||
function nullOrEmpty(val) {
|
||||
return val == null || val.length === 0;
|
||||
return val == null || val.length === 0;
|
||||
}
|
||||
|
||||
function normalize(o) {
|
||||
const link = nullOrEmpty(o.link) ? 'NO LINK' : `https://www.neubaukompass.de${o.link.substring(o.link.indexOf('/neubau'))}`;
|
||||
const id = buildHash(o.link, o.price);
|
||||
return Object.assign(o, {id, link});
|
||||
const link = nullOrEmpty(o.link)
|
||||
? 'NO LINK'
|
||||
: `https://www.neubaukompass.de${o.link.substring(o.link.indexOf('/neubau'))}`;
|
||||
const id = buildHash(o.link, o.price);
|
||||
return Object.assign(o, { id, link });
|
||||
}
|
||||
|
||||
function applyBlacklist(o) {
|
||||
return !utils.isOneOf(o.title, appliedBlackList);
|
||||
return !utils.isOneOf(o.title, appliedBlackList);
|
||||
}
|
||||
|
||||
const config = {
|
||||
url: null,
|
||||
crawlContainer: '.col-12.mb-4',
|
||||
sortByDateParam: 'Sortierung=Id&Richtung=DESC',
|
||||
waitForSelector: '.nbk-section',
|
||||
crawlFields: {
|
||||
id: 'a@href',
|
||||
title: 'a@title | removeNewline | trim',
|
||||
link: 'a@href',
|
||||
address: '.nbk-project-card__description | removeNewline | trim',
|
||||
price: '.nbk-project-card__spec-item .nbk-project-card__spec-value | removeNewline | trim',
|
||||
},
|
||||
normalize: normalize,
|
||||
filter: applyBlacklist,
|
||||
url: null,
|
||||
crawlContainer: '.col-12.mb-4',
|
||||
sortByDateParam: 'Sortierung=Id&Richtung=DESC',
|
||||
waitForSelector: '.nbk-section',
|
||||
crawlFields: {
|
||||
id: 'a@href',
|
||||
title: 'a@title | removeNewline | trim',
|
||||
link: 'a@href',
|
||||
address: '.nbk-project-card__description | removeNewline | trim',
|
||||
price: '.nbk-project-card__spec-item .nbk-project-card__spec-value | removeNewline | trim',
|
||||
},
|
||||
normalize: normalize,
|
||||
filter: applyBlacklist,
|
||||
};
|
||||
export const init = (sourceConfig, blacklist) => {
|
||||
config.enabled = sourceConfig.enabled;
|
||||
config.url = sourceConfig.url;
|
||||
appliedBlackList = blacklist || [];
|
||||
config.enabled = sourceConfig.enabled;
|
||||
config.url = sourceConfig.url;
|
||||
appliedBlackList = blacklist || [];
|
||||
};
|
||||
export const metaInformation = {
|
||||
name: 'Neubau Kompass',
|
||||
baseUrl: 'https://www.neubaukompass.de/',
|
||||
id: 'neubauKompass',
|
||||
name: 'Neubau Kompass',
|
||||
baseUrl: 'https://www.neubaukompass.de/',
|
||||
id: 'neubauKompass',
|
||||
};
|
||||
export {config};
|
||||
export { config };
|
||||
|
||||
@@ -1,43 +1,43 @@
|
||||
import utils, {buildHash} from '../utils.js';
|
||||
import utils, { buildHash } from '../utils.js';
|
||||
|
||||
let appliedBlackList = [];
|
||||
|
||||
function normalize(o) {
|
||||
const id = buildHash(o.id, o.price);
|
||||
const link = `https://www.wg-gesucht.de${o.link}`;
|
||||
return Object.assign(o, { id, link });
|
||||
const id = buildHash(o.id, o.price);
|
||||
const link = `https://www.wg-gesucht.de${o.link}`;
|
||||
return Object.assign(o, { id, link });
|
||||
}
|
||||
|
||||
function applyBlacklist(o) {
|
||||
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
|
||||
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
|
||||
return o.id != null && titleNotBlacklisted && descNotBlacklisted;
|
||||
const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
|
||||
const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
|
||||
return o.id != null && titleNotBlacklisted && descNotBlacklisted;
|
||||
}
|
||||
|
||||
const config = {
|
||||
url: null,
|
||||
crawlContainer: '#main_column .wgg_card',
|
||||
sortByDateParam: 'sort_column=0&sort_order=0',
|
||||
waitForSelector: 'body',
|
||||
crawlFields: {
|
||||
id: '@data-id',
|
||||
details: '.row .noprint .col-xs-11 |removeNewline |trim',
|
||||
price: '.middle .col-xs-3 |removeNewline |trim',
|
||||
size: '.middle .text-right |removeNewline |trim',
|
||||
title: '.truncate_title a |removeNewline |trim',
|
||||
link: '.truncate_title a@href',
|
||||
},
|
||||
normalize: normalize,
|
||||
filter: applyBlacklist,
|
||||
url: null,
|
||||
crawlContainer: '#main_column .wgg_card',
|
||||
sortByDateParam: 'sort_column=0&sort_order=0',
|
||||
waitForSelector: 'body',
|
||||
crawlFields: {
|
||||
id: '@data-id',
|
||||
details: '.row .noprint .col-xs-11 |removeNewline |trim',
|
||||
price: '.middle .col-xs-3 |removeNewline |trim',
|
||||
size: '.middle .text-right |removeNewline |trim',
|
||||
title: '.truncate_title a |removeNewline |trim',
|
||||
link: '.truncate_title a@href',
|
||||
},
|
||||
normalize: normalize,
|
||||
filter: applyBlacklist,
|
||||
};
|
||||
export const init = (sourceConfig, blacklist) => {
|
||||
config.enabled = sourceConfig.enabled;
|
||||
config.url = sourceConfig.url;
|
||||
appliedBlackList = blacklist || [];
|
||||
config.enabled = sourceConfig.enabled;
|
||||
config.url = sourceConfig.url;
|
||||
appliedBlackList = blacklist || [];
|
||||
};
|
||||
export const metaInformation = {
|
||||
name: 'Wg gesucht',
|
||||
baseUrl: 'https://www.wg-gesucht.de/',
|
||||
id: 'wgGesucht',
|
||||
name: 'Wg gesucht',
|
||||
baseUrl: 'https://www.wg-gesucht.de/',
|
||||
id: 'wgGesucht',
|
||||
};
|
||||
export {config};
|
||||
export { config };
|
||||
|
||||
@@ -8,7 +8,7 @@ export function loadParser(text) {
|
||||
|
||||
export function parse(crawlContainer, crawlFields, text, url) {
|
||||
if (!text) {
|
||||
console.warn('Cannot parse, text was empty for url ', url);
|
||||
console.warn('No content found for ', url);
|
||||
return null;
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { JSONFileSync } from 'lowdb/node';
|
||||
import {config, getDirName} from '../../utils.js';
|
||||
import { config, getDirName } from '../../utils.js';
|
||||
import * as hasher from '../security/hash.js';
|
||||
import { nanoid } from 'nanoid';
|
||||
import * as jobStorage from './jobStorage.js';
|
||||
@@ -7,23 +7,23 @@ import path from 'path';
|
||||
import LowdashAdapter from './LowDashAdapter.js';
|
||||
|
||||
const defaultData = {
|
||||
user: [
|
||||
//you probably want to change the default password ;)
|
||||
{
|
||||
id: nanoid(),
|
||||
lastLogin: Date.now(),
|
||||
username: 'admin',
|
||||
password: hasher.hash('admin'),
|
||||
isAdmin: true,
|
||||
},
|
||||
{
|
||||
id: nanoid(),
|
||||
lastLogin: Date.now(),
|
||||
username: 'demo',
|
||||
password: hasher.hash('demo'),
|
||||
isAdmin: true,
|
||||
},
|
||||
],
|
||||
user: [
|
||||
//you probably want to change the default password ;)
|
||||
{
|
||||
id: nanoid(),
|
||||
lastLogin: Date.now(),
|
||||
username: 'admin',
|
||||
password: hasher.hash('admin'),
|
||||
isAdmin: true,
|
||||
},
|
||||
{
|
||||
id: nanoid(),
|
||||
lastLogin: Date.now(),
|
||||
username: 'demo',
|
||||
password: hasher.hash('demo'),
|
||||
isAdmin: true,
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const file = path.join(getDirName(), '../', 'db/users.json');
|
||||
@@ -86,34 +86,38 @@ export const removeUser = (userId) => {
|
||||
db.chain
|
||||
.set(
|
||||
'user',
|
||||
user.filter((u) => u.id !== userId)
|
||||
user.filter((u) => u.id !== userId),
|
||||
)
|
||||
.value();
|
||||
db.write();
|
||||
};
|
||||
|
||||
export const handleDemoUser = () => {
|
||||
if(!config.demoMode){
|
||||
const user = db.chain.get('user').value();
|
||||
db.chain.get('user').value();
|
||||
db.chain.set('user', user.filter((u) => u.username !== 'demo')).value();
|
||||
db.write();
|
||||
}else {
|
||||
const demoUser = db.chain
|
||||
.get('user')
|
||||
.filter((u) => u.username === 'demo')
|
||||
.value();
|
||||
if (demoUser == null || demoUser.length === 0) {
|
||||
db.chain.get('user')
|
||||
.value()
|
||||
.push({
|
||||
id: nanoid(),
|
||||
username: 'demo',
|
||||
password: hasher.hash('demo'),
|
||||
isAdmin: true,
|
||||
});
|
||||
db.write();
|
||||
}
|
||||
if (!config.demoMode) {
|
||||
const user = db.chain.get('user').value();
|
||||
db.chain
|
||||
.set(
|
||||
'user',
|
||||
user.filter((u) => u.username !== 'demo'),
|
||||
)
|
||||
.value();
|
||||
db.write();
|
||||
} else {
|
||||
const demoUser = db.chain
|
||||
.get('user')
|
||||
.filter((u) => u.username === 'demo')
|
||||
.value();
|
||||
if (demoUser == null || demoUser.length === 0) {
|
||||
db.chain
|
||||
.get('user')
|
||||
.value()
|
||||
.push({
|
||||
id: nanoid(),
|
||||
username: 'demo',
|
||||
password: hasher.hash('demo'),
|
||||
isAdmin: true,
|
||||
});
|
||||
db.write();
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -9,12 +9,9 @@ function inDevMode(){
|
||||
}
|
||||
|
||||
function isOneOf(word, arr) {
|
||||
if (arr == null || arr.length === 0) {
|
||||
return false;
|
||||
}
|
||||
const expression = String.raw`\b(${arr.join('|')})\b`;
|
||||
const blacklist = new RegExp(expression, 'ig');
|
||||
return blacklist.test(word);
|
||||
if (!arr || arr.length === 0 || word == null) return false;
|
||||
const lowerWord = word.toLowerCase();
|
||||
return arr.some(item => lowerWord.indexOf(item.toLowerCase()) !== -1);
|
||||
}
|
||||
|
||||
function nullOrEmpty(val) {
|
||||
|
||||
60
package.json
60
package.json
@@ -1,21 +1,25 @@
|
||||
{
|
||||
"name": "fredy",
|
||||
"version": "11.2.2",
|
||||
"version": "11.3.0",
|
||||
"description": "[F]ind [R]eal [E]states [d]amn eas[y].",
|
||||
"scripts": {
|
||||
"start": "node prod.js",
|
||||
"dev": "yarn && rm -rf ./ui/public/* && vite",
|
||||
"ui": "rm -rf ./ui/public/* && vite",
|
||||
"prod": "yarn && vite build --emptyOutDir",
|
||||
"format": "prettier --write lib/**/*.js ui/src/**/*.jsx test/**/*.js *.js --single-quote --print-width 120",
|
||||
"prepare": "husky",
|
||||
"start:backend": "x-var NODE_ENV=production node index.js",
|
||||
"start:backend:dev": "nodemon --watch index.js --watch lib",
|
||||
"start:frontend": "vite -m production",
|
||||
"start:frontend:dev": "vite",
|
||||
"build:frontend": "vite build",
|
||||
"format": "prettier --write lib/**/*.js ui/src/**/*.jsx test/**/*.js *.js",
|
||||
"format:check": "prettier --check lib/**/*.js ui/src/**/*.jsx test/**/*.js *.js",
|
||||
"test": "mocha --loader=esmock --timeout 3000000 test/**/*.test.js",
|
||||
"lint": "eslint ./index.js ./lib/**/*.js ./test/**/*.js ./ui/src/**/*.jsx"
|
||||
"lint": "eslint index.js lib/**/*.js test/**/*.js ui/src/**/*.jsx",
|
||||
"lint:fix": "yarn lint --fix"
|
||||
},
|
||||
"type": "module",
|
||||
"lint-staged": {
|
||||
"*.js": [
|
||||
"eslint ./index.js ./lib/**/*.js ./test/**/*.js",
|
||||
"prettier --single-quote --print-width 120 --write"
|
||||
"*.{js,jsx}": [
|
||||
"yarn lint",
|
||||
"yarn format"
|
||||
]
|
||||
},
|
||||
"main": "index.js",
|
||||
@@ -50,17 +54,17 @@
|
||||
"Firefox ESR"
|
||||
],
|
||||
"dependencies": {
|
||||
"@douyinfe/semi-ui": "2.79.0",
|
||||
"@douyinfe/semi-ui": "2.83.0",
|
||||
"@rematch/core": "2.2.0",
|
||||
"@rematch/loading": "2.1.2",
|
||||
"@sendgrid/mail": "8.1.5",
|
||||
"@vitejs/plugin-react": "4.4.1",
|
||||
"@vitejs/plugin-react": "4.7.0",
|
||||
"better-sqlite3": "^11.10.0",
|
||||
"body-parser": "2.2.0",
|
||||
"cheerio": "^1.0.0",
|
||||
"cookie-session": "2.1.0",
|
||||
"cheerio": "^1.1.0",
|
||||
"cookie-session": "2.1.1",
|
||||
"handlebars": "4.7.8",
|
||||
"highcharts": "12.2.0",
|
||||
"highcharts": "12.3.0",
|
||||
"highcharts-react-official": "3.2.2",
|
||||
"lodash": "4.17.21",
|
||||
"lowdb": "6.0.1",
|
||||
@@ -70,10 +74,10 @@
|
||||
"node-fetch": "3.3.2",
|
||||
"node-mailjet": "6.0.8",
|
||||
"package-up": "^5.0.0",
|
||||
"puppeteer": "^24.8.2",
|
||||
"puppeteer": "^24.14.0",
|
||||
"puppeteer-extra": "^3.3.6",
|
||||
"puppeteer-extra-plugin-stealth": "^2.11.2",
|
||||
"query-string": "9.1.2",
|
||||
"query-string": "9.2.2",
|
||||
"react": "18.3.1",
|
||||
"react-dom": "18.3.1",
|
||||
"react-redux": "9.2.0",
|
||||
@@ -81,28 +85,30 @@
|
||||
"react-router-dom": "5.3.0",
|
||||
"redux": "5.0.1",
|
||||
"redux-thunk": "3.1.0",
|
||||
"restana": "4.9.9",
|
||||
"serve-static": "1.16.2",
|
||||
"restana": "5.0.0",
|
||||
"serve-static": "2.2.0",
|
||||
"slack": "11.0.2",
|
||||
"string-similarity": "^4.0.4",
|
||||
"vite": "5.4.11"
|
||||
"vite": "7.0.5",
|
||||
"x-var": "^2.1.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@babel/core": "7.27.1",
|
||||
"@babel/eslint-parser": "7.27.1",
|
||||
"@babel/core": "7.27.3",
|
||||
"@babel/eslint-parser": "7.27.5",
|
||||
"@babel/preset-env": "7.27.2",
|
||||
"@babel/preset-react": "7.27.1",
|
||||
"chai": "5.2.0",
|
||||
"chai": "5.2.1",
|
||||
"eslint": "8.56.0",
|
||||
"eslint-config-prettier": "8.8.0",
|
||||
"eslint-plugin-react": "7.37.4",
|
||||
"esmock": "2.7.0",
|
||||
"eslint-plugin-react": "7.37.5",
|
||||
"esmock": "2.7.1",
|
||||
"history": "5.3.0",
|
||||
"husky": "9.1.7",
|
||||
"less": "4.3.0",
|
||||
"less": "4.4.0",
|
||||
"lint-staged": "15.5.2",
|
||||
"mocha": "10.8.2",
|
||||
"prettier": "3.5.3",
|
||||
"nodemon": "^3.1.10",
|
||||
"prettier": "3.6.2",
|
||||
"redux-logger": "3.0.6"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -77,4 +77,4 @@ curl -H "User-Agent: ImmoScout24_1410_30_._" \
|
||||
|
||||
|
||||
## Parameters
|
||||
The parameters between web and mobile are very different which is why we have to translate them. Please see `immoscout-web-translator.js`.
|
||||
The parameters between web and mobile are very different which is why we have to translate them. Please see [/lib/services/immoscout/immoscout-web-translator.js](https://github.com/orangecoding/fredy/blob/master/lib/services/immoscout/immoscout-web-translator.js).
|
||||
|
||||
@@ -1,38 +1,38 @@
|
||||
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
|
||||
import {get} from '../mocks/mockNotification.js';
|
||||
import {mockFredy, providerConfig} from '../utils.js';
|
||||
import {expect} from 'chai';
|
||||
import { get } from '../mocks/mockNotification.js';
|
||||
import { mockFredy, providerConfig } from '../utils.js';
|
||||
import { expect } from 'chai';
|
||||
import * as provider from '../../lib/provider/immonet.js';
|
||||
|
||||
describe('#immonet testsuite()', () => {
|
||||
after(() => {
|
||||
similarityCache.stopCacheCleanup();
|
||||
});
|
||||
provider.init(providerConfig.immonet, [], []);
|
||||
it('should test immonet provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
return await new Promise((resolve) => {
|
||||
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immonet', similarityCache);
|
||||
fredy.execute().then((listing) => {
|
||||
expect(listing).to.be.a('array');
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).to.be.a('object');
|
||||
expect(notificationObj.serviceName).to.equal('immonet');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).to.be.a('string');
|
||||
expect(notify.price).to.be.a('string');
|
||||
expect(notify.size).to.be.a('string');
|
||||
expect(notify.title).to.be.a('string');
|
||||
expect(notify.link).to.be.a('string');
|
||||
expect(notify.address).to.be.a('string');
|
||||
after(() => {
|
||||
similarityCache.stopCacheCleanup();
|
||||
});
|
||||
provider.init(providerConfig.immonet, [], []);
|
||||
it('should test immonet provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
return await new Promise((resolve) => {
|
||||
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immonet', similarityCache);
|
||||
fredy.execute().then((listing) => {
|
||||
expect(listing).to.be.a('array');
|
||||
const notificationObj = get();
|
||||
expect(notificationObj).to.be.a('object');
|
||||
expect(notificationObj.serviceName).to.equal('immonet');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).to.be.a('string');
|
||||
expect(notify.price).to.be.a('string');
|
||||
expect(notify.size).to.be.a('string');
|
||||
expect(notify.title).to.be.a('string');
|
||||
expect(notify.link).to.be.a('string');
|
||||
expect(notify.address).to.be.a('string');
|
||||
|
||||
expect(notify.size).that.does.include('m²');
|
||||
expect(notify.title).to.be.not.empty;
|
||||
expect(notify.address).to.be.not.empty;
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
expect(notify.size).that.does.include('m²');
|
||||
expect(notify.title).to.be.not.empty;
|
||||
expect(notify.address).to.be.not.empty;
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,36 +1,36 @@
|
||||
import * as similarityCache from '../../lib/services/similarity-check/similarityCache.js';
|
||||
import {get} from '../mocks/mockNotification.js';
|
||||
import {mockFredy, providerConfig} from '../utils.js';
|
||||
import {expect} from 'chai';
|
||||
import { get } from '../mocks/mockNotification.js';
|
||||
import { mockFredy, providerConfig } from '../utils.js';
|
||||
import { expect } from 'chai';
|
||||
import * as provider from '../../lib/provider/neubauKompass.js';
|
||||
|
||||
describe('#neubauKompass testsuite()', () => {
|
||||
after(() => {
|
||||
similarityCache.stopCacheCleanup();
|
||||
});
|
||||
provider.init(providerConfig.neubauKompass, [], []);
|
||||
it('should test neubauKompass provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
return await new Promise((resolve) => {
|
||||
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'neubauKompass', similarityCache);
|
||||
fredy.execute().then((listing) => {
|
||||
expect(listing).to.be.a('array');
|
||||
const notificationObj = get();
|
||||
expect(notificationObj.serviceName).to.equal('neubauKompass');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
expect(notify).to.be.a('object');
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).to.be.a('string');
|
||||
expect(notify.title).to.be.a('string');
|
||||
expect(notify.link).to.be.a('string');
|
||||
expect(notify.address).to.be.a('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.title).to.be.not.empty;
|
||||
expect(notify.link).that.does.include('https://www.neubaukompass.de');
|
||||
expect(notify.address).to.be.not.empty;
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
after(() => {
|
||||
similarityCache.stopCacheCleanup();
|
||||
});
|
||||
provider.init(providerConfig.neubauKompass, [], []);
|
||||
it('should test neubauKompass provider', async () => {
|
||||
const Fredy = await mockFredy();
|
||||
return await new Promise((resolve) => {
|
||||
const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'neubauKompass', similarityCache);
|
||||
fredy.execute().then((listing) => {
|
||||
expect(listing).to.be.a('array');
|
||||
const notificationObj = get();
|
||||
expect(notificationObj.serviceName).to.equal('neubauKompass');
|
||||
notificationObj.payload.forEach((notify) => {
|
||||
expect(notify).to.be.a('object');
|
||||
/** check the actual structure **/
|
||||
expect(notify.id).to.be.a('string');
|
||||
expect(notify.title).to.be.a('string');
|
||||
expect(notify.link).to.be.a('string');
|
||||
expect(notify.address).to.be.a('string');
|
||||
/** check the values if possible **/
|
||||
expect(notify.title).to.be.not.empty;
|
||||
expect(notify.link).that.does.include('https://www.neubaukompass.de');
|
||||
expect(notify.address).to.be.not.empty;
|
||||
});
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -34,6 +34,7 @@ describe('similarityCheck', () => {
|
||||
check.setCacheEntry(
|
||||
'where |X| and |Y| are the cardinalities of the two sets (i.e. the number of elements in each set). The Sørensen index equals twice the number of elements common to both sets divided by the sum of the number of elements in each set.',
|
||||
);
|
||||
expect(check.hasSimilarEntries('unrelated text')).to.be.false;
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
import { expect } from 'chai';
|
||||
import {buildHash} from '../../lib/utils.js';
|
||||
import { buildHash } from '../../lib/utils.js';
|
||||
|
||||
describe('utilsCheck', () => {
|
||||
describe('#utilsCheck()', () => {
|
||||
it('should be null when null input', () => {
|
||||
expect(buildHash(null)).to.be.null;
|
||||
});
|
||||
it('should be null when null empty', () => {
|
||||
expect(buildHash('')).to.be.null;
|
||||
});
|
||||
it('should return a value', () => {
|
||||
expect(buildHash('bla', '', null)).to.be.a.string;
|
||||
});
|
||||
it('should be null when null input', () => {
|
||||
expect(buildHash(null)).to.be.null;
|
||||
});
|
||||
it('should be null when null empty', () => {
|
||||
expect(buildHash('')).to.be.null;
|
||||
});
|
||||
it('should return a value', () => {
|
||||
expect(buildHash('bla', '', null)).to.be.a.string;
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
168
ui/src/App.jsx
168
ui/src/App.jsx
@@ -1,4 +1,4 @@
|
||||
import React, {useEffect} from 'react';
|
||||
import React, { useEffect } from 'react';
|
||||
|
||||
import InsufficientPermission from './components/permission/InsufficientPermission';
|
||||
import PermissionAwareRoute from './components/permission/PermissionAwareRoute';
|
||||
@@ -6,106 +6,108 @@ import GeneralSettings from './views/generalSettings/GeneralSettings';
|
||||
import JobMutation from './views/jobs/mutation/JobMutation';
|
||||
import UserMutator from './views/user/mutation/UserMutator';
|
||||
import JobInsight from './views/jobs/insights/JobInsight.jsx';
|
||||
import {useDispatch, useSelector} from 'react-redux';
|
||||
import {Switch, Redirect} from 'react-router-dom';
|
||||
import { useDispatch, useSelector } from 'react-redux';
|
||||
import { Switch, Redirect } from 'react-router-dom';
|
||||
import Logout from './components/logout/Logout';
|
||||
import Logo from './components/logo/Logo';
|
||||
import Menu from './components/menu/Menu';
|
||||
import Login from './views/login/Login';
|
||||
import Users from './views/user/Users';
|
||||
import Jobs from './views/jobs/Jobs';
|
||||
import {Route} from 'react-router';
|
||||
import { Route } from 'react-router';
|
||||
|
||||
import './App.less';
|
||||
import TrackingModal from './components/tracking/TrackingModal.jsx';
|
||||
import {Banner} from '@douyinfe/semi-ui';
|
||||
import { Banner } from '@douyinfe/semi-ui';
|
||||
|
||||
export default function FredyApp() {
|
||||
const dispatch = useDispatch();
|
||||
const [loading, setLoading] = React.useState(true);
|
||||
const currentUser = useSelector((state) => state.user.currentUser);
|
||||
const settings = useSelector((state) => state.generalSettings.settings);
|
||||
const dispatch = useDispatch();
|
||||
const [loading, setLoading] = React.useState(true);
|
||||
const currentUser = useSelector((state) => state.user.currentUser);
|
||||
const settings = useSelector((state) => state.generalSettings.settings);
|
||||
|
||||
useEffect(() => {
|
||||
async function init() {
|
||||
await dispatch.user.getCurrentUser();
|
||||
if (!needsLogin()) {
|
||||
await dispatch.provider.getProvider();
|
||||
await dispatch.jobs.getJobs();
|
||||
await dispatch.jobs.getProcessingTimes();
|
||||
await dispatch.notificationAdapter.getAdapter();
|
||||
await dispatch.generalSettings.getGeneralSettings();
|
||||
}
|
||||
setLoading(false);
|
||||
}
|
||||
useEffect(() => {
|
||||
async function init() {
|
||||
await dispatch.user.getCurrentUser();
|
||||
if (!needsLogin()) {
|
||||
await dispatch.provider.getProvider();
|
||||
await dispatch.jobs.getJobs();
|
||||
await dispatch.jobs.getProcessingTimes();
|
||||
await dispatch.notificationAdapter.getAdapter();
|
||||
await dispatch.generalSettings.getGeneralSettings();
|
||||
}
|
||||
setLoading(false);
|
||||
}
|
||||
|
||||
init();
|
||||
}, [currentUser?.userId]);
|
||||
init();
|
||||
}, [currentUser?.userId]);
|
||||
|
||||
const needsLogin = () => {
|
||||
return currentUser == null || Object.keys(currentUser).length === 0;
|
||||
};
|
||||
const needsLogin = () => {
|
||||
return currentUser == null || Object.keys(currentUser).length === 0;
|
||||
};
|
||||
|
||||
const isAdmin = () => currentUser != null && currentUser.isAdmin;
|
||||
const isAdmin = () => currentUser != null && currentUser.isAdmin;
|
||||
|
||||
const login = () => (
|
||||
const login = () => (
|
||||
<Switch>
|
||||
<Route name="Login" path={'/login'} component={Login} />
|
||||
<Redirect from="*" to={'/login'} />
|
||||
</Switch>
|
||||
);
|
||||
|
||||
return loading ? null : needsLogin() ? (
|
||||
login()
|
||||
) : (
|
||||
<div className="app">
|
||||
<div className="app__container">
|
||||
<Logout />
|
||||
<Logo width={190} white />
|
||||
<Menu isAdmin={isAdmin()} />
|
||||
|
||||
{settings.demoMode && (
|
||||
<>
|
||||
<Banner
|
||||
fullMode={true}
|
||||
type="info"
|
||||
bordered
|
||||
closeIcon={null}
|
||||
description="You're currently viewing the demo version of Fredy. Jobs won't scrape websites, and any changes you make will be reverted at midnight."
|
||||
/>
|
||||
<br />
|
||||
</>
|
||||
)}
|
||||
{settings.analyticsEnabled === null && !settings.demoMode && <TrackingModal />}
|
||||
<Switch>
|
||||
<Route name="Login" path={'/login'} component={Login}/>
|
||||
<Redirect from="*" to={'/login'}/>
|
||||
<Route name="Insufficient Permission" path={'/403'} component={InsufficientPermission} />
|
||||
<Route name="Create new Job" path={'/jobs/new'} component={JobMutation} />
|
||||
<Route name="Edit a Job" path={'/jobs/edit/:jobId'} component={JobMutation} />
|
||||
<Route name="Insights into a Job" path={'/jobs/insights/:jobId'} component={JobInsight} />
|
||||
<Route name="Job overview" path={'/jobs'} component={Jobs} />
|
||||
<PermissionAwareRoute
|
||||
name="Create new User"
|
||||
path="/users/new"
|
||||
component={<UserMutator />}
|
||||
currentUser={currentUser}
|
||||
/>
|
||||
<PermissionAwareRoute
|
||||
name="Edit a user"
|
||||
path="/users/edit/:userId"
|
||||
component={<UserMutator />}
|
||||
currentUser={currentUser}
|
||||
/>
|
||||
<PermissionAwareRoute name="Users" path="/users" component={<Users />} currentUser={currentUser} />
|
||||
<PermissionAwareRoute
|
||||
name="General Settings"
|
||||
path="/generalSettings"
|
||||
component={<GeneralSettings />}
|
||||
currentUser={currentUser}
|
||||
/>
|
||||
|
||||
<Redirect from="/" to={'/jobs'} />
|
||||
</Switch>
|
||||
);
|
||||
|
||||
return loading ? null : needsLogin() ? (
|
||||
login()
|
||||
) : (
|
||||
<div className="app">
|
||||
<div className="app__container">
|
||||
<Logout/>
|
||||
<Logo width={190} white/>
|
||||
<Menu isAdmin={isAdmin()}/>
|
||||
|
||||
{settings.demoMode && (
|
||||
<>
|
||||
<Banner fullMode={true}
|
||||
type="info"
|
||||
bordered
|
||||
closeIcon={null}
|
||||
description="You're currently viewing the demo version of Fredy. Jobs won't scrape websites, and any changes you make will be reverted at midnight."
|
||||
/>
|
||||
<br/>
|
||||
</>)}
|
||||
{(settings.analyticsEnabled === null && !settings.demoMode) && <TrackingModal/>}
|
||||
<Switch>
|
||||
<Route name="Insufficient Permission" path={'/403'} component={InsufficientPermission}/>
|
||||
<Route name="Create new Job" path={'/jobs/new'} component={JobMutation}/>
|
||||
<Route name="Edit a Job" path={'/jobs/edit/:jobId'} component={JobMutation}/>
|
||||
<Route name="Insights into a Job" path={'/jobs/insights/:jobId'} component={JobInsight}/>
|
||||
<Route name="Job overview" path={'/jobs'} component={Jobs}/>
|
||||
<PermissionAwareRoute
|
||||
name="Create new User"
|
||||
path="/users/new"
|
||||
component={<UserMutator/>}
|
||||
currentUser={currentUser}
|
||||
/>
|
||||
<PermissionAwareRoute
|
||||
name="Edit a user"
|
||||
path="/users/edit/:userId"
|
||||
component={<UserMutator/>}
|
||||
currentUser={currentUser}
|
||||
/>
|
||||
<PermissionAwareRoute name="Users" path="/users" component={<Users/>} currentUser={currentUser}/>
|
||||
<PermissionAwareRoute
|
||||
name="General Settings"
|
||||
path="/generalSettings"
|
||||
component={<GeneralSettings/>}
|
||||
currentUser={currentUser}
|
||||
/>
|
||||
|
||||
<Redirect from="/" to={'/jobs'}/>
|
||||
</Switch>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
FredyApp.displayName = 'FredyApp';
|
||||
|
||||
@@ -40,4 +40,8 @@ a:active {
|
||||
color: #54a9ff;
|
||||
background-color: transparent;
|
||||
text-decoration: underline;
|
||||
}
|
||||
}
|
||||
|
||||
.semi-icon:not(.semi-tabs-bar .semi-tabs-tab .semi-icon) {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
@@ -23,5 +23,5 @@ root.render(
|
||||
<App />
|
||||
</LocaleProvider>
|
||||
</HashRouter>
|
||||
</Provider>
|
||||
</Provider>,
|
||||
);
|
||||
|
||||
@@ -30,7 +30,7 @@ export default function ProviderTable({ providerData = [], onRemove } = {}) {
|
||||
render: (_, record) => {
|
||||
return (
|
||||
<div style={{ float: 'right' }}>
|
||||
<Button type="danger" icon={<IconDelete />} onClick={() => onRemove(record.id)} />
|
||||
<Button type="danger" icon={<IconDelete />} onClick={() => onRemove(record.url)} />
|
||||
</div>
|
||||
);
|
||||
},
|
||||
|
||||
@@ -1,48 +1,56 @@
|
||||
import React from 'react';
|
||||
import {Modal} from '@douyinfe/semi-ui';
|
||||
import { Modal } from '@douyinfe/semi-ui';
|
||||
import Logo from '../logo/Logo.jsx';
|
||||
import {xhrPost} from '../../services/xhr.js';
|
||||
import { xhrPost } from '../../services/xhr.js';
|
||||
|
||||
import './TrackingModal.less';
|
||||
import inDevelopment from '../../services/developmentMode.js';
|
||||
|
||||
const saveResponse = async (analyticsEnabled) => {
|
||||
await xhrPost('/api/admin/generalSettings', {
|
||||
analyticsEnabled
|
||||
});
|
||||
await xhrPost('/api/admin/generalSettings', {
|
||||
analyticsEnabled,
|
||||
});
|
||||
};
|
||||
|
||||
export default function TrackingModal() {
|
||||
if (inDevelopment()) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return <Modal
|
||||
visible={true}
|
||||
onOk={async () => {
|
||||
await saveResponse(true);
|
||||
location.reload();
|
||||
}}
|
||||
onCancel={async () => {
|
||||
await saveResponse(false);
|
||||
location.reload();
|
||||
}}
|
||||
maskClosable={false}
|
||||
closable={false}
|
||||
okText="Yes! I want to help"
|
||||
cancelText="No, thanks"
|
||||
return (
|
||||
<Modal
|
||||
visible={true}
|
||||
onOk={async () => {
|
||||
await saveResponse(true);
|
||||
location.reload();
|
||||
}}
|
||||
onCancel={async () => {
|
||||
await saveResponse(false);
|
||||
location.reload();
|
||||
}}
|
||||
maskClosable={false}
|
||||
closable={false}
|
||||
okText="Yes! I want to help"
|
||||
cancelText="No, thanks"
|
||||
>
|
||||
<Logo white/>
|
||||
<div className="trackingModal__description">
|
||||
<p>Hey 👋</p>
|
||||
<p>Fed up with popups? Yeah, me too. But this one’s important, and I promise it will only appear once ;)</p>
|
||||
<p>Fredy is completely free (and will always remain free). If you’d like, you can support me by donating
|
||||
through my GitHub, but there’s absolutely no obligation to do so.</p>
|
||||
<p>However, it would be a huge
|
||||
help if you’d allow me to collect some analytical data. Wait, before you click "no", let me explain. If
|
||||
you
|
||||
agree, Fredy will send a ping to my Mixpanel project each time it runs.</p>
|
||||
<p>The data includes: names of
|
||||
active adapters/providers, OS, architecture, Node version, and language. The information is entirely
|
||||
anonymous and helps me understand which adapters/providers are most frequently used.</p>
|
||||
<p>Thanks🤘</p>
|
||||
</div>
|
||||
</Modal>;
|
||||
|
||||
}
|
||||
<Logo white />
|
||||
<div className="trackingModal__description">
|
||||
<p>Hey 👋</p>
|
||||
<p>Fed up with popups? Yeah, me too. But this one’s important, and I promise it will only appear once ;)</p>
|
||||
<p>
|
||||
Fredy is completely free (and will always remain free). If you’d like, you can support me by donating through
|
||||
my GitHub, but there’s absolutely no obligation to do so.
|
||||
</p>
|
||||
<p>
|
||||
However, it would be a huge help if you’d allow me to collect some analytical data. Wait, before you click
|
||||
"no", let me explain. If you agree, Fredy will send a ping to my Mixpanel project each time it runs.
|
||||
</p>
|
||||
<p>
|
||||
The data includes: names of active adapters/providers, OS, architecture, Node version, and language. The
|
||||
information is entirely anonymous and helps me understand which adapters/providers are most frequently used.
|
||||
</p>
|
||||
<p>Thanks🤘</p>
|
||||
</div>
|
||||
</Modal>
|
||||
);
|
||||
}
|
||||
|
||||
4
ui/src/services/developmentMode.js
Normal file
4
ui/src/services/developmentMode.js
Normal file
@@ -0,0 +1,4 @@
|
||||
export default function isDevelopmentMode(){
|
||||
const inDevMode= import.meta.env.MODE;
|
||||
return inDevMode != null && inDevMode === 'development';
|
||||
}
|
||||
@@ -1,261 +1,251 @@
|
||||
import React from 'react';
|
||||
|
||||
import {useDispatch, useSelector} from 'react-redux';
|
||||
import { useDispatch, useSelector } from 'react-redux';
|
||||
|
||||
import {Divider, TimePicker, Button, Checkbox} from '@douyinfe/semi-ui';
|
||||
import {InputNumber} from '@douyinfe/semi-ui';
|
||||
import { Divider, TimePicker, Button, Checkbox } from '@douyinfe/semi-ui';
|
||||
import { InputNumber } from '@douyinfe/semi-ui';
|
||||
import Headline from '../../components/headline/Headline';
|
||||
import {xhrPost} from '../../services/xhr';
|
||||
import {SegmentPart} from '../../components/segment/SegmentPart';
|
||||
import {Banner, Toast} from '@douyinfe/semi-ui';
|
||||
import {IconSave, IconCalendar, IconRefresh, IconSignal, IconLineChartStroked, IconSearch} from '@douyinfe/semi-icons';
|
||||
import { xhrPost } from '../../services/xhr';
|
||||
import { SegmentPart } from '../../components/segment/SegmentPart';
|
||||
import { Banner, Toast } from '@douyinfe/semi-ui';
|
||||
import {
|
||||
IconSave,
|
||||
IconCalendar,
|
||||
IconRefresh,
|
||||
IconSignal,
|
||||
IconLineChartStroked,
|
||||
IconSearch,
|
||||
} from '@douyinfe/semi-icons';
|
||||
import './GeneralSettings.less';
|
||||
|
||||
function formatFromTimestamp(ts) {
|
||||
const date = new Date(ts);
|
||||
return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
|
||||
const date = new Date(ts);
|
||||
return `${date.getHours()}:${date.getMinutes() > 9 ? date.getMinutes() : '0' + date.getMinutes()}`;
|
||||
}
|
||||
|
||||
function formatFromTBackend(time) {
|
||||
if (time == null || time.length === 0) {
|
||||
return null;
|
||||
}
|
||||
const date = new Date();
|
||||
const split = time.split(':');
|
||||
date.setHours(split[0]);
|
||||
date.setMinutes(split[1]);
|
||||
return date.getTime();
|
||||
if (time == null || time.length === 0) {
|
||||
return null;
|
||||
}
|
||||
const date = new Date();
|
||||
const split = time.split(':');
|
||||
date.setHours(split[0]);
|
||||
date.setMinutes(split[1]);
|
||||
return date.getTime();
|
||||
}
|
||||
|
||||
const GeneralSettings = function GeneralSettings() {
|
||||
const dispatch = useDispatch();
|
||||
const [loading, setLoading] = React.useState(true);
|
||||
const dispatch = useDispatch();
|
||||
const [loading, setLoading] = React.useState(true);
|
||||
|
||||
const settings = useSelector((state) => state.generalSettings.settings);
|
||||
const settings = useSelector((state) => state.generalSettings.settings);
|
||||
|
||||
const [interval, setInterval] = React.useState('');
|
||||
const [port, setPort] = React.useState('');
|
||||
const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
|
||||
const [workingHourTo, setWorkingHourTo] = React.useState(null);
|
||||
const [demoMode, setDemoMode] = React.useState(null);
|
||||
const [analyticsEnabled, setAnalyticsEnabled] = React.useState(null);
|
||||
const [interval, setInterval] = React.useState('');
|
||||
const [port, setPort] = React.useState('');
|
||||
const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
|
||||
const [workingHourTo, setWorkingHourTo] = React.useState(null);
|
||||
const [demoMode, setDemoMode] = React.useState(null);
|
||||
const [analyticsEnabled, setAnalyticsEnabled] = React.useState(null);
|
||||
|
||||
React.useEffect(() => {
|
||||
async function init() {
|
||||
await dispatch.generalSettings.getGeneralSettings();
|
||||
setLoading(false);
|
||||
}
|
||||
React.useEffect(() => {
|
||||
async function init() {
|
||||
await dispatch.generalSettings.getGeneralSettings();
|
||||
setLoading(false);
|
||||
}
|
||||
|
||||
init();
|
||||
}, []);
|
||||
init();
|
||||
}, []);
|
||||
|
||||
React.useEffect(() => {
|
||||
async function init() {
|
||||
setInterval(settings?.interval);
|
||||
setPort(settings?.port);
|
||||
setWorkingHourFrom(settings?.workingHours?.from);
|
||||
setWorkingHourTo(settings?.workingHours?.to);
|
||||
setAnalyticsEnabled(settings?.analyticsEnabled || false);
|
||||
setDemoMode(settings?.demoMode || false);
|
||||
}
|
||||
React.useEffect(() => {
|
||||
async function init() {
|
||||
setInterval(settings?.interval);
|
||||
setPort(settings?.port);
|
||||
setWorkingHourFrom(settings?.workingHours?.from);
|
||||
setWorkingHourTo(settings?.workingHours?.to);
|
||||
setAnalyticsEnabled(settings?.analyticsEnabled || false);
|
||||
setDemoMode(settings?.demoMode || false);
|
||||
}
|
||||
|
||||
init();
|
||||
}, [settings]);
|
||||
init();
|
||||
}, [settings]);
|
||||
|
||||
const nullOrEmpty = (val) => val == null || val.length === 0;
|
||||
const nullOrEmpty = (val) => val == null || val.length === 0;
|
||||
|
||||
const throwMessage = (message, type) => {
|
||||
if (type === 'error') {
|
||||
Toast.error(message);
|
||||
} else {
|
||||
Toast.success(message);
|
||||
}
|
||||
};
|
||||
const throwMessage = (message, type) => {
|
||||
if (type === 'error') {
|
||||
Toast.error(message);
|
||||
} else {
|
||||
Toast.success(message);
|
||||
}
|
||||
};
|
||||
|
||||
const onStore = async () => {
|
||||
if (nullOrEmpty(interval)) {
|
||||
throwMessage('Interval may not be empty.', 'error');
|
||||
return;
|
||||
}
|
||||
if (nullOrEmpty(port)) {
|
||||
throwMessage('Port may not be empty.', 'error');
|
||||
return;
|
||||
}
|
||||
if (
|
||||
(!nullOrEmpty(workingHourFrom) && nullOrEmpty(workingHourTo)) ||
|
||||
(nullOrEmpty(workingHourFrom) && !nullOrEmpty(workingHourTo))
|
||||
) {
|
||||
throwMessage('Working hours to and from must be set if either to or from has been set before.', 'error');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await xhrPost('/api/admin/generalSettings', {
|
||||
interval,
|
||||
port,
|
||||
workingHours: {
|
||||
from: workingHourFrom,
|
||||
to: workingHourTo,
|
||||
},
|
||||
demoMode,
|
||||
analyticsEnabled
|
||||
});
|
||||
} catch (exception) {
|
||||
console.error(exception);
|
||||
if(exception?.json?.message != null){
|
||||
throwMessage(exception.json.message, 'error');
|
||||
}else {
|
||||
throwMessage('Error while trying to store settings.', 'error');
|
||||
}
|
||||
return;
|
||||
}
|
||||
throwMessage('Settings stored successfully. We will reload your browser in 3 seconds.', 'success');
|
||||
setTimeout(()=>{
|
||||
location.reload();
|
||||
}, 3000);
|
||||
};
|
||||
const onStore = async () => {
|
||||
if (nullOrEmpty(interval)) {
|
||||
throwMessage('Interval may not be empty.', 'error');
|
||||
return;
|
||||
}
|
||||
if (nullOrEmpty(port)) {
|
||||
throwMessage('Port may not be empty.', 'error');
|
||||
return;
|
||||
}
|
||||
if (
|
||||
(!nullOrEmpty(workingHourFrom) && nullOrEmpty(workingHourTo)) ||
|
||||
(nullOrEmpty(workingHourFrom) && !nullOrEmpty(workingHourTo))
|
||||
) {
|
||||
throwMessage('Working hours to and from must be set if either to or from has been set before.', 'error');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await xhrPost('/api/admin/generalSettings', {
|
||||
interval,
|
||||
port,
|
||||
workingHours: {
|
||||
from: workingHourFrom,
|
||||
to: workingHourTo,
|
||||
},
|
||||
demoMode,
|
||||
analyticsEnabled,
|
||||
});
|
||||
} catch (exception) {
|
||||
console.error(exception);
|
||||
if (exception?.json?.message != null) {
|
||||
throwMessage(exception.json.message, 'error');
|
||||
} else {
|
||||
throwMessage('Error while trying to store settings.', 'error');
|
||||
}
|
||||
return;
|
||||
}
|
||||
throwMessage('Settings stored successfully. We will reload your browser in 3 seconds.', 'success');
|
||||
setTimeout(() => {
|
||||
location.reload();
|
||||
}, 3000);
|
||||
};
|
||||
|
||||
return (
|
||||
<div>
|
||||
{!loading && (
|
||||
<React.Fragment>
|
||||
<Headline text="General Settings"/>
|
||||
<div>
|
||||
<SegmentPart
|
||||
name="Interval"
|
||||
helpText="Interval in minutes for running queries against the configured services."
|
||||
Icon={IconRefresh}
|
||||
>
|
||||
<InputNumber
|
||||
min={0}
|
||||
max={1440}
|
||||
placeholder="Interval in minutes"
|
||||
value={interval}
|
||||
formatter={(value) => `${value}`.replace(/\D/g, '')}
|
||||
onChange={(value) => setInterval(value)}
|
||||
suffix={'minutes'}
|
||||
/>
|
||||
</SegmentPart>
|
||||
<Divider margin="1rem"/>
|
||||
<SegmentPart name="Port" helpText="Port on which Fredy is running." Icon={IconSignal}>
|
||||
<InputNumber
|
||||
min={0}
|
||||
max={99999}
|
||||
placeholder="Port"
|
||||
value={port}
|
||||
formatter={(value) => `${value}`.replace(/\D/g, '')}
|
||||
onChange={(value) => setPort(value)}
|
||||
/>
|
||||
</SegmentPart>
|
||||
<Divider margin="1rem"/>
|
||||
<SegmentPart
|
||||
name="Working hours"
|
||||
helpText="During this hours, Fredy will search for new apartments. If nothing is configured, Fredy will search around the clock."
|
||||
Icon={IconCalendar}
|
||||
>
|
||||
<div className="generalSettings__timePickerContainer">
|
||||
<TimePicker
|
||||
format={'HH:mm'}
|
||||
insetLabel="From"
|
||||
value={formatFromTBackend(workingHourFrom)}
|
||||
placeholder=""
|
||||
onChange={(val) => {
|
||||
setWorkingHourFrom(val == null ? null : formatFromTimestamp(val));
|
||||
}}
|
||||
/>
|
||||
<TimePicker
|
||||
format={'HH:mm'}
|
||||
insetLabel="Until"
|
||||
value={formatFromTBackend(workingHourTo)}
|
||||
placeholder=""
|
||||
onChange={(val) => {
|
||||
setWorkingHourTo(val == null ? null : formatFromTimestamp(val));
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</SegmentPart>
|
||||
<Divider margin="1rem"/>
|
||||
return (
|
||||
<div>
|
||||
{!loading && (
|
||||
<React.Fragment>
|
||||
<Headline text="General Settings" />
|
||||
<div>
|
||||
<SegmentPart
|
||||
name="Interval"
|
||||
helpText="Interval in minutes for running queries against the configured services."
|
||||
Icon={IconRefresh}
|
||||
>
|
||||
<InputNumber
|
||||
min={0}
|
||||
max={1440}
|
||||
placeholder="Interval in minutes"
|
||||
value={interval}
|
||||
formatter={(value) => `${value}`.replace(/\D/g, '')}
|
||||
onChange={(value) => setInterval(value)}
|
||||
suffix={'minutes'}
|
||||
/>
|
||||
</SegmentPart>
|
||||
<Divider margin="1rem" />
|
||||
<SegmentPart name="Port" helpText="Port on which Fredy is running." Icon={IconSignal}>
|
||||
<InputNumber
|
||||
min={0}
|
||||
max={99999}
|
||||
placeholder="Port"
|
||||
value={port}
|
||||
formatter={(value) => `${value}`.replace(/\D/g, '')}
|
||||
onChange={(value) => setPort(value)}
|
||||
/>
|
||||
</SegmentPart>
|
||||
<Divider margin="1rem" />
|
||||
<SegmentPart
|
||||
name="Working hours"
|
||||
helpText="During this hours, Fredy will search for new apartments. If nothing is configured, Fredy will search around the clock."
|
||||
Icon={IconCalendar}
|
||||
>
|
||||
<div className="generalSettings__timePickerContainer">
|
||||
<TimePicker
|
||||
format={'HH:mm'}
|
||||
insetLabel="From"
|
||||
value={formatFromTBackend(workingHourFrom)}
|
||||
placeholder=""
|
||||
onChange={(val) => {
|
||||
setWorkingHourFrom(val == null ? null : formatFromTimestamp(val));
|
||||
}}
|
||||
/>
|
||||
<TimePicker
|
||||
format={'HH:mm'}
|
||||
insetLabel="Until"
|
||||
value={formatFromTBackend(workingHourTo)}
|
||||
placeholder=""
|
||||
onChange={(val) => {
|
||||
setWorkingHourTo(val == null ? null : formatFromTimestamp(val));
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</SegmentPart>
|
||||
<Divider margin="1rem" />
|
||||
|
||||
<SegmentPart
|
||||
name="Analytics"
|
||||
helpText="Insights into the usage of Fredy."
|
||||
Icon={IconLineChartStroked}
|
||||
>
|
||||
<Banner
|
||||
fullMode={false}
|
||||
type="info"
|
||||
closeIcon={null}
|
||||
title={
|
||||
<div style={{fontWeight: 600, fontSize: '14px', lineHeight: '20px'}}>
|
||||
Explanation
|
||||
</div>
|
||||
}
|
||||
style={{marginBottom: '1rem'}}
|
||||
description={
|
||||
<div>
|
||||
Analytics are disabled by default. If you choose to enable them, we will begin tracking the following:<br/>
|
||||
<ul>
|
||||
<li>Name of active provider (e.g. Immoscout)</li>
|
||||
<li>Name of active adapter (e.g. Console)</li>
|
||||
<li>language</li>
|
||||
<li>os</li>
|
||||
<li>node version</li>
|
||||
<li>arch</li>
|
||||
</ul>
|
||||
The data is sent anonymously and helps me understand which providers or adapters are being used the most. In the end it helps me to improve fredy.
|
||||
</div>
|
||||
}
|
||||
/>
|
||||
<SegmentPart name="Analytics" helpText="Insights into the usage of Fredy." Icon={IconLineChartStroked}>
|
||||
<Banner
|
||||
fullMode={false}
|
||||
type="info"
|
||||
closeIcon={null}
|
||||
title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Explanation</div>}
|
||||
style={{ marginBottom: '1rem' }}
|
||||
description={
|
||||
<div>
|
||||
Analytics are disabled by default. If you choose to enable them, we will begin tracking the
|
||||
following:
|
||||
<br />
|
||||
<ul>
|
||||
<li>Name of active provider (e.g. Immoscout)</li>
|
||||
<li>Name of active adapter (e.g. Console)</li>
|
||||
<li>language</li>
|
||||
<li>os</li>
|
||||
<li>node version</li>
|
||||
<li>arch</li>
|
||||
</ul>
|
||||
The data is sent anonymously and helps me understand which providers or adapters are being used the
|
||||
most. In the end it helps me to improve fredy.
|
||||
</div>
|
||||
}
|
||||
/>
|
||||
|
||||
<Checkbox
|
||||
checked={analyticsEnabled}
|
||||
onChange={(e) => setAnalyticsEnabled(e.target.checked)}
|
||||
> Enabled
|
||||
</Checkbox>
|
||||
<Checkbox checked={analyticsEnabled} onChange={(e) => setAnalyticsEnabled(e.target.checked)}>
|
||||
{' '}
|
||||
Enabled
|
||||
</Checkbox>
|
||||
</SegmentPart>
|
||||
|
||||
</SegmentPart>
|
||||
<Divider margin="1rem" />
|
||||
|
||||
<Divider margin="1rem"/>
|
||||
<SegmentPart name="Demo Mode" helpText="If enabled, Fredy runs in demo mode." Icon={IconSearch}>
|
||||
<Banner
|
||||
fullMode={false}
|
||||
type="info"
|
||||
closeIcon={null}
|
||||
title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Explanation</div>}
|
||||
style={{ marginBottom: '1rem' }}
|
||||
description={
|
||||
<div>
|
||||
In demo mode, Fredy will not (really) search for any real estates. Fredy is in a lockdown mode. Also
|
||||
all database files will be set back to the default values at midnight.
|
||||
</div>
|
||||
}
|
||||
/>
|
||||
|
||||
<SegmentPart
|
||||
name="Demo Mode"
|
||||
helpText="If enabled, Fredy runs in demo mode."
|
||||
Icon={IconSearch}
|
||||
>
|
||||
<Banner
|
||||
fullMode={false}
|
||||
type="info"
|
||||
closeIcon={null}
|
||||
title={
|
||||
<div style={{fontWeight: 600, fontSize: '14px', lineHeight: '20px'}}>
|
||||
Explanation
|
||||
</div>
|
||||
}
|
||||
style={{marginBottom: '1rem'}}
|
||||
description={
|
||||
<div>
|
||||
In demo mode, Fredy will not (really) search for any real estates. Fredy is in a lockdown mode. Also
|
||||
all database files will be set back to the default values at midnight.
|
||||
</div>
|
||||
}
|
||||
/>
|
||||
<Checkbox checked={demoMode} onChange={(e) => setDemoMode(e.target.checked)}>
|
||||
{' '}
|
||||
Enabled
|
||||
</Checkbox>
|
||||
</SegmentPart>
|
||||
|
||||
<Checkbox
|
||||
checked={demoMode}
|
||||
onChange={(e) => setDemoMode(e.target.checked)}
|
||||
> Enabled
|
||||
</Checkbox>
|
||||
|
||||
</SegmentPart>
|
||||
|
||||
<Divider margin="1rem"/>
|
||||
<Button type="primary" theme="solid" onClick={onStore} icon={<IconSave/>}>
|
||||
Save
|
||||
</Button>
|
||||
</div>
|
||||
</React.Fragment>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
<Divider margin="1rem" />
|
||||
<Button type="primary" theme="solid" onClick={onStore} icon={<IconSave />}>
|
||||
Save
|
||||
</Button>
|
||||
</div>
|
||||
</React.Fragment>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
export default GeneralSettings;
|
||||
|
||||
@@ -1,32 +1,32 @@
|
||||
import React from 'react';
|
||||
import {format} from '../../services/time/timeService';
|
||||
import {Banner, Descriptions} from '@douyinfe/semi-ui';
|
||||
import { format } from '../../services/time/timeService';
|
||||
import { Descriptions } from '@douyinfe/semi-ui';
|
||||
|
||||
export default function ProcessingTimes({processingTimes = {}}) {
|
||||
if (Object.keys(processingTimes).length === 0) {
|
||||
return null;
|
||||
}
|
||||
return (
|
||||
<>
|
||||
<Descriptions
|
||||
row
|
||||
size="small"
|
||||
style={{
|
||||
backgroundColor: '#35363c',
|
||||
borderRadius: '4px',
|
||||
padding: '10px',
|
||||
}}
|
||||
>
|
||||
<Descriptions.Item itemKey="Processing Interval">{processingTimes.interval} min</Descriptions.Item>
|
||||
{processingTimes.lastRun && (
|
||||
<>
|
||||
<Descriptions.Item itemKey="Last run">{format(processingTimes.lastRun)}</Descriptions.Item>
|
||||
<Descriptions.Item itemKey="Next run">
|
||||
{format(processingTimes.lastRun + processingTimes.interval * 60000)}
|
||||
</Descriptions.Item>
|
||||
</>
|
||||
)}
|
||||
</Descriptions>
|
||||
</>
|
||||
);
|
||||
export default function ProcessingTimes({ processingTimes = {} }) {
|
||||
if (Object.keys(processingTimes).length === 0) {
|
||||
return null;
|
||||
}
|
||||
return (
|
||||
<>
|
||||
<Descriptions
|
||||
row
|
||||
size="small"
|
||||
style={{
|
||||
backgroundColor: '#35363c',
|
||||
borderRadius: '4px',
|
||||
padding: '10px',
|
||||
}}
|
||||
>
|
||||
<Descriptions.Item itemKey="Processing Interval">{processingTimes.interval} min</Descriptions.Item>
|
||||
{processingTimes.lastRun && (
|
||||
<>
|
||||
<Descriptions.Item itemKey="Last run">{format(processingTimes.lastRun)}</Descriptions.Item>
|
||||
<Descriptions.Item itemKey="Next run">
|
||||
{format(processingTimes.lastRun + processingTimes.interval * 60000)}
|
||||
</Descriptions.Item>
|
||||
</>
|
||||
)}
|
||||
</Descriptions>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -94,7 +94,7 @@ export default function JobMutator() {
|
||||
<form>
|
||||
<SegmentPart name="Name">
|
||||
<Input
|
||||
autofocus
|
||||
autoFocus
|
||||
type="text"
|
||||
maxLength={40}
|
||||
placeholder="Name"
|
||||
@@ -124,8 +124,8 @@ export default function JobMutator() {
|
||||
|
||||
<ProviderTable
|
||||
providerData={providerData}
|
||||
onRemove={(providerId) => {
|
||||
setProviderData(providerData.filter((provider) => provider.id !== providerId));
|
||||
onRemove={(providerUrl) => {
|
||||
setProviderData(providerData.filter((provider) => provider.url !== providerUrl));
|
||||
}}
|
||||
/>
|
||||
</SegmentPart>
|
||||
|
||||
@@ -27,7 +27,7 @@ const validate = (selectedAdapter) => {
|
||||
}
|
||||
if (uiElement.type === 'number') {
|
||||
const numberValue = parseFloat(uiElement.value);
|
||||
if(isNaN(numberValue) || numberValue < 0) {
|
||||
if (isNaN(numberValue) || numberValue < 0) {
|
||||
results.push('A number field cannot contain anything else and must be > 0.');
|
||||
continue;
|
||||
}
|
||||
@@ -83,7 +83,7 @@ export default function NotificationAdapterMutator({
|
||||
id: selectedAdapter.id,
|
||||
name: selectedAdapter.name,
|
||||
fields: selectedAdapter.fields || {},
|
||||
})
|
||||
}),
|
||||
);
|
||||
|
||||
setSelectedAdapter(null);
|
||||
@@ -114,7 +114,7 @@ export default function NotificationAdapterMutator({
|
||||
setSuccessMessage('It seems like it worked! Please check your service.');
|
||||
})
|
||||
.catch((error) =>
|
||||
setValidationMessage(`This did not work :-( I've received the following error: ${error.json.message}`)
|
||||
setValidationMessage(`This did not work :-( I've received the following error: ${error.json.message}`),
|
||||
);
|
||||
};
|
||||
|
||||
@@ -229,7 +229,7 @@ export default function NotificationAdapterMutator({
|
||||
.filter((option) =>
|
||||
editNotificationAdapter != null
|
||||
? true
|
||||
: selected.find((selectedOption) => selectedOption.id === option.key) == null
|
||||
: selected.find((selectedOption) => selectedOption.id === option.key) == null,
|
||||
)
|
||||
.sort(sortAdapter)}
|
||||
onChange={(value) => {
|
||||
|
||||
@@ -45,7 +45,7 @@ export default function ProviderMutator({ onVisibilityChanged, visible = false,
|
||||
url: providerUrl,
|
||||
id: selectedProvider.id,
|
||||
name: selectedProvider.name,
|
||||
})
|
||||
}),
|
||||
);
|
||||
setProviderUrl(null);
|
||||
setSelectedProvider(null);
|
||||
@@ -101,7 +101,7 @@ export default function ProviderMutator({ onVisibilityChanged, visible = false,
|
||||
description={
|
||||
<div>
|
||||
<p>
|
||||
Currently, our Immoscout implementation does not drawing shapes on a map. Use a radius instead.
|
||||
Currently, our Immoscout implementation does not support drawing shapes on a map. Use a radius instead.
|
||||
</p>
|
||||
</div>
|
||||
}
|
||||
|
||||
@@ -1,102 +1,105 @@
|
||||
import React, {useEffect} from 'react';
|
||||
import React, { useEffect } from 'react';
|
||||
|
||||
import cityBackground from '../../assets/city_background.jpg';
|
||||
import Logo from '../../components/logo/Logo';
|
||||
import {xhrPost} from '../../services/xhr';
|
||||
import {useHistory} from 'react-router';
|
||||
import {useDispatch, useSelector} from 'react-redux';
|
||||
import {Input, Button, Banner} from '@douyinfe/semi-ui';
|
||||
import { xhrPost } from '../../services/xhr';
|
||||
import { useHistory } from 'react-router';
|
||||
import { useDispatch, useSelector } from 'react-redux';
|
||||
import { Input, Button, Banner } from '@douyinfe/semi-ui';
|
||||
|
||||
import './login.less';
|
||||
import {IconUser, IconLock} from '@douyinfe/semi-icons';
|
||||
import { IconUser, IconLock } from '@douyinfe/semi-icons';
|
||||
|
||||
export default function Login() {
|
||||
const dispatch = useDispatch();
|
||||
const [username, setUserName] = React.useState('');
|
||||
const [password, setPassword] = React.useState('');
|
||||
const [error, setError] = React.useState(null);
|
||||
const demoMode = useSelector((state) => state.demoMode.demoMode || false);
|
||||
const history = useHistory();
|
||||
const dispatch = useDispatch();
|
||||
const [username, setUserName] = React.useState('');
|
||||
const [password, setPassword] = React.useState('');
|
||||
const [error, setError] = React.useState(null);
|
||||
const demoMode = useSelector((state) => state.demoMode.demoMode || false);
|
||||
const history = useHistory();
|
||||
|
||||
useEffect(() => {
|
||||
async function init() {
|
||||
await dispatch.demoMode.getDemoMode();
|
||||
}
|
||||
useEffect(() => {
|
||||
async function init() {
|
||||
await dispatch.demoMode.getDemoMode();
|
||||
}
|
||||
|
||||
init();
|
||||
}, []);
|
||||
init();
|
||||
}, []);
|
||||
|
||||
const tryLogin = async () => {
|
||||
if (username.length === 0 || password.length === 0) {
|
||||
setError('Username and password are mandatory.');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await xhrPost('/api/login', {
|
||||
username,
|
||||
password,
|
||||
});
|
||||
setError(null);
|
||||
} catch (Exception) {
|
||||
setError('Login not successful...');
|
||||
return;
|
||||
}
|
||||
await dispatch.user.getCurrentUser();
|
||||
history.push('/jobs');
|
||||
};
|
||||
const tryLogin = async () => {
|
||||
if (username.length === 0 || password.length === 0) {
|
||||
setError('Username and password are mandatory.');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await xhrPost('/api/login', {
|
||||
username,
|
||||
password,
|
||||
});
|
||||
setError(null);
|
||||
} catch (Exception) {
|
||||
setError('Login not successful...');
|
||||
return;
|
||||
}
|
||||
await dispatch.user.getCurrentUser();
|
||||
history.push('/jobs');
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="login">
|
||||
<div className="login__bgImage" style={{background: `url("${cityBackground}")`}}/>
|
||||
<Logo/>
|
||||
<form>
|
||||
<div className="login__loginWrapper">
|
||||
{error && <Banner type="danger" closeIcon={null} description={error}/>}
|
||||
<Input
|
||||
size="large"
|
||||
prefix={<IconUser/>}
|
||||
placeholder="Username"
|
||||
value={username}
|
||||
showClear
|
||||
style={{marginTop: error ? '1rem' : '4rem'}}
|
||||
autoFocus
|
||||
onChange={(value) => setUserName(value)}
|
||||
onKeyPress={async (e) => {
|
||||
if (e.key === 'Enter') {
|
||||
await tryLogin();
|
||||
}
|
||||
}}
|
||||
/>
|
||||
return (
|
||||
<div className="login">
|
||||
<div className="login__bgImage" style={{ background: `url("${cityBackground}")` }} />
|
||||
<Logo />
|
||||
<form>
|
||||
<div className="login__loginWrapper">
|
||||
{error && <Banner type="danger" closeIcon={null} description={error} />}
|
||||
<Input
|
||||
size="large"
|
||||
prefix={<IconUser />}
|
||||
placeholder="Username"
|
||||
value={username}
|
||||
showClear
|
||||
style={{ marginTop: error ? '1rem' : '4rem' }}
|
||||
autoFocus
|
||||
onChange={(value) => setUserName(value)}
|
||||
onKeyPress={async (e) => {
|
||||
if (e.key === 'Enter') {
|
||||
await tryLogin();
|
||||
}
|
||||
}}
|
||||
/>
|
||||
|
||||
<Input
|
||||
size="large"
|
||||
mode="password"
|
||||
prefix={<IconLock/>}
|
||||
value={password}
|
||||
placeholder="Password"
|
||||
style={{marginTop: '2rem'}}
|
||||
onChange={(value) => setPassword(value)}
|
||||
onKeyPress={async (e) => {
|
||||
if (e.key === 'Enter') {
|
||||
await tryLogin();
|
||||
}
|
||||
}}
|
||||
/>
|
||||
<Input
|
||||
size="large"
|
||||
mode="password"
|
||||
prefix={<IconLock />}
|
||||
value={password}
|
||||
placeholder="Password"
|
||||
style={{ marginTop: '2rem' }}
|
||||
onChange={(value) => setPassword(value)}
|
||||
onKeyPress={async (e) => {
|
||||
if (e.key === 'Enter') {
|
||||
await tryLogin();
|
||||
}
|
||||
}}
|
||||
/>
|
||||
|
||||
<Button type="primary" onClick={tryLogin} theme="solid" style={{marginTop: '3rem'}}>
|
||||
Login
|
||||
</Button>
|
||||
<br/>
|
||||
{demoMode && <Banner fullMode={true}
|
||||
type="info"
|
||||
bordered
|
||||
closeIcon={null}
|
||||
description="This is the demo version of Fredy. Use 'demo' as both the username and password to log in."
|
||||
/>}
|
||||
</div>
|
||||
</form>
|
||||
<Button type="primary" onClick={tryLogin} theme="solid" style={{ marginTop: '3rem' }}>
|
||||
Login
|
||||
</Button>
|
||||
<br />
|
||||
{demoMode && (
|
||||
<Banner
|
||||
fullMode={true}
|
||||
type="info"
|
||||
bordered
|
||||
closeIcon={null}
|
||||
description="This is the demo version of Fredy. Use 'demo' as both the username and password to log in."
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
</form>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
Login.displayName = 'Login';
|
||||
|
||||
@@ -6,6 +6,7 @@ export default defineConfig({
|
||||
build: {
|
||||
chunkSizeWarningLimit: 9999999,
|
||||
outDir: './ui/public',
|
||||
emptyOutDir: true,
|
||||
},
|
||||
plugins: [react()],
|
||||
server: {
|
||||
|
||||
Reference in New Issue
Block a user