fixing end of file issue by upgrading node-fetch

* Add provider for immobilien.de

.eslintrc.js

@@ -6,7 +6,7 @@ module.exports = {
     browser: true,
     mocha: true,
   },
-  parser: 'babel-eslint',
+  parser: '@babel/eslint-parser',
   extends: ['eslint:recommended', 'prettier'],
   plugins: ['react'],
   globals: {

CHANGELOG.md

@@ -2,6 +2,12 @@ Newer release changelog see https://github.com/orangecoding/fredy/releases
 
 ------------
 
+###### [V5.5.0]
+- Upgrading dependencies
+- fixing provider
+- allow multiple instances of 1 provider  
+- __BREAKING__: Minimum node version is now 16
+
 ###### [V5.4.6]
 - Adding Instana node.js monitoring
 - 

README.md

@@ -8,9 +8,16 @@ _Fredy_ scrapes multiple services (Immonet, Immowelt etc.) and send new listings
 
 If _Fredy_ finds matching results, it will send them to you via Slack, Email, Telegram etc. (More adapters can be configured.) As _Fredy_ stores the listings it has found, new results will not be sent to you twice (and as a side-effect, _Fredy_ can show some statistics). Furthermore, _Fredy_ checks duplicates per scraping so that the same listings are not being sent twice or more when posted on various platforms (which happens more often than one might think).
 
+# Sponsorship [![](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=%23fe8e86)](https://github.com/sponsors/orangecoding)    
+If you like my work, consider becoming a sponsor. I'm not expecting anybody to pay for _Fredy_ or any other Open Source Project I'm maintaining, however keep in mind, I'm doing all of this in my spare time :) Thanks.
+
+<img src="https://github.com/orangecoding/fredy/blob/master/doc/jetbrains.png" width="200">   
+
+_Fredy_ is supported by JetBrains under Open Source Support Program
+
 ## Usage  
 
-- Make sure to use Node.js 12 or above
+- Make sure to use Node.js 16 or above
 - Run the following commands:
 ```ssh
 yarn (or npm install)
@@ -33,20 +40,20 @@ _Fredy_ will start with the default port, set to `9998`. You can access _Fredy_
 ## Understanding the fundamentals
 There are 3 important parts in Fredy, that you need to understand to leverage the full power of _Fredy_.
 
-#### Adapter
-_Fredy_ supports multiple services. Immonet, Immowelt and Ebay are just a few examples. Those services are called adapters within _Fredy_. When creating a new job, you can choose one or more adapters.    
-An adapter contains the URL that points to the search results for the respective service. If you go to immonet.de and search for something, the displayed URL in the browser is what the adapter needs to do its magic.   
+#### Provider
+_Fredy_ supports multiple services. Immonet, Immowelt and Ebay are just a few examples. Those services are called providers within _Fredy_. When creating a new job, you can choose one or more providers.
+A provider contains the URL that points to the search results for the respective service. If you go to immonet.de and search for something, the displayed URL in the browser is what the provider needs to do its magic.
 **It is important that you order the search results by date, so that _Fredy_ always picks the latest results first!**
 
-#### Provider
-_Fredy_ supports multiple providers, such as Slack, SendGrid, Telegram etc. A search job can have as many providers as supported by _Fredy_. Each provider needs different configuration values, which you have to provide when using them. A provider dictactes how the frontend renders by telling the frontend what information it needs in order to send listings to the user.
+#### Adapter
+_Fredy_ supports multiple adapters, such as Slack, SendGrid, Telegram etc. A search job can have as many adapters as supported by _Fredy_. Each adapter needs different configuration values, which you have to provide when using them. A adapter dictactes how the frontend renders by telling the frontend what information it needs in order to send listings to the user.
 
 #### Jobs
 A Job wraps adapters and providers. _Fredy_ runs the configured jobs in a specific interval (can be configured in `/conf/config.json`).
 
 ## Creating your first job
 To create your first job, click on the button "Create New Job" on the job table. The job creation dialog should be self-explanatory, however there is one important thing.
-When configuring adapters, before copying the URL from your browser, make sure that you have sorted the results by date to make sure _Fredy_ always picks the latest results first.
+When configuring providers, before copying the URL from your browser, make sure that you have sorted the results by date to make sure _Fredy_ always picks the latest results first.
 
 ## User management
 As an administrator, you can create, edit and remove users from _Fredy_. Be careful, each job is connected to the user that has created the job. If you remove the user, their jobs will also be removed.
@@ -54,11 +61,16 @@ As an administrator, you can create, edit and remove users from _Fredy_. Be care
 # Development
 
 ### Running Fredy in development mode
-To run _Fredy_ in development mode, you need to run the backend & frontend separately. Run the backend in your favorite IDE, the frontend can be started from the terminal.
+To run _Fredy_ in development mode, you need to run the backend & frontend separately.
+Start the backend with:
+```shell
+yarn run start
+```
+For the frontend, run:
 ```shell
 yarn run dev
 ```
-You should now be able to access _Fredy_ from your browser. Go to `http://localhost:9000`.
+You should now be able to access _Fredy_ from your browser. Check your Terminal to see what port the frontend is running on.
 
 ### Running Tests
 To run the tests, run

conf/config.json

@@ -1 +1 @@
-{"interval":"60","port":9998,"scrapingAnt":{"apiKey":""},"workingHours":{"from":"","to":""}}
+{"interval":"60","port":9998,"scrapingAnt":{"apiKey":"","proxy":"datacenter"},"workingHours":{"from":"","to":""}}

index.html

@@ -4,7 +4,7 @@
     <meta charset="UTF-8"
           name="viewport"
           content="user-scalable=no, width=device-width, initial-scale=1, maximum-scale=1">
-    <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css">
+    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/semantic-ui@2/dist/semantic.min.css">
     <meta name="google" content="notranslate">
 
     <title>Fredy</title>
@@ -13,5 +13,5 @@
 
 <div id="fredy" style="position: absolute;top: 0;left: 0;right: 0;bottom: 0;"></div>
 </body>
-<script src="fredy.bundle.js"></script>
+<script type="module" src="/ui/src/Index.jsx"></script>
 </html>

index.js

@@ -20,6 +20,7 @@ const { duringWorkingHoursOrNotSet } = require('./lib/utils');
 require('./lib/api/api');
 
 //assuming interval is always in minutes
+
 const INTERVAL = config.interval * 60 * 1000;
 
 /* eslint-disable no-console */
@@ -31,33 +32,20 @@ setInterval(
 
     if (isDuringWorkingHoursOrNotSet) {
       config.lastRun = Date.now();
+      const fetchedProvider = provider
+        .filter((provider) => provider.endsWith('.js'))
+        .map((pro) => require(`${path}/${pro}`));
+
       jobStorage
         .getJobs()
         .filter((job) => job.enabled)
         .forEach((job) => {
-          const providerIds = job.provider.map((provider) => provider.id);
-
-          provider
-            .filter((provider) => provider.endsWith('.js'))
-            .map((pro) => require(`${path}/${pro}`))
-            .filter((provider) => providerIds.indexOf(provider.metaInformation.id) !== -1)
-            .forEach(async (pro) => {
-              const providerId = pro.metaInformation.id;
-              if (providerId == null || providerId.length === 0) {
-                throw new Error('Provider id must not be empty. => ' + pro);
-              }
-              const providerConfig = job.provider.find((jobProvider) => jobProvider.id === providerId);
-              if (providerConfig == null) {
-                throw new Error(`Provider Config for provider with id ${providerId} not found.`);
-              }
-              pro.init(providerConfig, job.blacklist);
-              await new FredyRuntime(
-                pro.config,
-                job.notificationAdapter,
-                providerId,
-                job.id,
-                similarityCache
-              ).execute();
+          job.provider
+            .filter((p) => fetchedProvider.find((fp) => fp.metaInformation.id === p.id) != null)
+            .forEach(async (prov) => {
+              const pro = fetchedProvider.find((fp) => fp.metaInformation.id === prov.id);
+              pro.init(prov, job.blacklist);
+              await new FredyRuntime(pro.config, job.notificationAdapter, prov.id, job.id, similarityCache).execute();
               setLastJobExecution(job.id);
             });
         });

lib/api/routes/jobRouter.js

@@ -1,6 +1,6 @@
 const service = require('restana')();
 const jobRouter = service.newRouter();
-const axios = require('axios');
+const fetch = require('node-fetch');
 const jobStorage = require('../../services/storage/jobStorage');
 const userStorage = require('../../services/storage/userStorage');
 const immoscoutProvider = require('../../provider/immoscout');
@@ -34,10 +34,8 @@ jobRouter.get('/processingTimes', async (req, res) => {
 
   if (config.scrapingAnt.apiKey != null && config.scrapingAnt.apiKey.length > 0) {
     try {
-      const result = await axios({
-        url: `https://api.scrapingant.com/v1/usage?x-api-key=${config.scrapingAnt.apiKey}`,
-      });
-      scrapingAntData = result.data;
+      const response = await fetch(`https://api.scrapingant.com/v1/usage?x-api-key=${config.scrapingAnt.apiKey}`);
+      scrapingAntData = await response.json();
     } catch (Exception) {
       console.error('Could not query plan data from scraping ant.', Exception);
     }

lib/notification/adapter/mattermost.js

@@ -1,6 +1,6 @@
 const { markdown2Html } = require('../../services/markdown');
 const { getJob } = require('../../services/storage/jobStorage');
-const axios = require('axios');
+const fetch = require('node-fetch');
 
 /**
  * sends new listings to mattermost
@@ -21,9 +21,13 @@ exports.send = ({ serviceName, newListings, notificationConfig, jobKey }) => {
     (o) => `| [${o.title}](${o.link}) | ` + [o.address, o.size.replace(/2m/g, '$m^2$'), o.price].join(' | ') + ' |\n'
   );
 
-  return axios.post(`${webhook}`, {
-    channel: channel,
-    text: message,
+  return fetch(webhook, {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: {
+      channel: channel,
+      text: message,
+    },
   });
 };
 

lib/notification/adapter/telegram.js

@@ -1,7 +1,9 @@
 const { markdown2Html } = require('../../services/markdown');
 const { getJob } = require('../../services/storage/jobStorage');
-const axios = require('axios');
+const fetch = require('node-fetch');
 
+const MAX_ENTITIES_PER_CHUNK = 8;
+const RATE_LIMIT_INTERVAL = 1010;
 /**
  * splitting an array into chunks because Telegram only allows for messages up to
  * 4096 chars, thus we have to split messages into chunks
@@ -29,7 +31,7 @@ exports.send = ({ serviceName, newListings, notificationConfig, jobKey }) => {
   const jobName = job == null ? jobKey : job.name;
 
   //we have to split messages into chunk, because otherwise messages are going to become too big and will fail
-  const chunks = arrayChunks(newListings, 3);
+  const chunks = arrayChunks(newListings, MAX_ENTITIES_PER_CHUNK);
 
   const promises = chunks.map((chunk) => {
     let message = `<i>${jobName}</i> (${serviceName}) found <b>${newListings.length}</b> new listings:\n\n`;
@@ -40,11 +42,28 @@ exports.send = ({ serviceName, newListings, notificationConfig, jobKey }) => {
         '\n\n'
     );
 
-    return axios.post(`https://api.telegram.org/bot${token}/sendMessage`, {
-      chat_id: chatId,
-      text: message,
-      parse_mode: 'HTML',
-      disable_web_page_preview: true,
+    /**
+     * This is to not break the rate limit. It is to only send 1 message per second
+     */
+    return new Promise((resolve, reject) => {
+      setTimeout(() => {
+        fetch(`https://api.telegram.org/bot${token}/sendMessage`, {
+          method: 'post',
+          body: JSON.stringify({
+            chat_id: chatId,
+            text: message,
+            parse_mode: 'HTML',
+            disable_web_page_preview: true,
+          }),
+          headers: { 'Content-Type': 'application/json' },
+        })
+          .then(() => {
+            resolve();
+          })
+          .catch(() => {
+            reject();
+          });
+      }, RATE_LIMIT_INTERVAL);
     });
   });
 

lib/provider/immobilienDe.js

@@ -0,0 +1,59 @@
+const utils = require('../utils');
+
+let appliedBlackList = [];
+
+function shortenLink(link) {
+  return link.substring(0, link.indexOf('?'));
+}
+
+function parseId(shortenedLink) {
+  return shortenedLink.substring(shortenedLink.lastIndexOf('/') + 1);
+}
+
+function normalize(o) {
+  const id = parseId(shortenLink(o.link));
+  const size = o.size || 'N/A m²';
+  const price = o.price || 'N/A €';
+  const title = o.title || 'No title available';
+  const address = o.address || 'No address available';
+  const link = shortenLink(o.link);
+  return Object.assign(o, { id, price, size, title, address, link });
+}
+
+function applyBlacklist(o) {
+  const titleNotBlacklisted = !utils.isOneOf(o.title, appliedBlackList);
+  const descNotBlacklisted = !utils.isOneOf(o.description, appliedBlackList);
+
+  return titleNotBlacklisted && descNotBlacklisted;
+}
+
+const config = {
+  url: null,
+  crawlContainer: '.estates_list .list_immo a._ref',
+  sortByDateParam: 'sort_col=*created_ts&sort_dir=desc',
+  crawlFields: {
+    price: '.list_entry .immo_preis .label_info',
+    size: '.list_entry .flaeche .label_info | removeNewline | trim',
+    title: '.list_entry .part_text h3 span',
+    description: '.list_entry .description | trim',
+    link: '@href',
+    address: '.list_entry .place',
+  },
+  paginate: '.list_immo .blocknav .blocknav_list li.next a@href',
+  normalize: normalize,
+  filter: applyBlacklist,
+};
+
+exports.init = (sourceConfig, blacklist) => {
+  config.enabled = sourceConfig.enabled;
+  config.url = sourceConfig.url;
+  appliedBlackList = blacklist || [];
+};
+
+exports.metaInformation = {
+  name: 'Immobilien.de',
+  baseUrl: 'https://www.immobilien.de/',
+  id: __filename.slice(__dirname.length + 1, -3),
+};
+
+exports.config = config;

lib/services/requestDriver.js

@@ -1,31 +1,73 @@
-const axios = require('axios');
-const axiosRetry = require('axios-retry');
+const fetch = require('node-fetch');
+const config = require('../../conf/config.json');
 
-axiosRetry(axios, { retryDelay: axiosRetry.exponentialDelay, retries: 3 });
+const { makeUrlResidential } = require('./scrapingAnt');
+//if ScrapingAnt got blocked, this http status is returned
+const BLOCKED_HTTP_STATUS = 423;
+const NOT_FOUND_HTTP_STATUS = 404;
+const MAX_RETRIES_SCRAPING_ANT = 10;
+const EXPECTED_STATUS_CODES = [BLOCKED_HTTP_STATUS, NOT_FOUND_HTTP_STATUS];
 
 function makeDriver(headers = {}) {
   let cookies = '';
 
-  return async function driver(context, callback) {
+  async function scrapingAntDriver(context, callback, retryCounter = 0) {
+    const proxyType = config.scrapingAnt?.proxy || 'datacenter';
+
     try {
-      const url = context.url;
-      const result = await axios({
-        url,
+      const url = proxyType === 'residential' ? makeUrlResidential(context.url) : context.url;
+      const response = await fetch(url, {
+        headers: {
+          ...headers,
+          cookie: cookies,
+        },
+      });
+
+      const result = await response.text();
+      if (cookies.length === 0) {
+        cookies = response.headers.raw()['set-cookie'] || [];
+      }
+
+      callback(null, result);
+    } catch (exception) {
+      /* eslint-disable no-console */
+      if (!EXPECTED_STATUS_CODES.includes(exception.response?.status)) {
+        console.error(`Error while trying to scrape data from scraping ant. Received error: ${exception.message}`);
+        callback(null, []);
+        return;
+      }
+
+      if (retryCounter <= MAX_RETRIES_SCRAPING_ANT) {
+        retryCounter++;
+        console.debug(`ScrapingAnt got blocked. Retrying ${retryCounter} / ${MAX_RETRIES_SCRAPING_ANT}`);
+        await scrapingAntDriver(context, callback, retryCounter);
+      } else {
+        console.error(`Error while trying to scrape data from scraping ant. Received error: ${exception.message}`);
+        callback(null, []);
+      }
+      /* eslint-enable no-console */
+    }
+  }
+
+  /**
+   * The regular request driver is taking care of everyting, that doesn't need to be scraped by ScrapingAnt (which is
+   * everything != Immoscout as of writing this)
+   */
+  return async function driver(context, callback) {
+    if (context.url.toLowerCase().indexOf('scrapingant') !== -1) {
+      return scrapingAntDriver(context, callback);
+    }
+
+    try {
+      const response = await fetch(context.url, {
         headers: {
           ...headers,
           Cookie: cookies,
         },
       });
 
-      if (typeof result.data === 'object' && url.toLowerCase().indexOf('scrapingant') !== -1) {
-        //assume we have gotten a response from scrapingAnt
-        if (cookies.length === 0) {
-          cookies = result.data.cookies;
-        }
-        callback(null, result.data.content);
-      } else {
-        callback(null, result.data);
-      }
+      const result = await response.text();
+      callback(null, result);
     } catch (exception) {
       console.error(`Error while trying to scrape data. Received error: ${exception.message}`);
       callback(null, []);

lib/services/scrapingAnt.js

@@ -1,6 +1,5 @@
 const { metaInformation } = require('../provider/immoscout');
 //to better configure re-capture chose a random proxy each time we do a call
-const proxies = ['ae', 'br', 'cn', 'de', 'es', 'fr', 'gb', 'hk', 'in', 'it', 'il', 'jp', 'nl', 'ru', 'sa', 'us', 'cz'];
 const config = require('../../conf/config.json');
 
 const isImmoscout = (id) => {
@@ -8,13 +7,9 @@ const isImmoscout = (id) => {
 };
 
 exports.transformUrlForScrapingAnt = (url, id) => {
-  const randomProxy = proxies[Math.floor(Math.random() * proxies.length)];
-
   if (isImmoscout(id)) {
     //only do calls to scrapingAnt when dealing with Immoscout
-    url = `https://api.scrapingant.com/v1/general?url=${encodeURIComponent(
-      url
-    )}&proxy_country=${randomProxy}&proxy_type=residential`;
+    url = `https://api.scrapingant.com/v1/general?url=${encodeURIComponent(url)}&proxy_type=datacenter`;
   }
   return url;
 };
@@ -24,3 +19,7 @@ exports.isScrapingAntApiKeySet = () => {
 };
 
 exports.isImmoscout = isImmoscout;
+
+exports.makeUrlResidential = (url) => {
+  return url.replace('datacenter', 'residential');
+};

package.json

@@ -1,13 +1,14 @@
 {
   "name": "fredy",
-  "version": "5.5.0",
+  "version": "6.0.2",
   "description": "[F]ind [R]eal [E]states [d]amn eas[y].",
   "scripts": {
     "start": "node index.js",
-    "dev": "yarn && export BUILD_DEV='true' && export NODE_ENV='development' && webpack serve --progress --color --config ./webpack.dev.js",
-    "prod": "export BUILD_DEV='false' && webpack --node-env=production --config ./webpack.prod.js",
+    "dev": "yarn && rm -rf ./ui/public/* && vite",
+    "ui": "rm -rf ./ui/public/* && vite",
+    "prod": "yarn && vite build --emptyOutDir",
     "format": "prettier --write lib/**/*.js ui/src/**/*.js test/**/*.js *.js --single-quote --print-width 120",
-    "test": "mocha --timeout 20000 test/**/*.test.js",
+    "test": "mocha --timeout 3000000 test/**/*.test.js",
     "lint": "eslint ./index.js ./lib/**/*.js ./test/**/*.js"
   },
   "husky": {
@@ -43,7 +44,7 @@
   },
   "license": "MIT",
   "engines": {
-    "node": ">=14.0.0",
+    "node": ">=16.0.0",
     "npm": ">=7.0.0"
   },
   "browserslist": [
@@ -55,63 +56,52 @@
   "dependencies": {
     "@rematch/core": "2.2.0",
     "@rematch/loading": "2.1.2",
-    "@sendgrid/mail": "7.6.2",
-    "axios": "0.26.1",
-    "axios-retry": "^3.2.4",
-    "better-sqlite3": "^7.5.0",
-    "body-parser": "1.19.2",
+    "@sendgrid/mail": "7.7.0",
+    "@vitejs/plugin-react": "3.1.0",
+    "better-sqlite3": "8.1.0",
+    "body-parser": "1.20.1",
     "cookie-session": "2.0.0",
     "handlebars": "4.7.7",
-    "highcharts": "10.0.0",
+    "highcharts": "10.3.3",
     "highcharts-react-official": "3.1.0",
     "lowdb": "1.0.0",
     "markdown": "^0.5.0",
-    "nanoid": "3.3.1",
-    "node-mailjet": "3.3.7",
-    "query-string": "7.1.1",
-    "react": "17.0.2",
-    "react-dom": "17.0.2",
-    "react-redux": "7.2.6",
+    "nanoid": "3.3.3",
+    "node-fetch": "2.6.9",
+    "node-mailjet": "3.3.13",
+    "query-string": "7.1.3",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-redux": "8.0.5",
     "react-router": "5.2.1",
     "react-router-dom": "5.3.0",
-    "react-switch": "^6.0.0",
-    "redux": "4.1.2",
-    "redux-thunk": "2.4.1",
-    "restana": "4.9.3",
-    "semantic-ui-react": "2.1.2",
+    "react-switch": "7.0.0",
+    "redux": "4.2.1",
+    "redux-thunk": "2.4.2",
+    "restana": "4.9.7",
+    "semantic-ui-react": "2.1.4",
     "serve-static": "1.15.0",
     "slack": "11.0.2",
     "string-similarity": "^4.0.4",
+    "vite": "4.1.1",
     "x-ray": "2.3.4"
   },
   "devDependencies": {
-    "@babel/core": "7.17.8",
-    "@babel/preset-env": "7.16.11",
-    "@babel/preset-react": "7.16.7",
-    "babel-eslint": "10.1.0",
-    "babel-loader": "8.2.4",
-    "chai": "4.3.6",
-    "clean-webpack-plugin": "4.0.0",
-    "copy-webpack-plugin": "10.2.4",
-    "css-loader": "6.7.1",
-    "eslint": "7.32.0",
-    "eslint-config-prettier": "8.5.0",
-    "eslint-plugin-react": "7.29.4",
-    "file-loader": "6.2.0",
+    "@babel/core": "7.20.12",
+    "@babel/eslint-parser": "^7.19.1",
+    "@babel/preset-env": "7.20.2",
+    "@babel/preset-react": "7.18.6",
+    "chai": "4.3.7",
+    "eslint": "8.34.0",
+    "eslint-config-prettier": "8.6.0",
+    "eslint-plugin-react": "7.32.2",
     "history": "5.3.0",
     "husky": "4.3.8",
-    "less": "4.1.2",
-    "less-loader": "10.2.0",
-    "lint-staged": "12.3.7",
-    "mocha": "9.2.2",
-    "prettier": "2.6.1",
+    "less": "4.1.3",
+    "lint-staged": "13.1.2",
+    "mocha": "10.2.0",
+    "prettier": "2.8.4",
     "proxyquire": "2.1.3",
-    "redux-logger": "3.0.6",
-    "style-loader": "3.3.1",
-    "url-loader": "4.1.1",
-    "webpack": "5.70.0",
-    "webpack-cli": "4.9.2",
-    "webpack-dev-server": "3.11.2",
-    "webpack-merge": "5.8.0"
+    "redux-logger": "3.0.6"
   }
 }

test/provider/einsAImmobilien.test.js

@@ -22,7 +22,7 @@ describe('#einsAImmobilien testsuite()', () => {
 
   it('should test einsAImmobilien provider', async () => {
     return await new Promise((resolve) => {
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'einsAImmobilien', similarityCache);
       fredy.execute().then((listings) => {
         expect(listings).to.be.a('array');
 
@@ -39,7 +39,6 @@ describe('#einsAImmobilien testsuite()', () => {
           expect(notify.link).to.be.a('string');
 
           /** check the values if possible **/
-          expect(notify.price).that.does.include('EUR');
           expect(notify.size).to.be.not.empty;
           expect(notify.title).to.be.not.empty;
           expect(notify.link).that.does.include('https://www.1a-immobilienmarkt.de');

test/provider/immobilienDe.test.js

@@ -0,0 +1,51 @@
+const similarityCache = require('../../lib/services/similarity-check/similarityCache');
+const mockNotification = require('../mocks/mockNotification');
+const providerConfig = require('./testProvider.json');
+const mockStore = require('../mocks/mockStore');
+const proxyquire = require('proxyquire').noCallThru();
+const expect = require('chai').expect;
+const provider = require('../../lib/provider/immobilienDe');
+
+describe('#immobilien.de testsuite()', () => {
+  after(() => {
+    similarityCache.stopCacheCleanup();
+  });
+
+  provider.init(providerConfig.immobilienDe, [], []);
+  const Fredy = proxyquire('../../lib/FredyRuntime', {
+    './services/storage/listingsStorage': {
+      ...mockStore,
+    },
+    './notification/notify': mockNotification,
+  });
+
+  it('should test immobilien.de provider', async () => {
+    return await new Promise((resolve) => {
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      fredy.execute().then((listing) => {
+        expect(listing).to.be.a('array');
+
+        const notificationObj = mockNotification.get();
+        expect(notificationObj).to.be.a('object');
+        expect(notificationObj.serviceName).to.equal('immobilienDe');
+        notificationObj.payload.forEach((notify) => {
+          /** check the actual structure **/
+          expect(notify.id).to.be.a('string');
+          expect(notify.price).to.be.a('string');
+          expect(notify.size).to.be.a('string');
+          expect(notify.title).to.be.a('string');
+          expect(notify.link).to.be.a('string');
+          expect(notify.address).to.be.a('string');
+
+          /** check the values if possible **/
+          expect(notify.price).that.does.include('€');
+          expect(notify.size).that.does.include('m²');
+          expect(notify.title).to.be.not.empty;
+          expect(notify.link).that.does.include('https://www.immobilien.de');
+          expect(notify.address).to.be.not.empty;
+        });
+        resolve();
+      });
+    });
+  });
+});

test/provider/immonet.test.js

@@ -21,7 +21,7 @@ describe('#immonet testsuite()', () => {
 
   it('should test immonet provider', async () => {
     return await new Promise((resolve) => {
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immonet', similarityCache);
       fredy.execute().then((listing) => {
         expect(listing).to.be.a('array');
 

test/provider/immoscout.test.js

@@ -29,7 +29,7 @@ describe('#immoscout testsuite()', () => {
         return;
       }
 
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immoscout', similarityCache);
       fredy.execute().then((listing) => {
         expect(listing).to.be.a('array');
 

test/provider/immoswp.test.js

@@ -21,7 +21,7 @@ describe('#immoswp testsuite()', () => {
 
   it('should test immoswp provider', async () => {
     return await new Promise((resolve) => {
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immoswp', similarityCache);
       fredy.execute().then((listing) => {
         expect(listing).to.be.a('array');
 

test/provider/immowelt.test.js

@@ -20,7 +20,7 @@ describe('#immowelt testsuite()', () => {
     });
 
     return await new Promise((resolve) => {
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'immowelt', similarityCache);
       fredy.execute().then((listing) => {
         expect(listing).to.be.a('array');
 

test/provider/kleinanzeigen.test.js

@@ -20,7 +20,7 @@ describe('#kleinanzeigen testsuite()', () => {
     });
 
     return await new Promise((resolve) => {
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'kleinanzeigen', similarityCache);
       fredy.execute().then((listing) => {
         expect(listing).to.be.a('array');
 
@@ -31,10 +31,8 @@ describe('#kleinanzeigen testsuite()', () => {
         notificationObj.payload.forEach((notify) => {
           /** check the actual structure **/
           expect(notify.id).to.be.a('number');
-          expect(notify.size).to.be.a('string');
           expect(notify.title).to.be.a('string');
           expect(notify.link).to.be.a('string');
-          expect(notify.price).to.be.a('string');
           expect(notify.address).to.be.a('string');
 
           /** check the values if possible **/

test/provider/neubauKompass.test.js

@@ -20,7 +20,7 @@ describe('#neubauKompass testsuite()', () => {
 
   it('should test neubauKompass provider', async () => {
     return await new Promise((resolve) => {
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'neubauKompass', similarityCache);
       fredy.execute().then((listing) => {
         expect(listing).to.be.a('array');
 

test/provider/testProvider.json

@@ -4,6 +4,10 @@
     "enabled": true,
     "id": "einsAImmobilien"
   },
+  "immobilienDe": {
+    "url": "https://www.immobilien.de/Wohnen/Suchergebnisse-51797.html?search._digest=true&search._filter=wohnen&search.flaeche_von=50&search.objektart=wohnung&search.preis_bis=1200&search.typ=mieten&search.umkreis=15&search.wo=district%3A2434%2C2695%2C2621%2C2700%2C2967%2C2734%2C2909%2C2955%2C2392%2C2746%2C2767%2C2982%2C2904%2C2612%2C2892%2C2587%2C2871%2C2975%2C2591%2C2887%2C2569%2C2640%2C2735&sort_col=*created_ts&sort_dir=desc",
+    "enabled": true
+  },
   "immonet": {
     "url": "https://www.immonet.de/immobiliensuche/sel.do?pageoffset=1&listsize=100&objecttype=1&locationname=Düsseldorf&acid=&actype=&district=8717&district=8718&district=8719&district=8720&district=8721&district=8723&district=8724&district=8725&district=8727&district=8728&district=8729&district=8730&district=8731&district=8732&district=8733&district=8737&district=8738&district=8741&district=8745&district=8747&district=8750&district=8752&district=8754&district=8755&district=8756&district=8759&district=8760&district=8761&district=8763&district=8764&district=8765&ajaxIsRadiusActive=false&sortby=19&suchart=1&radius=0&pcatmtypes=1_1&pCatMTypeStoragefield=&parentcat=1&marketingtype=1&fromprice=&toprice=420000&fromarea=90&toarea=&fromplotarea=&toplotarea=&fromrooms=3&torooms=&objectcat=225&objectcat=18&objectcat=17&objectcat=12&objectcat=16&objectcat=181&objectcat=14&objectcat=15&objectcat=226&objectcat=13&wbs=-1&fromyear=&toyear=",
     "enabled": true
@@ -36,4 +40,4 @@
     "url": "https://www.wg-gesucht.de/wg-zimmer-in-Duesseldorf.30.0.1.0.html?offer_filter=1&noDeact=1&city_id=30&category=0&rent_type=0&rMax=5000",
     "enabled": true
   }
-}
+}

test/provider/wgGesucht.test.js

@@ -20,7 +20,7 @@ describe('#wgGesucht testsuite()', () => {
 
   it('should test wgGesucht provider', async () => {
     return await new Promise((resolve) => {
-      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'test1', similarityCache);
+      const fredy = new Fredy(provider.config, null, provider.metaInformation.id, 'wgGesucht', similarityCache);
       fredy.execute().then((listing) => {
         expect(listing).to.be.a('array');
         const notificationObj = mockNotification.get();

ui/src/App.jsx

@@ -7,7 +7,7 @@ import ToastsContainer from './components/toasts/ToastContainer';
 import JobMutation from './views/jobs/mutation/JobMutation';
 import UserMutator from './views/user/mutation/UserMutator';
 import ToastContext from './components/toasts/ToastContext';
-import JobInsight from './views/jobs/insights/JobInsight';
+import JobInsight from './views/jobs/insights/JobInsight.jsx';
 import { useDispatch, useSelector } from 'react-redux';
 import useToast from './components/toasts/useToast';
 import { Switch, Redirect } from 'react-router-dom';
@@ -27,14 +27,17 @@ export default function FredyApp() {
   const [loading, setLoading] = React.useState(true);
   const currentUser = useSelector((state) => state.user.currentUser);
 
-  useEffect(async () => {
-    await dispatch.provider.getProvider();
-    await dispatch.jobs.getJobs();
-    await dispatch.jobs.getProcessingTimes();
-    await dispatch.notificationAdapter.getAdapter();
-    await dispatch.user.getCurrentUser();
+  useEffect(() => {
+    async function init() {
+      await dispatch.provider.getProvider();
+      await dispatch.jobs.getJobs();
+      await dispatch.jobs.getProcessingTimes();
+      await dispatch.notificationAdapter.getAdapter();
+      await dispatch.user.getCurrentUser();
 
-    setLoading(false);
+      setLoading(false);
+    }
+    init();
   }, [currentUser?.userId]);
 
   const needsLogin = () => {

ui/src/App.less

@@ -7,7 +7,7 @@
     width: 100%;
 
     padding: 1rem 1rem;
-    background-color: #3f3e3ef5;
+    background-color: #595959f5;
     color: #f1f1f1;
   }
 }

ui/src/Index.jsx

@@ -4,7 +4,9 @@ import { reduxStore } from './services/rematch/store';
 import { HashRouter } from 'react-router-dom';
 import { createHashHistory } from 'history';
 import { Provider } from 'react-redux';
-import ReactDOM from 'react-dom';
+import { createRoot } from 'react-dom/client';
+const container = document.getElementById('fredy');
+const root = createRoot(container);
 
 const history = createHashHistory();
 
@@ -12,11 +14,10 @@ import App from './App';
 
 import './Index.less';
 
-ReactDOM.render(
+root.render(
   <Provider store={reduxStore}>
     <HashRouter history={history}>
       <App />
     </HashRouter>
-  </Provider>,
-  document.getElementById('fredy')
+  </Provider>
 );

ui/src/Index.less

@@ -2,5 +2,5 @@ body, html {
   margin: 0;
   height: 100%;
   width: 100%;
-  background-color: #3f3e3ef5;
+  background-color: #595959f5;
 }

ui/src/views/generalSettings/GeneralSettings.jsx

@@ -2,7 +2,7 @@ import React from 'react';
 
 import { useDispatch, useSelector } from 'react-redux';
 
-import { Button, Form, Icon, Message, Segment } from 'semantic-ui-react';
+import { Button, Form, Icon, Message, Segment, Radio } from 'semantic-ui-react';
 import ToastContext from '../../components/toasts/ToastContext';
 import Headline from '../../components/headline/Headline';
 import { xhrPost } from '../../services/xhr';
@@ -18,21 +18,29 @@ const GeneralSettings = function Users() {
   const [interval, setInterval] = React.useState('');
   const [port, setPort] = React.useState('');
   const [scrapingAntApiKey, setScrapingAntApiKey] = React.useState('');
+  const [scrapingAntProxy, setScrapingAntProxy] = React.useState('');
   const [workingHourFrom, setWorkingHourFrom] = React.useState(null);
   const [workingHourTo, setWorkingHourTo] = React.useState(null);
   const ctx = React.useContext(ToastContext);
 
-  React.useEffect(async () => {
-    await dispatch.generalSettings.getGeneralSettings();
-    setLoading(false);
+  React.useEffect(() => {
+    async function init() {
+      await dispatch.generalSettings.getGeneralSettings();
+      setLoading(false);
+    }
+    init();
   }, []);
 
-  React.useEffect(async () => {
-    setInterval(settings?.interval);
-    setPort(settings?.port);
-    setScrapingAntApiKey(settings?.scrapingAnt?.apiKey);
-    setWorkingHourFrom(settings?.workingHours?.from);
-    setWorkingHourTo(settings?.workingHours?.to);
+  React.useEffect(() => {
+    async function init() {
+      setInterval(settings?.interval);
+      setPort(settings?.port);
+      setScrapingAntApiKey(settings?.scrapingAnt?.apiKey);
+      setWorkingHourFrom(settings?.workingHours?.from);
+      setWorkingHourTo(settings?.workingHours?.to);
+      setScrapingAntProxy(settings?.scrapingAnt?.proxy || 'datacenter');
+    }
+    init();
   }, [settings]);
 
   const nullOrEmpty = (val) => val == null || val.length === 0;
@@ -69,6 +77,7 @@ const GeneralSettings = function Users() {
         port,
         scrapingAnt: {
           apiKey: scrapingAntApiKey,
+          proxy: scrapingAntProxy,
         },
         workingHours: {
           from: workingHourFrom,
@@ -144,6 +153,48 @@ const GeneralSettings = function Users() {
               />
             </SegmentPart>
 
+            <SegmentPart
+              name="ScrapingAnt proxy settings"
+              helpText="Scraping ant provides different proxies."
+              icon="key"
+            >
+              <Message info>
+                ScrapingAnt is needed to scrape Immoscout. ScrapingAnt itself is using 2 different types of proxies.{' '}
+                <br />
+                <h4>Datacenter-Proxy</h4>
+                Proxy server located in one of the datacenters across the world. Datacenter proxies are slower and more
+                likely to fail, but they are cheaper. A call with a datacenter proxy cost 10 credits.
+                <h4>Residential-Proxy</h4>
+                High-quality proxy server located in one of the real people houses across the world. Datacenter proxies
+                are faster and more likely to success, but they are more expensive. A call with a datacenter proxy cost
+                250 credits.
+                <br />
+                <br />
+                <b>
+                  On the free tier, you have 10.000 credits, so chose your option wisely. Keep in mind, only successful
+                  calls will be charged.
+                </b>
+              </Message>
+              <Form.Field>
+                <Radio
+                  label="Datacenter proxy"
+                  name="scrapingAntProxy"
+                  value="datacenter"
+                  checked={scrapingAntProxy === 'datacenter'}
+                  onChange={(e, { value }) => setScrapingAntProxy(value)}
+                />
+              </Form.Field>
+              <Form.Field>
+                <Radio
+                  label="Residential proxy"
+                  name="scrapingAntProxy"
+                  value="residential"
+                  checked={scrapingAntProxy === 'residential'}
+                  onChange={(e, { value }) => setScrapingAntProxy(value)}
+                />
+              </Form.Field>
+            </SegmentPart>
+
             <SegmentPart
               name="Working hours"
               helpText="During this hours, Fredy will search for new apartments. If nothing is configured, Fredy will search around the clock."
@@ -153,7 +204,7 @@ const GeneralSettings = function Users() {
                 <Form.Input
                   className="generalSettings__time"
                   type="time"
-                  placeholder="ScrapingAnt Api Key"
+                  placeholder="Working hours from"
                   inverted
                   size="mini"
                   width={2}
@@ -163,7 +214,7 @@ const GeneralSettings = function Users() {
                 <div className="generalSettings__until">until</div>
                 <Form.Input
                   type="time"
-                  placeholder="ScrapingAnt Api Key"
+                  placeholder="Working hours to"
                   inverted
                   size="mini"
                   width={2}

ui/src/views/generalSettings/GeneralSettings.less

@@ -15,7 +15,7 @@
   }
 
   &__message{
-    background: #60c5df!important;
+    background: #8fe8ff!important;
   }
 
 }

ui/src/views/jobs/ProcessingTimes.jsx

@@ -43,7 +43,7 @@ export default function ProcessingTimes({ processingTimes }) {
             ScrapingAnt
           </a>
           . You can use the code <b>FREDY10</b> to get 10% off. (No affiliation, we are <b>not</b> getting paid to
-          recommend ScrapingAnt.
+          recommend ScrapingAnt.)
         </Segment>
       )}
     </React.Fragment>

ui/src/views/jobs/mutation/JobMutation.jsx

@@ -81,7 +81,6 @@ export default function JobMutator() {
       <ProviderMutator
         visible={providerCreationVisible}
         onVisibilityChanged={(visible) => setProviderCreationVisibility(visible)}
-        selected={providerData}
         onData={(data) => {
           setProviderData([...providerData, data]);
         }}

ui/src/views/jobs/mutation/components/provider/ProviderMutator.jsx

@@ -16,7 +16,7 @@ const sortProvider = (a, b) => {
   return 0;
 };
 
-export default function ProviderMutator({ onVisibilityChanged, visible = false, selected = [], onData } = {}) {
+export default function ProviderMutator({ onVisibilityChanged, visible = false, onData } = {}) {
   const provider = useSelector((state) => state.provider);
   const [selectedProvider, setSelectedProvider] = useState(null);
   const [providerUrl, setProviderUrl] = useState(null);
@@ -107,8 +107,6 @@ export default function ProviderMutator({ onVisibilityChanged, visible = false,
                   text: pro.name,
                 };
               })
-              //filter out those, that have already been selected
-              .filter((option) => selected.find((selectedOption) => selectedOption.id === option.key) == null)
               .sort(sortProvider)}
             onChange={(e, { value }) => {
               const selectedProvider = provider.find((pro) => pro.id === value);

ui/src/views/user/Users.jsx

@@ -18,9 +18,12 @@ const Users = function Users() {
   const [userIdToBeRemoved, setUserIdToBeRemoved] = React.useState(null);
   const history = useHistory();
 
-  React.useEffect(async () => {
-    await dispatch.user.getUsers();
-    setLoading(false);
+  React.useEffect(() => {
+    async function init() {
+      await dispatch.user.getUsers();
+      setLoading(false);
+    }
+    init();
   }, []);
 
   const onUserRemoval = async () => {

ui/src/views/user/mutation/UserMutator.jsx

@@ -21,21 +21,24 @@ const UserMutator = function UserMutator() {
   const ctx = React.useContext(ToastContext);
   const dispatch = useDispatch();
 
-  React.useEffect(async () => {
-    if (params.userId != null) {
-      try {
-        const userJson = await xhrGet(`/api/admin/users/${params.userId}`);
-        const user = userJson.json;
+  React.useEffect(() => {
+    async function init() {
+      if (params.userId != null) {
+        try {
+          const userJson = await xhrGet(`/api/admin/users/${params.userId}`);
+          const user = userJson.json;
 
-        const defaultName = user?.username || '';
-        const defaultIsAdmin = user?.isAdmin || false;
+          const defaultName = user?.username || '';
+          const defaultIsAdmin = user?.isAdmin || false;
 
-        setUsername(defaultName);
-        setIsAdmin(defaultIsAdmin);
-      } catch (Exception) {
-        console.error(Exception);
+          setUsername(defaultName);
+          setIsAdmin(defaultIsAdmin);
+        } catch (Exception) {
+          console.error(Exception);
+        }
       }
     }
+    init();
   }, [params.userId]);
 
   const saveUser = async () => {

vite.config.js

@@ -0,0 +1,23 @@
+import react from '@vitejs/plugin-react';
+import { defineConfig } from 'vite';
+
+// https://vitejs.dev/config/
+export default defineConfig({
+  base: '',
+  build: {
+    chunkSizeWarningLimit: 9999999,
+    outDir: './ui/public',
+  },
+  plugins: [react()],
+  server: {
+    proxy: {
+      '/api': {
+        target: {
+          host: '0.0.0.0',
+          protocol: 'http:',
+          port: 9998,
+        },
+      },
+    },
+  },
+});

webpack.common.js

@@ -1,39 +0,0 @@
-const { CleanWebpackPlugin } = require('clean-webpack-plugin');
-const CopyPlugin = require('copy-webpack-plugin');
-const path = require('path');
-
-module.exports = {
-  entry: path.join(__dirname, 'ui', 'src') + '/Index.js',
-  resolve: {
-    extensions: ['.js'],
-  },
-  plugins: [
-    new CleanWebpackPlugin(),
-    new CopyPlugin({
-      patterns: [{ from: path.join(__dirname, 'ui', 'src') + '/index.html', to: path.join(__dirname, 'ui', 'public') }],
-    }),
-  ],
-  output: {
-    path: path.join(__dirname, 'ui', 'public'),
-    publicPath: '/',
-    filename: 'fredy.bundle.js',
-  },
-  performance: { hints: false },
-  module: {
-    rules: [
-      {
-        test: /\.js?$/,
-        exclude: /(node_modules|bower_components)/,
-        use: [{ loader: 'babel-loader' }],
-      },
-      {
-        test: /\.(css|less)$/i,
-        use: ['style-loader', 'css-loader', 'less-loader'],
-      },
-      {
-        test: /\.(png|jpg|gif|svg|eot|ttf|woff|woff2)$/,
-        use: [{ loader: 'url-loader?limit=3000!image-webpack?bypassOnDebug&optimizationLevel=7&interlaced=false' }],
-      },
-    ],
-  },
-};

webpack.dev.js

@@ -1,26 +0,0 @@
-const { merge } = require('webpack-merge');
-const common = require('./webpack.common.js');
-const webpack = require('webpack');
-const path = require('path');
-
-module.exports = merge(common, {
-  devtool: 'inline-source-map',
-  devServer: {
-    contentBase: path.join(__dirname, 'ui', 'public'),
-    port: 9000,
-    proxy: {
-      '/api': {
-        target: {
-          host: '0.0.0.0',
-          protocol: 'http:',
-          port: 9998,
-        },
-      },
-    },
-  },
-  plugins: [
-    new webpack.DefinePlugin({
-      __DEV__: JSON.stringify(true),
-    }),
-  ],
-});

webpack.prod.js

@@ -1,4 +0,0 @@
-const { merge } = require('webpack-merge');
-const common = require('./webpack.common.js');
-
-module.exports = merge(common, {});