Release v1.0.0 🎉

.gitignore

@@ -0,0 +1,5 @@
+node_modules/
+npm-debug.log
+config.json
+store.json
+.DS_Store

.travis.yml

@@ -0,0 +1,4 @@
+sudo: false
+language: node_js
+node_js:
+  - "8"

CONTRIBUTION.md

@@ -0,0 +1,28 @@
+# Contributing
+
+If you want to contribute, please make sure you've executed the tests.
+
+
+### How to write new provider?
+- create the provider filer under `/lib/provider`
+- create the corresponding config both in `config/config.example` and `config/config.test.json` 
+- make sure the selector matches and that the needed fields are available
+- create a test under /test and make sure it is running successfully 
+
+### How to write new notification adapter?
+- create the provider filer under `/lib/notification/adapter`
+- make sure it exports a function `enabled` and a function `send`
+- create the corresponding config both in `config/config.example` and `config/config.test.json` 
+
+#### Running Tests
+If you've written a new provider you are an awesome person. You know it and I do. If you now write tests for it, you are even more awesome. And who doesn't want to be more awesome right?
+
+To write tests for provider, you need to use Node 8 as the tests are using `async / await`
+
+##### To do before merging:
+
+- executed tests? (`npm run test`)
+- executed reformat? (`npm run format`)
+- sure the changes are useful for everybody? Or is it maybe a custom modification just for your case?
+
+_Thanks!_ :heart:

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 Christian Kellner
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,103 @@
+# Fredy
+
+[F]ind [R]eal [E]states [D]amn Eas[y] :heart:
+
+My wife and I wanted to buy an apartment in germany. As the prices are quite high and good deals are very rare, we searched the "big 4" every morning.   
+
+This however can get pretty frustrating. _Fredy_ will take this work away from you. It crawls the `provider`, mentioned below (Immonet, Immoscout...) every _x_ minutes. (The provider list can be extended easily...)   
+
+If _Fredy_ found matching results, it will send them to via Slack. (More adapter possible.) _Fredy_ is remembering what it already has found to not send stuff twice.
+
+## Usage
+
+- Make sure to use Node 8 and above
+- Install the dependencies using `npm install`
+- create your configuration file. Use the example config for this. `cp conf/config.example conf/config.json`
+- configure the desired values
+- start _Fredy_ using `npm start`
+
+
+## Configuration
+
+Before running _Fredy_ for the first time, you need to create a configuration file:
+
+Copy the example config as a start.
+```
+cp conf/config.example conf/config.json
+```
+
+### 1. Notification
+
+You want to get notified when _Fredy_ found a new listing. Currently _Fredy_ support Slack to send notification. _Fredy_ also includes a notification adapter to print to the console instead of sending to a services.
+
+Adding new notification adapter is easy however. See [Contribution](https://github.com/orangecoding/fredy/blob/master/CONTRIBUTION.md) 
+```json
+"slack": {
+  "channel": "someChannel",
+  "token": "someToken",
+  "enabled": true
+}
+```
+
+### 2. Configure the providers
+
+Configure the providers like described below. To disable a provider just remove its entry from the configuration or set it to `false`.
+
+#### Ebay Kleinanzeigen, Immoscout, Immowelt, Immonet and Kalaydo
+
+```json
+"kleinanzeigen": {
+    "url": "https://www.ebay-kleinanzeigen.de/...",
+    "enabled": true
+}
+"immoscout": {
+  "url": "https://www.immobilienscout24.de/...",
+  "enabled": true
+},
+"immowelt": {
+  "url": "https://www.immowelt.de/...",
+  "enabled": true
+},
+"immonet": {
+  "url": "http://www.immonet.de/...",
+  "enabled": true
+},
+"kalaydo": {
+  "url": "http://www.kalaydo.de/...",
+  "enabled": true
+},
+```
+
+Go to the respective provider page and create your custom search queries by
+using the provided filter options. Then just copy and paste the whole URL of
+the resulting listings page.
+
+**IMPORTANT:** Make sure to always sort by newest listings! This way _Fredy_ makes sure to not accidentally report stuff twice.
+
+#### Custom provider
+
+See [Contribution](https://github.com/orangecoding/fredy/blob/master/CONTRIBUTION.md)
+
+### 3. Add Filters (optional)
+
+
+#### Blacklist
+
+```json
+"blacklist": [
+  "vermietet"
+]
+```
+
+Listings which contain at least on of the given terms (ignoring case, only
+whole words) are removed. 
+
+#### District blacklist
+```json
+"blacklistedDistrics": [
+  "Altstadt"
+]
+```
+Districts that are unwanted can be blacklisted here. 
+
+This makes sense for provider that only offer limited filter functions like Kalaydo/Ebay.

conf/config.example

@@ -0,0 +1,50 @@
+{
+  "interval": 10,
+  "enableStats": false,
+  "statsPort": 9875,
+  "notification": {
+    "slack": {
+      "token": "someToken",
+      "channel": "someChannel",
+      "enabled": false
+    },
+    "console": {
+      "enabled": true
+    }
+  },
+  "sources": {
+    "immoscout": {
+      "url": "https://www.immobilienscout24.de/...",
+      "enabled": true
+    },
+    "immonet": {
+      "url": "https://www.immonet.de/...",
+      "enabled": true
+    },
+    "immowelt": {
+      "url": "https://www.immowelt.de/...",
+      "enabled": true
+    },
+    "kleinanzeigen": {
+      "url": "https://www.ebay-kleinanzeigen.de/...",
+      "enabled": true
+    },
+    "kalaydo": {
+      "url": "https://www.kalaydo.de/...",
+      "enabled": true
+    }
+  },
+  "blacklist": [
+    "swap",
+    "tausch",
+    "wg",
+    "Vermietete",
+    "Vermietet",
+    "vermietete",
+    "vermietet"
+  ],
+  "blacklistedDistrics": [
+    "some District you don't want"
+  ],
+  "userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36"
+}

conf/config.test.json

@@ -0,0 +1,76 @@
+{
+  "interval": 10,
+  "enableStats": false,
+  "statsPort": 9875,
+  "notification": {
+    "slack": {
+      "token": "",
+      "channel": "test",
+      "enabled": false
+    },
+    "console": {
+      "enabled": true
+    }
+  },
+  "sources": {
+    "immoscout": {
+      "url": "https://www.immobilienscout24.de/Suche/S-2/Wohnung-Kauf/Nordrhein-Westfalen/Duesseldorf/-/-/-/-/false/-/3,6,7,8,40,113,117,118,127?enteredFrom=result_list",
+      "enabled": true
+    },
+    "immonet": {
+      "url": "https://www.immonet.de/immobiliensuche/sel.do?pageoffset=1&listsize=25&objecttype=1&locationname=D%C3%BCsseldorf&acid=&actype=&city=100207&ajaxIsRadiusActive=false&sortby=19&suchart=2&radius=0&pcatmtypes=1_1&pCatMTypeStoragefield=1_1&parentcat=1&marketingtype=1&fromprice=&toprice=&fromarea=&toarea=&fromplotarea=&toplotarea=&fromrooms=&torooms=&objectcat=225&objectcat=18&objectcat=17&objectcat=12&objectcat=16&objectcat=181&objectcat=14&objectcat=15&objectcat=226&objectcat=13&wbs=-1&fromyear=&toyear=&fulltext=&absenden=Ergebnisse+anzeigen",
+      "enabled": true
+    },
+    "immowelt": {
+      "url": "https://www.immowelt.de/liste/duesseldorf/wohnungen/kaufen?lat=51.22061&lon=6.80575&sort=createdate%2Bdesc",
+      "enabled": true
+    },
+    "kleinanzeigen": {
+      "url": "https://www.ebay-kleinanzeigen.de/s-wohnung-kaufen/duesseldorf/anzeige:angebote/wohnung/k0c196l2068r5",
+      "enabled": true
+    },
+    "kalaydo": {
+      "url": "https://www.kalaydo.de/immobilien/eigentumswohnung-kaufen/o/duesseldorf/4/",
+      "enabled": true
+    }
+  },
+  "blacklist": [
+    "swap",
+    "tausch",
+    "wg",
+    "Vermietete",
+    "Vermietet",
+    "vermietete",
+    "vermietet"
+  ],
+  "blacklistedDistrics": [
+    "Altstadt",
+    "Angermund",
+    "Carlstadt",
+    "Flehe",
+    "Friedrichstadt",
+    "Garath",
+    "Hafen",
+    "Hamm",
+    "Hassels",
+    "Heerdt",
+    "Hellerhof",
+    "Himmelgeist",
+    "Hubbelrath",
+    "Itter",
+    "Kaiserswerth",
+    "Kalkum",
+    "Lichtenbroich",
+    "Lohausen",
+    "Ludenberg",
+    "Niederkassel",
+    "Oberkassel",
+    "Stadtmittel",
+    "Stockum",
+    "Urdenbach",
+    "Vennhausen",
+    "Volmerswerth",
+    "Wittlaer"
+  ],
+  "userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36"
+}

index.js

@@ -0,0 +1,14 @@
+require('rootpath')();
+const fs = require('fs');
+const path = './lib/provider';
+const sources = fs.readdirSync(path);
+const config = require('conf/config.json');
+const stats = require('./lib/services/stats');
+
+setInterval(
+  (function exec() {
+    sources.forEach(source => require(`${path}/${source}`).run(stats));
+    return exec;
+  })(),
+  config.interval * 60 * 1000
+);

lib/errors.js

@@ -0,0 +1,16 @@
+class ExtendableError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = this.constructor.name;
+        if (typeof Error.captureStackTrace === 'function') {
+            Error.captureStackTrace(this, this.constructor)
+        } else {
+            this.stack = (new Error(message)).stack
+        }
+    }
+}
+
+class NoNewListingsError extends ExtendableError {
+}
+
+module.exports = {NoNewListingsError};

lib/fredy.js

@@ -0,0 +1,108 @@
+const {NoNewListingsError} = require('./errors');
+const Store = require('./services/store');
+
+const notify = require('./notification/notify');
+const xray = require('./services/scraper');
+
+class Fredy {
+    constructor(source) {
+        this._store = new Store(source.name);
+        this._fullCrawl = true;
+        this._source = source;
+        this._stats = null;
+    }
+
+    run(stats) {
+
+        if(!this._stats){
+            this._stats = stats;
+        }
+
+        if (!this._source.enabled) return Promise.resolve();
+
+        return Promise.resolve(this._source.url)
+            .then(this._store.warmup)
+            .then(this._getListings.bind(this))
+            .then(this._normalize.bind(this))
+            .then(this._filter.bind(this))
+            .then(this._findNew.bind(this))
+            .then(this._save.bind(this))
+            .then(this._notify.bind(this))
+            .then(this._updateStates.bind(this))
+            .catch(this._handleError.bind(this))
+    }
+
+    _getListings(url) {
+        return new Promise((resolve, reject) => {
+            let x = xray(url, this._source.crawlContainer, [this._source.crawlFields]);
+
+            if (this._source.paginage && this._fullCrawl) {
+                this._fullCrawl = false;
+                x = x.paginate(this._source.paginage)
+            }
+
+            x((err, listings) => {
+                if (err) reject(err);
+                else resolve(listings)
+            })
+        })
+    }
+
+    _normalize(listings) {
+        return listings.map(this._source.normalize)
+    }
+
+    _filter(listings) {
+        return listings.filter(this._source.filter)
+    }
+
+    _findNew(listings) {
+        const newListings = listings.filter(
+            o => this._store.knownListings.indexOf(o.id) === -1
+        );
+
+        if (newListings.length === 0) {
+            this._updateStates([]);
+            throw new NoNewListingsError();
+        }
+
+        return newListings
+    }
+
+    _notify(newListings) {
+        const sendNotifications = newListings.map(payload => {
+            return notify.send(this._source.name, payload);
+        }
+        );
+
+        return Promise.all(sendNotifications).then(() => newListings)
+    }
+
+    _updateStates(newListings){
+        this._stats.setLastScrape(this._source.name, newListings.length);
+        return newListings;
+    }
+
+    _save(newListings) {
+        this._store.knownListings = [
+            ...this._store.knownListings,
+            ...newListings.map(l => l.id)
+        ];
+        return newListings;
+    }
+
+    _handleError(err) {
+        if (err.name !== 'NoNewListingsError') console.error(err)
+    }
+
+    /**
+     * for testing purposes only
+     * @returns {Store}
+     * @private
+     */
+    _getStore(){
+        return this._store;
+    }
+}
+
+module.exports = Fredy;

lib/notification/adapter/console.js

@@ -0,0 +1,18 @@
+const config = require('../../../conf/config.json');
+
+/**
+ * simply prints out the found data to the console
+ * @param serviceName e.g immoscout
+ * @param payload the actual payload that is used to construct the message
+ * @returns {Promise<Void> | void}
+ */
+exports.send = (serviceName, payload) => {
+    return Promise.resolve(console.info(`Found entry from service ${serviceName}:`, payload))
+};
+
+/**
+ * each integration needs to implement this method
+ */
+exports.enabled = () => {
+  return config.notification.console.enabled;
+};

lib/notification/adapter/slack.js

@@ -0,0 +1,54 @@
+const Slack = require('slack');
+const config = require('../../conf/config.json');
+const msg = Slack.chat.postMessage;
+
+const {token, channel} = config.notification.slack;
+
+/**
+ * sends a new listing to slack
+ * @param serviceName e.g immoscout
+ * @param payload the actual payload that is used to construct the message
+ * @returns {Promise<Chat.PostMessage.Response> | void}
+ */
+exports.send = (serviceName, payload) => {
+    return msg({
+            token,
+            channel,
+            text: `*(${serviceName})* - ${payload.title}`,
+            "attachments": [
+                {
+                    "fallback": payload.title,
+                    "color": "#36a64f",
+                    "title": "Link to Exposé",
+                    "title_link": payload.link,
+                    "fields": [
+                        {
+                            "title": "Price",
+                            "value": payload.price,
+                            "short": false
+                        },
+                        {
+                            "title": "Size",
+                            "value": payload.size,
+                            "short": false
+                        },
+                        {
+                            "title": "Address",
+                            "value": payload.address,
+                            "short": false
+                        }
+                    ],
+                    "footer": "Powered by Fredy",
+                    ts: new Date().getTime() / 1000
+                }
+            ]
+        }
+    );
+};
+
+/**
+ * each integration needs to implement this method
+ */
+exports.enabled = () => {
+  return config.notification.slack.enabled;
+};

lib/notification/notify.js

@@ -0,0 +1,17 @@
+const fs = require('fs');
+const path = './adapter';
+
+/** Read every integration existing in ./adapter **/
+const adapter = fs
+    .readdirSync('./lib/notification/adapter')
+    .map(integPath => require(`${path}/${integPath}`))
+    .filter(integration => integration.enabled());
+
+if (adapter.length === 0) {
+    throw new Error('Please specify at least one notification provider');
+}
+
+exports.send = (serviceName, payload) => {
+    //this is not being used in tests, therefor adapter are always set
+    return adapter.map(a => a.send(serviceName, payload));
+};

lib/provider/immonet.js

@@ -0,0 +1,38 @@
+const config = require('../../conf/config.json');
+const Fredy = require('../fredy');
+const utils = require('../utils');
+
+function normalize(o) {
+  const id = parseInt(o.id.split('_')[1], 10);
+  const title = o.title.replace('NEU ', '');
+  const address = o.address.split(' - ')[1];
+
+  return Object.assign(o, { id, title, address });
+}
+
+function applyBlacklist(o) {
+  const titleNotBlacklisted = !utils.isOneOf(o.title, config.blacklist);
+  const descNotBlacklisted = !utils.isOneOf(o.description, config.blacklist);
+
+  return titleNotBlacklisted && descNotBlacklisted;
+}
+
+const immonet = {
+  name: 'immonet',
+  enabled: config.sources.immonet.enabled,
+  url: config.sources.immonet.url,
+  crawlContainer: '#idResultList .search-object',
+  crawlFields: {
+    id: '.search-info a@id',
+    price: '#keyfacts-bar div:first-child span',
+    size: '#keyfacts-bar div:nth-child(2) .text-primary-highlight',
+    title: '.search-info a | removeNewline | trim',
+    link: '.search-info a@href',
+    address: '.search-info p | removeNewline | trim'
+  },
+  paginate: '#idResultList .margin-bottom-6.margin-bottom-sm-12 .panel a.pull-right@href',
+  normalize: normalize,
+  filter: applyBlacklist
+};
+
+module.exports = new Fredy(immonet);

lib/provider/immoscout.js

@@ -0,0 +1,34 @@
+const config = require('../../conf/config.json');
+const Fredy = require('../fredy');
+const utils = require('../utils');
+
+function normalize(o) {
+  const title = o.title.replace('NEU', '');
+  const address = (o.address || '').replace(/\(.*\),.*$/, '').trim();
+
+  return Object.assign(o, { title, address });
+}
+
+function applyBlacklist(o) {
+  return !utils.isOneOf(o.title, config.blacklist);
+}
+
+const immoscout = {
+  name: 'immoscout',
+  enabled: config.sources.immoscout.enabled,
+  url: config.sources.immoscout.url,
+  crawlContainer: '#resultListItems li.result-list__listing',
+  crawlFields: {
+    id: '.result-list-entry@data-obid | int',
+    price: '.result-list-entry .result-list-entry__criteria .grid-item:first-child dd | removeNewline | trim',
+    size: '.result-list-entry .result-list-entry__criteria .grid-item:nth-child(2) dd | removeNewline | trim',
+    title: '.result-list-entry .result-list-entry__brand-title-container h5 | removeNewline | trim',
+    link: '.result-list-entry .result-list-entry__brand-title-container@href',
+    address: '.result-list-entry .result-list-entry__address a'
+  },
+  paginate: '#pager .align-right a@href',
+  normalize: normalize,
+  filter: applyBlacklist
+};
+
+module.exports = new Fredy(immoscout);

lib/provider/immowelt.js

@@ -0,0 +1,38 @@
+const Fredy = require('../fredy');
+const config = require('../../conf/config.json');
+const utils = require('../utils');
+
+function normalize(o) {
+  const size = o.size.split('Wohnfläche')[0];
+  const address = o.address;
+
+  return Object.assign(o, { size, address });
+}
+
+function applyBlacklist(o) {
+  const titleNotBlacklisted = !utils.isOneOf(o.title, config.blacklist);
+  const descNotBlacklisted = !utils.isOneOf(o.description, config.blacklist);
+
+  return titleNotBlacklisted && descNotBlacklisted;
+}
+
+const immowelt = {
+  name: 'immowelt',
+  enabled: config.sources.immowelt.enabled,
+  url: config.sources.immowelt.url,
+  crawlContainer: '.immoliste .js-object.listitem_wrap ',
+  crawlFields: {
+    id: '@data-estateid | int',
+    price: '.hardfacts_3 strong | removeNewline | trim',
+    size: '.hardfacts_3 div:nth-child(2):not(.hardfactlabel)| removeNewline | trim',
+    title: '.listcontent.clear h2',
+    link: 'a@href',
+    description: '.listconten_offset .listmerkmale| removeNewline | trim',
+    address: '.listconten_offset .listlocation| removeNewline | trim'
+  },
+  paginate: '#pnlPaging #nlbPlus@href',
+  normalize: normalize,
+  filter: applyBlacklist
+};
+
+module.exports = new Fredy(immowelt);

lib/provider/kalaydo.js

@@ -0,0 +1,46 @@
+const config = require('../../conf/config.json');
+const Fredy = require('../fredy');
+const utils = require('../utils');
+
+function normalize(o) {
+  const id = o.id
+    .split('/')
+    .filter(Boolean)
+    .reverse()[0];
+  const price = o.price.replace('Preis: ', '');
+  let size = o.size.replace('Wohnfläche: ', '').replace('ca. ', '');
+  size += ' / ' + o.rooms;
+  const address = '---';
+
+  return Object.assign(o, { id, price, size, address });
+}
+
+function applyBlacklist(o) {
+  const titleNotBlacklisted = !utils.isOneOf(o.title, config.blacklist);
+  const descNotBlacklisted = !utils.isOneOf(o.description, config.blacklist);
+
+  const isBlacklistedDistrict =
+    config.blacklistedDistrics.length === 0 ? false : utils.isOneOf(o.title, config.blacklistedDistrics);
+
+  return !isBlacklistedDistrict && titleNotBlacklisted && descNotBlacklisted;
+}
+
+const kalaydo = {
+  name: 'kalaydo',
+  enabled: config.sources.kalaydo.enabled,
+  url: config.sources.kalaydo.url,
+  crawlContainer: '#resultList .resultitem-content-container',
+  crawlFields: {
+    id: '.resultitem-content-container a@href',
+    price: '.clear-row .rent | removeNewline | trim',
+    title: '.resultitem-content-container a@title',
+    link: '.resultitem-content-container a@href',
+    rooms: '.resultitem-content-container .no-of-rooms | removeNewline  | trim',
+    size: '.resultitem-content-container .living-area | removeNewline | trim'
+  },
+  paginate: '.markt_pagination_pageLinkNext .markt_pagination_link@href',
+  normalize: normalize,
+  filter: applyBlacklist
+};
+
+module.exports = new Fredy(kalaydo);

lib/provider/kleinanzeigen.js

@@ -0,0 +1,39 @@
+const Fredy = require('../fredy');
+const config = require('../../conf/config.json');
+const utils = require('../utils');
+
+function normalize(o) {
+  const address = o.address.split('\n')[4].trim();
+
+  return Object.assign(o, { address });
+}
+
+function applyBlacklist(o) {
+  const titleNotBlacklisted = !utils.isOneOf(o.title, config.blacklist);
+  const descNotBlacklisted = !utils.isOneOf(o.description, config.blacklist);
+  const isBlacklistedDistrict =
+    config.blacklistedDistrics.length === 0 ? false : utils.isOneOf(o.description, config.blacklistedDistrics);
+
+  return !isBlacklistedDistrict && titleNotBlacklisted && descNotBlacklisted;
+}
+
+const kleinanzeigen = {
+  name: 'kleinanzeigen',
+  enabled: config.sources.kleinanzeigen.enabled,
+  url: config.sources.kleinanzeigen.url,
+  crawlContainer: '#srchrslt-adtable .ad-listitem',
+  crawlFields: {
+    id: '.aditem@data-adid | int',
+    price: '.aditem-details strong | removeNewline | trim',
+    size: '.aditem-main .text-module-end span:nth-child(2) | removeNewline | trim',
+    title: '.aditem-main .text-module-begin a | removeNewline | trim',
+    link: '.aditem-main .text-module-begin a@href | removeNewline | trim',
+    description: '.aditem-main p:not(.text-module-end) | removeNewline | trim',
+    address: '.aditem-details'
+  },
+  paginate: '#srchrslt-pagination .pagination-next@href',
+  normalize: normalize,
+  filter: applyBlacklist
+};
+
+module.exports = new Fredy(kleinanzeigen);

lib/services/scraper.js

@@ -0,0 +1,42 @@
+const makeDriver = require('request-x-ray');
+const config = require('../../conf/config.json');
+const Xray = require('x-ray');
+
+class Scraper {
+  constructor() {
+    const filters = {
+      removeNewline: this._removeNewline,
+      trim: this._trim,
+      int: this._int
+    };
+
+    const driver = makeDriver({
+      headers: {
+        'User-Agent': config.userAgent
+      }
+    });
+
+    const xray = Xray({ filters });
+    xray.driver(driver);
+
+    this.xray = xray;
+  }
+
+  get x() {
+    return this.xray;
+  }
+
+  _removeNewline(value) {
+    return typeof value === 'string' ? value.replace(/\\n/g, '') : value;
+  }
+
+  _trim(value) {
+    return typeof value === 'string' ? value.replace(/\s+/g, ' ').trim() : value;
+  }
+
+  _int(value) {
+    return typeof value === 'string' ? parseInt(value, 10) : value;
+  }
+}
+
+module.exports = new Scraper().x;

lib/services/stats.js

@@ -0,0 +1,25 @@
+const config = require('../../conf/config.json');
+let lastScrape = {};
+
+if (config.enableStats) {
+  const http = require('http');
+  http
+    .createServer((req, res) => {
+      res.writeHead(200, { 'Content-Type': 'application/json' });
+      res.end(
+        JSON.stringify({
+          config,
+          lastScrape
+        })
+      );
+    })
+    .listen(config.statsPort, '127.0.0.1');
+}
+
+exports.setLastScrape = (serviceName, numberFound) => {
+  lastScrape[serviceName] = lastScrape[serviceName] || [];
+  lastScrape[serviceName].push({
+    scapeTime: new Date().toString(),
+    numberFound: numberFound
+  });
+};

lib/services/store.js

@@ -0,0 +1,36 @@
+const path = require('path');
+const DB_PATH = path.dirname(require.main.filename) + '/conf/store.json';
+
+const FileAsync = require('lowdb/adapters/FileAsync');
+const adapter = new FileAsync(DB_PATH);
+const low = require('lowdb');
+
+const lowdb = low(adapter);
+
+class Store {
+  constructor(name) {
+    this._name = name;
+    this._db = null;
+  }
+
+  get warmup() {
+    return new Promise(resolve => {
+      lowdb.then(db => {
+        this._db = db;
+        resolve();
+      });
+    });
+  }
+
+  set knownListings(value) {
+    if (!Array.isArray(value)) throw Error('Not a valid array');
+
+    return this._db.set(this._name, value).write();
+  }
+
+  get knownListings() {
+    return this._db.get(this._name).value() || [];
+  }
+}
+
+module.exports = Store;

lib/utils.js

@@ -0,0 +1,8 @@
+function isOneOf (word, arr) {
+  const expression = String.raw`\b(${arr.join('|')})\b`;
+  const blacklist = new RegExp(expression, 'ig');
+
+  return blacklist.test(word)
+}
+
+module.exports = { isOneOf };

package.json

@@ -0,0 +1,43 @@
+{
+  "name": "Fredy",
+  "version": "1.0.0",
+  "description": "[F]ind [R]eal [E]states [d]amn eas[y].",
+  "scripts": {
+    "start": "node index.js",
+    "format": "prettier --write lib/**/*.js test/**/*.js *.js --single-quote --print-width 120",
+    "test": "mocha --timeout 8000"
+  },
+  "main": "index.js",
+  "author": "Christian Kellner",
+  "keywords": [
+    "flat",
+    "flatfinder",
+    "fredy",
+    "real estates",
+    "germany",
+    "apartment",
+    "house",
+    "rent",
+    "immoscout",
+    "immonet",
+    "immowelt",
+    "immobilienscout24"
+  ],
+  "bugs": {
+    "url": "https://github.com/orangecoding/fredy/issues"
+  },
+  "license": "MIT",
+  "dependencies": {
+    "chai": "^4.1.2",
+    "lowdb": "^1.0.0",
+    "mocha": "^4.1.0",
+    "request-x-ray": "^0.1.4",
+    "rootpath": "^0.1.2",
+    "slack": "^10.0.0",
+    "x-ray": "^2.3.2"
+  },
+  "devDependencies": {
+    "prettier": "^1.0.0",
+    "proxyquire": "^1.8.0"
+  }
+}

test/immonet.test.js

@@ -0,0 +1,49 @@
+const mockNotification = require('./mocks/mockNotification');
+const mockConfig = require('../conf/config.test');
+const mockStore = require('./mocks/mockStore');
+const mockStats = require('./mocks/mockStats');
+const proxyquire = require('proxyquire').noCallThru();
+const expect = require('chai').expect;
+
+describe('#immonet testsuite()', () => {
+
+    const immonet = proxyquire('../lib/provider/immonet', {
+        '../../conf/config.json': mockConfig,
+        '../lib/fredy': proxyquire('../lib/fredy', {
+            './services/store': mockStore,
+            './notification/notify': mockNotification
+        })
+    });
+
+    it('should test immonet provider', async () => {
+        return await new Promise(resolve => {
+            immonet.run(mockStats).then(() => {
+                const immonetDbContent = immonet._getStore()._db;
+
+                expect(immonetDbContent.immonet).to.be.a('array');
+
+                const notificationObj = mockNotification.get();
+                expect(notificationObj).to.be.a('object');
+                expect(notificationObj.serviceName).to.equal('immonet');
+
+                /** check the actual structure **/
+                expect(notificationObj.payload.id).to.be.a('number');
+                expect(notificationObj.payload.price).to.be.a('string');
+                expect(notificationObj.payload.size).to.be.a('string');
+                expect(notificationObj.payload.title).to.be.a('string');
+                expect(notificationObj.payload.link).to.be.a('string');
+                expect(notificationObj.payload.address).to.be.a('string');
+
+                /** check the values if possible **/
+                expect(notificationObj.payload.id).to.equal(immonetDbContent.immonet[immonetDbContent.immonet.length - 1]);
+                expect(notificationObj.payload.price).that.does.include('€');
+                expect(notificationObj.payload.size).that.does.include('m²');
+                expect(notificationObj.payload.title).to.be.not.empty;
+                expect(notificationObj.payload.link).that.does.include('https://www.immonet.de');
+                expect(notificationObj.payload.address).to.be.not.empty;
+
+                resolve();
+            });
+        });
+    });
+});

test/immoscout.test.js

@@ -0,0 +1,50 @@
+const mockNotification = require('./mocks/mockNotification');
+const mockConfig = require('../conf/config.test');
+const mockStats = require('./mocks/mockStats');
+const mockStore = require('./mocks/mockStore');
+const proxyquire = require('proxyquire').noCallThru();
+const expect = require('chai').expect;
+
+describe('#immoscout testsuite()', () => {
+
+    const immoscout = proxyquire('../lib/provider/immoscout', {
+        '../../conf/config.json': mockConfig,
+        '../lib/fredy': proxyquire('../lib/fredy', {
+            './services/store': mockStore,
+            './notification/notify': mockNotification
+        })
+    });
+
+    it('should test immoscout provider', async () => {
+        return await new Promise(resolve => {
+            immoscout.run(mockStats).then(() => {
+                const immoscoutDbContent = immoscout._getStore()._db;
+                expect(immoscoutDbContent.immoscout).to.be.a('array');
+
+                const notificationObj = mockNotification.get();
+                expect(notificationObj).to.be.a('object');
+                expect(notificationObj.serviceName).to.equal('immoscout');
+
+                /** check the actual structure **/
+                expect(notificationObj.payload.id).to.be.a('number');
+                expect(notificationObj.payload.price).to.be.a('string');
+                expect(notificationObj.payload.size).to.be.a('string');
+                expect(notificationObj.payload.title).to.be.a('string');
+                expect(notificationObj.payload.link).to.be.a('string');
+                expect(notificationObj.payload.address).to.be.a('string');
+
+                /** check the values if possible **/
+                expect(notificationObj.payload.id).to.equal(
+                    immoscoutDbContent.immoscout[immoscoutDbContent.immoscout.length - 1]
+                );
+                expect(notificationObj.payload.price).that.does.include('€');
+                expect(notificationObj.payload.size).that.does.include('m²');
+                expect(notificationObj.payload.title).to.be.not.empty;
+                expect(notificationObj.payload.link).that.does.include('https://www.immobilienscout24.de');
+                expect(notificationObj.payload.address).to.be.not.empty;
+
+                resolve();
+            }).catch(resolve);
+        });
+    });
+});

test/immowelt.test.js

@@ -0,0 +1,51 @@
+const mockNotification = require('./mocks/mockNotification');
+const mockConfig = require('../conf/config.test');
+const mockStats = require('./mocks/mockStats');
+const mockStore = require('./mocks/mockStore');
+const proxyquire = require('proxyquire').noCallThru();
+const expect = require('chai').expect;
+
+describe('#immowelt testsuite()', () => {
+
+    it('should test immowelt provider', async () => {
+
+        const immowelt = proxyquire('../lib/provider/immowelt', {
+            '../../conf/config.json': mockConfig,
+            '../lib/fredy': proxyquire('../lib/fredy', {
+                './services/store': mockStore,
+                './notification/notify': mockNotification
+            })
+        });
+
+        return await new Promise(resolve => {
+            immowelt.run(mockStats).then(() => {
+                const immoweltDbContent = immowelt._getStore()._db;
+                expect(immoweltDbContent.immowelt).to.be.a('array');
+
+                const notificationObj = mockNotification.get();
+                expect(notificationObj).to.be.a('object');
+                expect(notificationObj.serviceName).to.equal('immowelt');
+
+                /** check the actual structure **/
+                expect(notificationObj.payload.id).to.be.a('number');
+                expect(notificationObj.payload.price).to.be.a('string');
+                expect(notificationObj.payload.size).to.be.a('string');
+                expect(notificationObj.payload.title).to.be.a('string');
+                expect(notificationObj.payload.link).to.be.a('string');
+                expect(notificationObj.payload.address).to.be.a('string');
+
+                /** check the values if possible **/
+                expect(notificationObj.payload.id).to.equal(
+                    immoweltDbContent.immowelt[immoweltDbContent.immowelt.length - 1]
+                );
+                expect(notificationObj.payload.price).that.does.include('€');
+                expect(notificationObj.payload.size).that.does.include('m²');
+                expect(notificationObj.payload.title).to.be.not.empty;
+                expect(notificationObj.payload.link).that.does.include('https://www.immowelt.de');
+                expect(notificationObj.payload.address).to.be.not.empty;
+
+                resolve();
+            });
+        });
+    });
+});

test/kalaydo.test.js

@@ -0,0 +1,48 @@
+const mockNotification = require('./mocks/mockNotification');
+const mockConfig = require('../conf/config.test');
+const mockStore = require('./mocks/mockStore');
+const mockStats = require('./mocks/mockStats');
+const proxyquire = require('proxyquire').noCallThru();
+const expect = require('chai').expect;
+
+describe('#kalaydo testsuite()', () => {
+
+    const kalaydo = proxyquire('../lib/provider/kalaydo', {
+        '../../conf/config.json': mockConfig,
+        '../lib/fredy': proxyquire('../lib/fredy', {
+            './services/store': mockStore,
+            './notification/notify': mockNotification
+        })
+    });
+
+    it('should test kalaydo provider', async () => {
+        return await new Promise(resolve => {
+            kalaydo.run(mockStats).then(() => {
+                const kalaydoDbContent = kalaydo._getStore()._db;
+
+                expect(kalaydoDbContent.kalaydo).to.be.a('array');
+
+
+                const notificationObj = mockNotification.get();
+                expect(notificationObj).to.be.a('object');
+                expect(notificationObj.serviceName).to.equal('kalaydo');
+
+                /** check the actual structure **/
+                expect(notificationObj.payload.id).to.be.a('string');
+                expect(notificationObj.payload.price).to.be.a('string');
+                expect(notificationObj.payload.size).to.be.a('string');
+                expect(notificationObj.payload.title).to.be.a('string');
+                expect(notificationObj.payload.link).to.be.a('string');
+
+                /** check the values if possible **/
+                expect(notificationObj.payload.id).to.equal(kalaydoDbContent.kalaydo[kalaydoDbContent.kalaydo.length - 1]);
+                expect(notificationObj.payload.price).that.does.include('€');
+                expect(notificationObj.payload.size).that.does.include('m²');
+                expect(notificationObj.payload.title).to.be.not.empty;
+                expect(notificationObj.payload.link).that.does.include('https://www.kalaydo.de');
+
+                resolve();
+            });
+        });
+    });
+});

test/kleinanzeigen.test.js

@@ -0,0 +1,51 @@
+const mockNotification = require('./mocks/mockNotification');
+const mockConfig = require('../conf/config.test');
+const mockStats = require('./mocks/mockStats');
+const mockStore = require('./mocks/mockStore');
+const proxyquire = require('proxyquire').noCallThru();
+const expect = require('chai').expect;
+
+describe('#kleinanzeigen testsuite()', () => {
+
+    it('should test kleinanzeigen provider', async () => {
+
+        const kleinanzeigen = proxyquire('../lib/provider/kleinanzeigen', {
+            '../../conf/config.json': mockConfig,
+            '../lib/fredy': proxyquire('../lib/fredy', {
+                './services/store': mockStore,
+                './notification/notify': mockNotification
+            })
+        });
+
+        return await new Promise(resolve => {
+            kleinanzeigen.run(mockStats).then(() => {
+                const kleinanzeigenDbContent = kleinanzeigen._getStore()._db;
+                expect(kleinanzeigenDbContent.kleinanzeigen).to.be.a('array');
+
+                const notificationObj = mockNotification.get();
+                expect(notificationObj).to.be.a('object');
+                expect(notificationObj.serviceName).to.equal('kleinanzeigen');
+
+                /** check the actual structure **/
+                expect(notificationObj.payload.id).to.be.a('number');
+                expect(notificationObj.payload.price).to.be.a('string');
+                expect(notificationObj.payload.size).to.be.a('string');
+                expect(notificationObj.payload.title).to.be.a('string');
+                expect(notificationObj.payload.link).to.be.a('string');
+                expect(notificationObj.payload.address).to.be.a('string');
+
+                /** check the values if possible **/
+                expect(notificationObj.payload.id).to.equal(
+                    kleinanzeigenDbContent.kleinanzeigen[kleinanzeigenDbContent.kleinanzeigen.length - 1]
+                );
+                expect(notificationObj.payload.price).that.does.include('€');
+                expect(notificationObj.payload.size).that.does.include('m²');
+                expect(notificationObj.payload.title).to.be.not.empty;
+                expect(notificationObj.payload.link).that.does.include('https://www.ebay-kleinanzeigen.de');
+                expect(notificationObj.payload.address).to.be.not.empty;
+
+                resolve();
+            });
+        });
+    });
+});

test/mocks/mockNotification.js

@@ -0,0 +1,11 @@
+module.exports = {
+  _tmpStore: {},
+
+  send: (serviceName, payload) => {
+    this._tmpStore = { serviceName, payload };
+  },
+
+  get: () => {
+    return this._tmpStore;
+  }
+};

test/mocks/mockStats.js

@@ -0,0 +1,5 @@
+module.exports = {
+  setLastScrape: () => {
+    /*noop*/
+  }
+};

test/mocks/mockStore.js

@@ -0,0 +1,27 @@
+class Store {
+  constructor(name) {
+    this._name = name;
+    this._db = {};
+  }
+
+  get warmup() {
+    this._db = {};
+    return new Promise(resolve => resolve());
+  }
+
+  set knownListings(value) {
+    if (!Array.isArray(value)) throw Error('Not a valid array');
+    return new Promise(resolve => {
+      this._db[this._name] = value;
+      resolve(value);
+    });
+  }
+
+  get bla() {}
+
+  get knownListings() {
+    return this._db[this._name] || [];
+  }
+}
+
+module.exports = Store;

test/utils.test.js

@@ -0,0 +1,13 @@
+const utils = require('../lib/utils');
+const assert = require('assert');
+
+describe('utils', () => {
+  describe('#isOneOf()', () => {
+    it('should be false', () => {
+      assert.equal(utils.isOneOf('bla', ['blub']), false);
+    });
+    it('should be true', () => {
+      assert.equal(utils.isOneOf('bla blub blubber', ['bla']), true);
+    });
+  });
+});