Christian Kellner
89d239c360
New Listings view ( #192 )
...
* completing found listings
---------
Co-authored-by: Christian Kellner <Christian.Kellner1@ibm.com >
2025-09-25 15:03:47 +02:00
Christian Kellner
11fd18e76a
Puppeteer improvements ( #186 )
...
* improving puppeteer handling
* upgrade dependencies
* reduce logging
* upgrade nanoid
2025-09-22 20:53:00 +02:00
Christian Kellner
c839f3abc9
Check if a listing is still active ( #184 )
...
* check if a listing is still active
* upgrade dependencies
2025-09-22 09:57:50 +02:00
orangecoding
f339a2e2cf
adding version banner to check if a new version of fredy is available
2025-09-20 19:37:27 +02:00
orangecoding
d1dad7fd3b
adding new unique index, adding button to start now
2025-09-18 20:48:25 +02:00
orangecoding
28e885f6c7
fixing migration checksum
2025-09-18 18:42:19 +02:00
orangecoding
1d99fc95f7
using cron to run demo cleanup every day at midnight
2025-09-18 18:04:49 +02:00
orangecoding
28f0a167e6
fixing docker migration path
2025-09-18 17:28:30 +02:00
Christian Kellner
8d95f052c6
Migrate to SQLite ( #174 )
...
* Migrating Fredy from LowDb to SqLite 🎉
* adding new sql migration system for future sql migrations
* adding setting to change sqlite path for db files
* create migration plan for graceful migration lowdb -> sqlite
* Improving Documentation
* adding test for sqliteconnection
* upgrading dependencies
* making nodejs 22 as min version
* improve scraper
* adding overwrite ability for db migra
2025-09-18 15:38:23 +02:00
Christian Kellner
de119c9199
Update logger.js
2025-09-14 15:46:31 +02:00
orangecoding
ae1c4d936b
do not log debug on production
2025-09-14 10:40:18 +02:00
orangecoding
bda4212249
improve logging
2025-09-14 10:32:39 +02:00
orangecoding
21415dcff3
using winston logger
2025-09-13 18:57:56 +02:00
orangecoding
d66dc2cd93
improve tracking
2025-09-13 17:06:18 +02:00
orangecoding
ac0ea64c07
remove unnecessary logging
2025-09-12 13:41:08 +02:00
orangecoding
85cea66051
improving tracking. now using internal tracking
2025-09-12 13:38:53 +02:00
orangecoding
09c6ce1d0b
improve similarity cache. It now checks for similarities independend from jobs
2025-09-07 22:15:14 +02:00
weakmap@gmail.com
3de81903a1
using eslint 9
2025-08-31 20:09:38 +02:00
Alexander Roidl
2b36f868e7
Project-wide linting and formatting ( #150 )
...
* chore: configure project-wide linting and formatting
* chore: run lint autofix and formatter
2025-07-26 20:42:58 +02:00
Alexander Roidl
2302f69ff3
Rename NPM startup scripts ( #144 )
...
* feat: rename npm start scripts
2025-07-25 13:13:04 +02:00
weakmap@gmail.com
53121742c2
improving error message
2025-06-14 19:03:23 +02:00
Christian Kellner
a42905d63f
fixing docker ignore issue
2025-06-04 09:46:07 +02:00
Christian Kellner
f9c6b10976
fixing tests
2025-05-26 10:43:13 +02:00
Leon C.
1f54bcfd3f
ImmoScout: Allow web paths with SEO optimization to be filtered to query params ( #128 )
2025-05-20 12:44:43 +02:00
Christian Kellner
25dfad4f5d
run cleanup once at start
2025-05-16 13:26:39 +02:00
Christian Kellner
b7a3823049
console log when removing demo jobs
2025-05-16 13:25:55 +02:00
Christian Kellner
6964998695
fixing removing demo jobs
2025-05-16 13:20:54 +02:00
Christian Kellner
030e0ca169
starting docu on reverse engineering immoscout api ( #127 )
...
* starting docu on reverse engineering immoscout api
* improving immoscout reverse engineering and adding support for most other types
2025-05-14 13:58:58 +02:00
Christian Kellner
1ecbbdd774
better logging
2025-01-07 13:34:43 +01:00
Christian Kellner
e1db3840f6
adding puppeteer timeout and fixing waitForSelector
2025-01-07 12:37:50 +01:00
Christian Kellner
90a4ee5dcf
better logging, fixing code smells
2025-01-07 12:25:19 +01:00
weakmap@gmail.com
3b8279c714
adding fredy version
2024-12-17 13:07:25 +01:00
Christian Kellner
214e714c03
Puppeteer rewrite ( #119 )
...
* Moving to puppeteer | removing scrapingAnt
2024-12-17 12:38:28 +01:00
Christian Kellner
324b14da50
improving tracking
2024-12-03 14:23:09 +01:00
Christian Kellner
f8f911aa00
improving tracking
2024-12-03 14:05:00 +01:00
Christian Kellner
337ee922a6
Demo Mode ( #117 )
...
* Adding Demo Mode to Fredy
2024-11-22 09:11:10 +01:00
Christian Kellner
8f91267b5d
sending tracking information ( #116 )
...
* Ability to send tracking information
2024-11-20 22:22:16 +01:00
Christian Kellner
92db8219b4
building multi platform docker images ( #101 )
...
* building multi platform docker images
* upgrading dependencies | using scraping ant for neubaukompass
2024-07-24 09:32:21 +02:00
Vladislav
e7db4e23f5
update error handling ( #100 )
2024-07-22 10:41:30 +02:00
Christian Kellner
3bda88a075
upgrade dependencies
2023-05-11 11:51:23 +02:00
weakmap@gmail.com
4ba098e0b6
bringing back immonet by using scrapingant
2023-04-15 18:24:51 +02:00
weakmap@gmail.com
3ec9ed3b2a
ignoring expired ssl certificate o0
2023-04-07 19:44:59 +02:00
Christian Kellner
2c5eceb0c1
Making Fredy an ESM project ( #70 )
...
Making Fredy an ESM project
2023-03-13 13:42:43 +01:00
weakmap@gmail.com
8a5fbcdf71
moving to node-fetch coz axios is causing issues with scrapingAnt
2022-12-20 10:21:15 +01:00
Christian Kellner
2062aa11a3
Scrapingant proxies ( #59 )
...
* preparing scraping ant proxies
* adding general settings for scraping ant proxy
* retrying with new ui settings
2022-06-13 08:10:30 +02:00
Christian Kellner
676d48807a
scraping ant retries
2022-04-29 13:22:39 +02:00
orangecoding
5347d0014d
adding scraping ant infos as we now need to use residental proxies
2022-03-25 11:19:17 +01:00
Jochen Schalanda
b6b8d6814c
Make requestDriver more resilient to errors ( #46 )
...
If the async request performed in `requestDriver.makeDriver()` fails, it would call the `callback` function with empty parameters but then continue the execution which can lead to the following error and crash of Fredy:
```
Error while trying to scrape data. Received error: Request failed with status code 504
/fredy/lib/services/requestDriver.js:25
if (typeof result.data === 'object' && url.toLowerCase().indexOf('scrapingant') !== -1) {
^
TypeError: Cannot read properties of undefined (reading 'data')
at driver (/fredy/lib/services/requestDriver.js:25:23)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
```
2022-01-31 10:48:11 +01:00
orangecoding
c1c4d55ede
adding or replacing sort params for provider urls when necessary
2021-11-26 21:02:09 +01:00
Christian Kellner
c97b323b35
fix: node would crash if axios throw an error
2021-11-11 15:31:53 +01:00
