fixing toasts not showing on certain pages / adding statement about ai 🤖

2026-06-16 12:31:07 +00:00 · 2026-04-27 15:58:41 +02:00
parent f30ec4645c
commit 8a7b14c079
10 changed files with 168 additions and 19 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -0,0 +1,120 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Fredy is a self-hosted real estate finder for Germany. It scrapes German real estate portals (ImmoScout24, Immowelt, Immonet, Kleinanzeigen, WG-Gesucht, etc.), deduplicates results across providers, and sends notifications via Slack, Telegram, Email, Discord, ntfy, etc. It includes a React web UI and a built-in MCP server for LLM access to listings data.
+
+- Node.js >= 22, ESM-only (`"type": "module"`)
+- Default port: 9998, default login: admin / admin
+- SQLite via `better-sqlite3` (synchronous — all DB ops are sync; only network I/O is async)
+
+## Commands
+
+```bash
+# Development
+yarn run start:backend:dev    # nodemon backend
+yarn run start:frontend:dev   # Vite dev server (proxies /api → :9998)
+
+# Production
+yarn run start:backend        # NODE_ENV=production node index.js
+yarn run build:frontend       # vite build → ui/public/
+
+# Tests
+yarn test                     # Live tests (hits actual providers)
+yarn test:offline             # Offline tests using HTML/JSON fixtures (fast, preferred)
+yarn test:download-fixtures   # Re-download fresh provider HTML fixtures
+
+# Single test file
+TEST_MODE=offline npx vitest run test/provider/immoscout.test.js
+
+# Lint / Format
+yarn lint && yarn lint:fix
+yarn format && yarn format:check
+
+# DB migrations
+yarn migratedb
+```
+
+## Architecture
+
+### Core data flow
+
+```
+index.js (startup)
+  ├── runMigrations()
+  ├── getProviders()            # lazily imports lib/provider/*.js
+  ├── similarityCache.init()    # preloads hash cache from DB
+  ├── api.js                    # starts restana HTTP server
+  └── initJobExecutionService() # registers event-bus listeners + starts scheduler
+
+scheduler (every N minutes) or manual trigger via POST /api/jobs/:id/run
+  └── FredyPipelineExecutioner.execute()
+      1. queryStringMutator(url)           # inject sort-by-date param
+      2. provider.getListings()            # API or Puppeteer+Cheerio
+      3. provider.normalize(listing)       # raw → ParsedListing
+      4. provider.filter(listing)          # blacklist + required fields
+      5. filter to hashes not yet in DB
+      6. provider.fetchDetails()           # optional enrichment
+      7. geocodeAddress()                  # optional lat/lng
+      8. storeListings()
+      9. similarityCache.checkAndAddEntry() # cross-provider dedup
+      10. _filterBySpecs() + _filterByArea()
+      11. notify.send()                    # fan-out to all adapters
+```
+
+### Plugin systems
+
+**Providers** (`lib/provider/*.js`) — each module exports:
+- `metaInformation` — `{ id, name, baseUrl }`
+- `config` — `ProviderConfig` with `requiredFieldNames`, `crawlContainer`, `crawlFields`, `sortByDateParam`, `normalize()`, `filter()`, optional `getListings()`, `fetchDetails()`, `activeTester()`
+- `init(sourceConfig, blacklist)` — called before each job run; providers are **stateful modules** holding mutable `url` and `appliedBlackList` at module scope
+
+**Notification adapters** (`lib/notification/adapter/*.js`) — each exports:
+- `config` — `{ id, name, description, fields }` (drives the UI form)
+- `send({ serviceName, newListings, notificationConfig, jobKey, baseUrl })`
+- Loaded dynamically at startup via `fs.readdirSync`
+
+### Key services
+
+| Service | Location | Notes |
+|---|---|---|
+| Event bus | `lib/services/events/event-bus.js` | Plain `EventEmitter`; events: `jobs:runAll`, `jobs:runOne`, `jobs:status` |
+| SSE broker | `lib/services/sse/sse-broker.js` | Per-userId `Set<ServerResponse>`; heartbeat every 25s; pushes job status to UI |
+| Similarity cache | `lib/services/similarity-check/` | In-memory SHA-256 Set; refreshes hourly; cross-provider dedup by title+price+address |
+| SqliteConnection | `lib/services/storage/SqliteConnection.js` | Singleton, WAL mode; `execute()`, `query()`, `withTransaction()` |
+| Migrations | `lib/services/storage/migrations/` | Numbered JS files each exporting `up(db)`; checksum-tracked in `schema_migrations` |
+| Extractor | `lib/services/extractor/` | Orchestrates Puppeteer + Cheerio; shared browser instance per job |
+
+### Frontend
+
+- React 19 SPA, Vite build → `ui/public/` (served as static by backend)
+- State: Zustand single store with per-domain slices
+- UI library: `@douyinfe/semi-ui`
+- Map: MapLibre GL + `@mapbox/mapbox-gl-draw` + `@turf/boolean-point-in-polygon` for GeoJSON polygon filters
+- In dev: Vite proxies `/api` to `:9998`
+
+### MCP server
+
+Two transports:
+1. **stdio** (`lib/mcp/stdio.js`) — for Claude Desktop/LM Studio; opens its own DB connection (main process need not be running)
+2. **HTTP** (`/api/mcp`) — authenticated via Bearer token (`mcp_token` column in `users` table)
+
+Tools: `list_jobs`, `get_job`, `list_listings`, `get_listing`, `get_current_date_time`. Responses are Markdown via `lib/mcp/mcpNormalizer.js`.
+
+## Key Conventions
+
+- **ESM only** — `import`/`export` everywhere, no CommonJS
+- **JSDoc typedefs** (no TypeScript) in `lib/types/` — `listing.js`, `job.js`, `filter.js`, `providerConfig.js`
+- **Copyright header** required on all `.js` files — enforced by `lint-staged` pre-commit hook via `copyright.js`
+- **`NoNewListingsWarning`** (`lib/errors.js`) is used as control flow to short-circuit the pipeline (not an error)
+- **Test fixtures** in `test/testFixtures/` — HTML/JSON snapshots per provider; `TEST_MODE=offline` mocks `puppeteerExtractor` and global `fetch` via `test/offlineFixtures.js`
+- **`conf/config.json`** is the only runtime config file; created with defaults if missing
+
+## Coding
+- After building the task, run the linter
+- After building the task, run the tests
+- New features must be tested
+- New features must be properly documented with JsDoc
+- You do **not** commit any changes, you do **not** create a new branch unless I told you so
--- a/README.md
+++ b/README.md
@@ -240,6 +240,20 @@ flowchart TD
    F1 --> F2
 ```

+------------------------------------------------------------------------
+## 🤖 Using AI such as Claude Code
+When I started building Fredy, LLMs were still basically the wet dream of a few nerdy scientists.
+
+Nowadays, it’s easier than ever to throw a prompt into the LLM of your choice and let 'the AI' build your stuff. I’m not against that. I use Claude Code myself for smaller tasks, and I do think these tools can be really useful.
+
+That said, I still believe humans should stay in charge. AI is great-ish at writing code, but it still lacks creativity, context, and the ability to see the full picture.
+
+So, if you want to contribute to Fredy, using AI tools to get things done is totally fine. Just please don’t stop thinking.
+
+I’ve had one too many PRs full of hallucinated bullshit.
+
+**Thanks ;)**
+
 ------------------------------------------------------------------------

 ## 👐 Contributing
--- a/lib/api/routes/generalSettingsRoute.js
+++ b/lib/api/routes/generalSettingsRoute.js
@@ -9,6 +9,7 @@ import fs from 'fs';
 import { ensureDemoUserExists } from '../../services/storage/userStorage.js';
 import logger from '../../services/logger.js';
 import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
+import { isAdmin } from '../security.js';
 const service = restana();
 const generalSettingsRouter = service.newRouter();

@@ -23,7 +24,7 @@ generalSettingsRouter.post('/', async (req, res) => {
  }
  const localSettings = await getSettings();

-  if (localSettings.demoMode) {
+  if (localSettings.demoMode && !isAdmin(req)) {
    res.send(new Error('In demo mode, it is not allowed to change these settings.'));
    return;
  }
--- a/lib/api/routes/jobRouter.js
+++ b/lib/api/routes/jobRouter.js
@@ -183,7 +183,7 @@ jobRouter.post('/', async (req, res) => {
      return;
    }

-    if (settings.demoMode && jobFromDb && jobFromDb.name === DEMO_JOB_NAME) {
+    if (settings.demoMode && !isAdmin(req) && jobFromDb && jobFromDb.name === DEMO_JOB_NAME) {
      res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
      return;
    }
@@ -212,7 +212,7 @@ jobRouter.delete('', async (req, res) => {
  const settings = await getSettings();
  try {
    const job = jobStorage.getJob(jobId);
-    if (settings.demoMode && job.name === DEMO_JOB_NAME) {
+    if (settings.demoMode && !isAdmin(req) && job.name === DEMO_JOB_NAME) {
      res.send(new Error('Sorry, but you cannot remove the Demo Job ;)'));
      return;
    }
@@ -235,7 +235,7 @@ jobRouter.put('/:jobId/status', async (req, res) => {
  try {
    const job = jobStorage.getJob(jobId);

-    if (settings.demoMode && job.name === DEMO_JOB_NAME) {
+    if (settings.demoMode && !isAdmin(req) && job.name === DEMO_JOB_NAME) {
      res.send(new Error('Sorry, but you cannot change the Status of our Demo Job ;)'));
      return;
    }
--- a/lib/api/routes/listingsRouter.js
+++ b/lib/api/routes/listingsRouter.js
@@ -110,7 +110,7 @@ listingsRouter.delete('/job', async (req, res) => {
  const { jobId, hardDelete = false } = req.body;
  const settings = await getSettings();
  try {
-    if (settings.demoMode) {
+    if (settings.demoMode && !isAdminFn(req)) {
      res.send(new Error('Sorry, but you cannot remove listings in demo mode ;)'));
      return;
    }
--- a/lib/api/routes/userRoute.js
+++ b/lib/api/routes/userRoute.js
@@ -7,6 +7,7 @@ import restana from 'restana';
 import * as userStorage from '../../services/storage/userStorage.js';
 import * as jobStorage from '../../services/storage/jobStorage.js';
 import { getSettings } from '../../services/storage/settingsStorage.js';
+import { isAdmin as isAdminUser } from '../security.js';
 const service = restana();
 const userRouter = service.newRouter();
 function checkIfAnyAdminAfterRemovingUser(userIdToBeRemoved, allUser) {
@@ -29,7 +30,7 @@ userRouter.get('/:userId', async (req, res) => {
 });
 userRouter.delete('/', async (req, res) => {
  const settings = await getSettings();
-  if (settings.demoMode) {
+  if (settings.demoMode && !isAdminUser(req)) {
    res.send(new Error('In demo mode, it is not allowed to remove user.'));
    return;
  }
@@ -51,7 +52,7 @@ userRouter.delete('/', async (req, res) => {
 });
 userRouter.post('/', async (req, res) => {
  const settings = await getSettings();
-  if (settings.demoMode) {
+  if (settings.demoMode && !isAdminUser(req)) {
    res.send(new Error('In demo mode, it is not allowed to change or add user.'));
    return;
  }
--- a/lib/api/routes/userSettingsRoute.js
+++ b/lib/api/routes/userSettingsRoute.js
@@ -6,6 +6,7 @@
 import restana from 'restana';
 import SqliteConnection from '../../services/storage/SqliteConnection.js';
 import { getSettings, upsertSettings } from '../../services/storage/settingsStorage.js';
+import { isAdmin } from '../security.js';
 import { resetGeocoordinatesAndDistanceForUser } from '../../services/storage/listingsStorage.js';
 import { geocodeAddress } from '../../services/geocoding/geoCodingService.js';
 import { autocompleteAddress } from '../../services/geocoding/autocompleteService.js';
@@ -46,7 +47,7 @@ userSettingsRouter.post('/home-address', async (req, res) => {
  const { home_address } = req.body;
  const settings = await getSettings();

-  if (settings.demoMode) {
+  if (settings.demoMode && !isAdmin(req)) {
    res.send(new Error('In demo mode, it is not allowed to change the home address.'));
    return;
  }
@@ -81,7 +82,7 @@ userSettingsRouter.post('/news-hash', async (req, res) => {
  const { news_hash } = req.body;

  const globalSettings = await getSettings();
-  if (globalSettings.demoMode) {
+  if (globalSettings.demoMode && !isAdmin(req)) {
    res.statusCode = 403;
    res.send({ error: 'In demo mode, it is not allowed to change settings.' });
    return;
@@ -102,7 +103,7 @@ userSettingsRouter.post('/provider-details', async (req, res) => {
  const { provider_details } = req.body;

  const globalSettings = await getSettings();
-  if (globalSettings.demoMode) {
+  if (globalSettings.demoMode && !isAdmin(req)) {
    res.statusCode = 403;
    res.send({ error: 'In demo mode, it is not allowed to change settings.' });
    return;
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
 {
  "name": "fredy",
-  "version": "21.0.0",
+  "version": "21.0.1",
  "description": "[F]ind [R]eal [E]states [d]amn eas[y].",
  "scripts": {
    "prepare": "husky",
@@ -77,12 +77,12 @@
    "cheerio": "^1.2.0",
    "cookie-session": "2.1.1",
    "handlebars": "4.7.9",
-    "maplibre-gl": "^5.23.0",
+    "maplibre-gl": "^5.24.0",
    "nanoid": "5.1.9",
    "node-cron": "^4.2.1",
    "node-fetch": "3.3.2",
    "node-mailjet": "6.0.11",
-    "nodemailer": "^8.0.5",
+    "nodemailer": "^8.0.6",
    "p-throttle": "^8.1.0",
    "package-up": "^5.0.0",
    "puppeteer": "^24.42.0",
@@ -93,14 +93,14 @@
    "react-chartjs-2": "^5.3.1",
    "react-dom": "19.2.5",
    "react-range-slider-input": "^3.3.5",
-    "react-router": "7.14.1",
-    "react-router-dom": "7.14.1",
+    "react-router": "7.14.2",
+    "react-router-dom": "7.14.2",
    "resend": "^6.12.2",
-    "restana": "5.2.0",
+    "restana": "6.0.0",
    "semver": "^7.7.4",
    "serve-static": "2.2.1",
    "slack": "11.0.2",
-    "vite": "8.0.9",
+    "vite": "8.0.10",
    "x-var": "^3.0.1",
    "zustand": "^5.0.12"
  },
@@ -121,6 +121,6 @@
    "lint-staged": "16.4.0",
    "nodemon": "^3.1.14",
    "prettier": "3.8.3",
-    "vitest": "^4.1.4"
+    "vitest": "^4.1.5"
  }
 }
--- a/ui/src/Index.jsx
+++ b/ui/src/Index.jsx
@@ -6,10 +6,15 @@
 import { HashRouter } from 'react-router-dom';
 import { createRoot } from 'react-dom/client';
 import en_US from '@douyinfe/semi-ui-19/lib/es/locale/source/en_US';
-import { LocaleProvider } from '@douyinfe/semi-ui-19';
+import { LocaleProvider, semiGlobal } from '@douyinfe/semi-ui-19';
 import App from './App';
 import './Index.less';

+// Semi UI uses react-dom (not react-dom/client) internally for imperative renders
+// like Toast, Notification, etc. In React 19, createRoot was removed from react-dom
+// and lives only in react-dom/client — inject it so Toast can create its own root.
+semiGlobal.config.createRoot = createRoot;
+
 const container = document.getElementById('fredy');
 const root = createRoot(container);

--- a/ui/src/Index.less
+++ b/ui/src/Index.less
@@ -88,6 +88,13 @@ button:focus-visible,
  box-shadow: none !important;
 }

+// Semi Toast dark theme overrides
+.semi-toast-content {
+  background-color: @color-elevated !important;
+  border: 1px solid @color-border-bright !important;
+  box-shadow: 0 4px 20px rgba(0, 0, 0, 0.5) !important;
+}
+
 // Semi Modal dark theme overrides
 .semi-modal-content {
  background: #161616 !important;