improve markdown readme's & and adding ability to send telegram messages to a topic in a supergroup

index.js

@@ -38,6 +38,7 @@ await runMigrations();
 const providers = await getProviders();
 
 similarityCache.initSimilarityCache();
+similarityCache.startSimilarityCacheReloader();
 
 //assuming interval is always in minutes
 const INTERVAL = config.interval * 60 * 1000;

lib/notification/adapter/apprise.md

@@ -1,3 +1,8 @@
 ### Apprise Adapter
 
-Refer to the [instructions](https://github.com/caronc/apprise-api#installation) on how to set up an Apprise instance and how to configure your preferred notification service.
+Use [Apprise](https://github.com/caronc/apprise-api#installation) to forward notifications to many different services.
+
+Quick start:
+- Set up an Apprise API instance (see the installation guide linked above).
+- Configure your preferred notification service(s) within Apprise.
+- In Fredy, point the Apprise adapter to your Apprise API endpoint.

lib/notification/adapter/console.md

@@ -1,4 +1,3 @@
 ### Console Adapter
 
-The console adapter prints everything found by Fredy into the console (not sending any notifications to you). This can be useful when you want to check if your search
-criteria meet the expectations.
+The console adapter prints everything found by Fredy to the console (it does not send notifications). This is useful to verify that your search criteria work as expected before enabling a real notification service.

lib/notification/adapter/discord_webhook.md

@@ -1,4 +1,8 @@
-### Discord Adapter
+### Discord Webhook Adapter
 
-To use the [Discord](https://discord.com/) Adapter, you need to create a webhook on the Discord channel of your choice. You can follow the instructions of _Making A Webhook_ on [this support website](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks).
-Once you have created a webhook, copy and paste the webhook URL.
+Use a Discord channel webhook to receive notifications.
+
+Quick start:
+- Create a webhook in your target Discord channel. See the "Intro to Webhooks" guide on the Discord support site: https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks
+- Copy the generated webhook URL.
+- In Fredy, configure the Discord adapter with this webhook URL.

lib/notification/adapter/mailJet.md

@@ -1,8 +1,8 @@
-### MailJet Adapter
+### Mailjet Adapter
 
-To use [MailJet](https://mailjet.com), you need to create an account. You'll need to decide from which email address you want Fredy to send from.
+To use [Mailjet](https://mailjet.com), create an account and decide which email address Fredy should send from.
 
-E.g. if you use yourGmailAccount@gmail.com, you have to add this to MailJet and verify it as well.  
-The given public/private api keys are needed in order to use MailJet with Fredy. Fredy will use the same template, it is using for SendGrid.
+For example, if you use yourGmailAccount@gmail.com, add and verify this address in Mailjet.
+Provide your public/private API keys in Fredy's configuration. Fredy uses the same email template as for SendGrid.
 
-If this email should be sent to multiple receiver, use a comma separator (some@email.com, someOther@email.com).
+To send to multiple recipients, separate email addresses with commas (e.g., some@email.com, someOther@email.com).

lib/notification/adapter/mattermost.md

@@ -1,5 +1,8 @@
 ### Mattermost Adapter
 
-For Mattermost, you need to create a incoming webhook. This is pretty easy. Please visit the steps in the [developer docs](https://docs.mattermost.com/developer/webhooks-incoming.html) and follow the instructions.
+Receive notifications in Mattermost via an incoming webhook.
 
-As a result, you get the webhook URL for configuration in fredy. In addition, the target channel must be defined.
+Quick start:
+- Create an incoming webhook following the Mattermost developer docs: https://docs.mattermost.com/developer/webhooks-incoming.html
+- Copy the webhook URL.
+- In Fredy, configure the Mattermost adapter with this URL and the target channel.

lib/notification/adapter/ntfy.md

@@ -1,5 +1,8 @@
 ### ntfy Adapter
 
-For ntfy, you need to create a topic on your preferred ntfy instance. This is pretty easy. Please visit the steps in the [docs](https://docs.ntfy.sh/publish/) and follow the instructions.
+Send push notifications using an ntfy topic.
 
-As a result, you get the URL for configuration in fredy. In addition, the priority must be defined.
+Quick start:
+- Create or choose a topic on your preferred ntfy instance (see docs: https://docs.ntfy.sh/publish/).
+- Copy the publish URL for that topic.
+- In Fredy, configure the ntfy adapter with the topic URL and set a priority.

lib/notification/adapter/pushover.md

@@ -1,5 +1,8 @@
 ### Pushover Adapter
 
-Refer to the [instructions](https://support.pushover.net/i7-what-is-pushover-and-how-do-i-use-it) to set up your Pushover application.
+Use Pushover to receive push notifications on your devices.
 
-After setting up the application, please enter both your newly created User key and API token.
+Setup:
+- Follow Pushover's getting-started guide: https://support.pushover.net/i7-what-is-pushover-and-how-do-i-use-it
+- Create an application and obtain your User Key and API Token.
+- In Fredy, configure the Pushover adapter with both values.

lib/notification/adapter/sendGrid.md

@@ -1,9 +1,12 @@
 ### SendGrid Adapter
 
-SendGrid is a free email service (free as in "you cannot send more than 100(Sendgrid) and 200(Mailjet) emails a day"), which is more than enough for Fredy.
+SendGrid is an email delivery service with a generous free tier, which is more than enough for Fredy.
 
-To use [SendGrid](https://sendgrid.com/), you need to create an account. You'll need to decided from which email address you want Fredy to send from. E.g. if you use yourGmailAccount@gmail.com, you have to add this to sendgrid and verify it as well.
+Setup:
+- Create a SendGrid account: https://sendgrid.com/
+- Decide which email address Fredy should send from (e.g., yourGmailAccount@gmail.com), add it to SendGrid, and complete the verification.
+- Create an API key and add it to Fredy's configuration.
+- Create a Dynamic Template in SendGrid. You can copy the template from `/lib/notification/emailTemplate/template.hbs`.
 
-Lastly you have to create an api-key and feed it into Fredy's config, as well as creating a new dynamic template. For this new template, I recommend copying and pasting the code from the one I have provided under `/lib/notification/emailTemplate/template.hbs`.
-
-If this email should be sent to multiple receiver use a comma separator (some@email.com, someOther@email.com).
+Sending to multiple recipients:
+- Separate email addresses with commas (e.g., some@email.com, someOther@email.com).

lib/notification/adapter/slack.md

@@ -1,4 +1,5 @@
-### Slack Adapter
-IMPORTANT:
-Don't use this adapter anymore, it is outdated and only here for backwards compatability reasons. Use the new Slack Adapter with webhooks!
+### Slack Adapter (Legacy)
+
+*IMPORTANT:*
+This legacy adapter is outdated and kept only for backward compatibility. Please use the Slack adapter with webhooks instead.
 

lib/notification/adapter/slack_with_webhooks.md

@@ -1,6 +1,10 @@
-### Slack Adapter
+### Slack Adapter (Webhooks)
 
-IMPORTANT:
-This is the new version of the Slack adapter. I strongly encourage you to use it, the old version is now unmaintained and only kept due to backwards compatability reasons.
+*IMPORTANT:*
+This is the recommended Slack adapter. The old Slack adapter is unmaintained and kept only for backward compatibility.
 
-In order to use [Slack](https://slack.com), you need to create an account. When done, create a new channel and add the Webhook integration to that channel. Copy the webhook url. That's it.
+Setup:
+- Create a Slack account and workspace if you don't have one: https://slack.com
+- Create a channel where you want to receive notifications.
+- Add the Incoming Webhooks integration to that channel and copy the Webhook URL.
+- In Fredy, configure the Slack Webhook adapter with this URL.

lib/notification/adapter/sqlite.md

@@ -1,9 +1,21 @@
 ### SQLite Adapter
 
-This adapter stores search results in an SQLite database. By default, the database is located at `db/listings.db`, but you can configure a custom location. This file can be used for further analysis later.
+This adapter stores search results in an SQLite database. By default, the database is located at `db/listings.db`, but you can configure a custom location. The file can be used for analysis later.
 
-The database table contains the following columns (all stored as `TEXT` type):
+The table contains the following columns (all stored as `TEXT`):
 
-```
-['serviceName', 'jobKey', 'id', 'size', 'rooms', 'price', 'address', 'title', 'link', 'description', 'image']
+```json
+[
+  "serviceName",
+  "jobKey",
+  "id",
+  "size",
+  "rooms",
+  "price",
+  "address",
+  "title",
+  "link",
+  "description",
+  "image"
+]
 ```

lib/notification/adapter/telegram.js

@@ -117,10 +117,24 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
   if (!adapterCfg || !adapterCfg.fields) {
     throw new Error(`Telegram adapter configuration missing for job '${jobKey || ''}'`);
   }
-  const { token, chatId } = adapterCfg.fields;
+  const { token, chatId, messageThreadId } = adapterCfg.fields;
   if (!token || !chatId) {
     throw new Error("Telegram 'token' and 'chatId' must be provided in notification config");
   }
+
+  // Optional Telegram topic/thread support (supergroups)
+  let message_thread_id;
+  if (messageThreadId !== undefined && messageThreadId !== null && `${messageThreadId}`.trim() !== '') {
+    const n = Number(messageThreadId);
+    if (Number.isInteger(n) && n > 0) {
+      message_thread_id = n;
+    } else {
+      logger.warn(
+        `Telegram adapter: 'messageThreadId' is invalid ('${messageThreadId}'). It must be a positive integer. Ignoring.`,
+      );
+    }
+  }
+
   const job = getJob(jobKey);
   const jobName = job == null ? jobKey : job.name;
 
@@ -147,6 +161,7 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
       text: buildText(jobName, serviceName, o),
       parse_mode: 'HTML',
       disable_web_page_preview: true,
+      ...(message_thread_id ? { message_thread_id } : {}),
     };
 
     if (!img) {
@@ -160,6 +175,7 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
       photo: img,
       caption: buildCaption(jobName, serviceName, o),
       parse_mode: 'HTML',
+      ...(message_thread_id ? { message_thread_id } : {}),
     }).catch(async (e) => {
       logger.error(`Error sending photo to Telegram and use a fallback: ${e.message}`);
       return await throttledCall('sendMessage', textPayload).catch((e) => {
@@ -174,7 +190,7 @@ export const send = ({ serviceName, newListings = [], notificationConfig, jobKey
 
 /**
  * Telegram notification adapter configuration schema.
- * @type {{id:string,name:string,readme:string,description:string,fields:{token:{type:string,label:string,description:string},chatId:{type:string,label:string,description:string}}}}
+ * @type {{id:string,name:string,readme:string,description:string,fields:{token:{type:string,label:string,description:string},chatId:{type:string,label:string,description:string},messageThreadId?:{type:string,label:string,description:string}}}}
  */
 export const config = {
   id: 'telegram',
@@ -192,5 +208,12 @@ export const config = {
       label: 'Chat Id',
       description: 'The chat id to send messages to you.',
     },
+    messageThreadId: {
+      type: 'text',
+      optional: true,
+      label: 'Message Thread Id (optional)',
+      description:
+        'Optional: The topic/thread id within a supergroup to post into (Telegram message_thread_id). Provide a positive integer.',
+    },
   },
 };

lib/notification/adapter/telegram.md

@@ -1,12 +1,55 @@
 ### Telegram Adapter
 
-For Telegram, you need to create a Bot. This is pretty easy. Open [this](https://telegram.me/BotFather) url on your smartphone and follow the instructions.
+Use this adapter to send notifications to Telegram via a bot. You will need:
+- A Telegram Bot token (from BotFather)
+- A chat ID (where messages will be sent)
+- Optionally: a thread ID if you want to post into a specific forum topic in a group
 
-A telegram bot is not allowed to send messages directly to a user, you as a user need to first contact the bot to get a chatId.  
-After the user has send a message to your bot the first time, you can gather the chatId like this:
+#### Create a bot
+Create a bot with BotFather: open https://telegram.me/BotFather on your phone or in Telegram Desktop and follow the instructions to get your bot token.
 
+#### Getting the chat ID
+A Telegram bot cannot message a user first; you must create a conversation (or add the bot to a group/channel) so Telegram assigns a chat the bot can access.
+
+Steps:
+1. Start a chat with your bot in Telegram (or add the bot to your group/supergroup/channel) and send any message.
+2. Fetch recent updates from the Bot API:
+   ```
+   curl -X GET "https://api.telegram.org/bot{YOUR_TELEGRAM_TOKEN}/getUpdates"
+   ```
+3. In the JSON response, find the message that you just sent and read `message.chat.id`. That value is your `chatId`.
+   - Private chats: `chat.id` is a positive number
+   - Groups/supergroups: `chat.id` is a negative number
+
+Keep your bot token secret. If `getUpdates` returns an empty list, send a new message and try again, or make sure your bot’s privacy settings allow it to see group messages when used in groups.
+
+#### Getting the thread ID (this is optional to be used for forum topics)
+If you want messages to appear inside a specific forum topic of a supergroup with Topics enabled, you also need a thread ID. In the Telegram Bot API this is called `message_thread_id`.
+
+When you need it:
+- Required only for supergroups with Topics enabled when targeting a topic
+- Not used for private chats, basic groups without Topics, or channels
+
+Steps to obtain it:
+1. In your supergroup, enable Topics (Group settings → Manage group → Topics → Enable). Now add a new topic.
+2. Add your created bot to the topic. (Click on the bot and on "Add to group")
+3. Open the desired topic (or create a new one) and send any message inside that topic.
+4. Call `getUpdates` again:
+   ```
+   curl -X GET "https://api.telegram.org/bot{YOUR_TELEGRAM_TOKEN}/getUpdates"
+   ```
+4. In the update for the message you sent inside the topic, read `message.message_thread_id`. That number is your `threadId` for this topic.
+
+Example (truncated):
 ```
-curl -X GET https://api.telegram.org/bot{YOUR_TELEGRAM_TOKEN}/getUpdates
+{
+  "message": {
+    "chat": { "id": -1001234567890, "type": "supergroup" },
+    "message_thread_id": 42,
+    "text": "hello from the topic"
+  }
+}
 ```
+Use `chat.id` as `chatId` and `message_thread_id` as `threadId` in your configuration.
 
-A more detailed list of instructions can be found here [https://core.telegram.org/bots#botfather](https://core.telegram.org/bots#botfather)
+More details about bots and BotFather: https://core.telegram.org/bots#botfather

lib/services/markdown.js

@@ -1,6 +1,4 @@
-import markdown$0 from 'markdown';
 import fs from 'fs';
-const markdown = markdown$0.markdown;
 export function markdown2Html(filePath) {
-  return markdown.toHTML(fs.readFileSync(filePath, 'utf8'));
+  return fs.readFileSync(filePath, 'utf8');
 }

lib/services/similarity-check/similarityCache.js

@@ -29,10 +29,12 @@ const reloadCycle = 60 * 60 * 1000; // every hour, refresh
  */
 let cache = new Set();
 
-// Periodically refresh the cache from storage
-setInterval(() => {
-  initSimilarityCache();
-}, reloadCycle);
+export const startSimilarityCacheReloader = () => {
+  // Periodically refresh the cache from storage
+  setInterval(() => {
+    initSimilarityCache();
+  }, reloadCycle);
+};
 
 /**
  * Initialize or refresh the similarity cache from persistent storage.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "fredy",
-  "version": "14.3.0",
+  "version": "14.3.1",
   "description": "[F]ind [R]eal [E]states [d]amn eas[y].",
   "scripts": {
     "prepare": "husky",
@@ -69,7 +69,6 @@
     "cookie-session": "2.1.1",
     "handlebars": "4.7.8",
     "lodash": "4.17.21",
-    "markdown": "^0.5.0",
     "nanoid": "5.1.6",
     "node-cron": "^4.2.1",
     "node-fetch": "3.3.2",

ui/src/views/jobs/mutation/components/notificationAdapter/NotificationAdapterMutator.jsx

@@ -21,7 +21,7 @@ const sortAdapter = (a, b) => {
 const validate = (selectedAdapter) => {
   const results = [];
   for (let uiElement of Object.values(selectedAdapter.fields || [])) {
-    if (uiElement.value == null) {
+    if (uiElement.value == null && !uiElement.optional) {
       results.push('All fields are mandatory and must be set.');
       continue;
     }
@@ -36,7 +36,7 @@ const validate = (selectedAdapter) => {
       results.push('A boolean field cannot be of a different type.');
       continue;
     }
-    if (typeof uiElement.value === 'string' && uiElement.value.length === 0) {
+    if (typeof uiElement.value === 'string' && uiElement.value.length === 0 && !uiElement.optional) {
       results.push('All fields are mandatory and must be set.');
     }
   }

ui/src/views/jobs/mutation/components/notificationAdapter/NotificationHelpDisplay.jsx

@@ -1,5 +1,5 @@
 import React from 'react';
-import { Banner } from '@douyinfe/semi-ui';
+import { Banner, MarkdownRender } from '@douyinfe/semi-ui';
 
 export default function Help({ readme }) {
   return (
@@ -8,7 +8,7 @@ export default function Help({ readme }) {
       type="info"
       closeIcon={null}
       title={<div style={{ fontWeight: 600, fontSize: '14px', lineHeight: '20px' }}>Information</div>}
-      description={<p dangerouslySetInnerHTML={{ __html: readme }} />}
+      description={<MarkdownRender raw={readme} />}
     />
   );
 }

yarn.lock

@@ -1909,11 +1909,6 @@
     "@types/babel__core" "^7.20.5"
     react-refresh "^0.17.0"
 
-abbrev@1:
-  version "1.1.1"
-  resolved "https://registry.yarnpkg.com/abbrev/-/abbrev-1.1.1.tgz#f8f2c887ad10bf67f634f005b6987fed3179aac8"
-  integrity sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q==
-
 abs-svg-path@~0.1.1:
   version "0.1.1"
   resolved "https://registry.yarnpkg.com/abs-svg-path/-/abs-svg-path-0.1.1.tgz#df601c8e8d2ba10d4a76d625e236a9a39c2723bf"
@@ -4689,13 +4684,6 @@ markdown-table@^3.0.0:
   resolved "https://registry.yarnpkg.com/markdown-table/-/markdown-table-3.0.4.tgz#fe44d6d410ff9d6f2ea1797a3f60aa4d2b631c2a"
   integrity sha512-wiYz4+JrLyb/DqW2hkFJxP7Vd7JuTDm77fvbM8VfEQdmSMqcImWeeRbHwZjBjIFki/VaMK2BhFi7oUUZeM5bqw==
 
-markdown@^0.5.0:
-  version "0.5.0"
-  resolved "https://registry.yarnpkg.com/markdown/-/markdown-0.5.0.tgz#28205b565a8ae7592de207463d6637dc182722b2"
-  integrity sha512-ctGPIcuqsYoJ493sCtFK7H4UEgMWAUdXeBhPbdsg1W0LsV9yJELAHRsMmWfTgao6nH0/x5gf9FmsbxiXnrgaIQ==
-  dependencies:
-    nopt "~2.1.1"
-
 math-intrinsics@^1.1.0:
   version "1.1.0"
   resolved "https://registry.yarnpkg.com/math-intrinsics/-/math-intrinsics-1.1.0.tgz#a0dd74be81e2aa5c2f27e65ce283605ee4e2b7f9"
@@ -5508,13 +5496,6 @@ nodemon@^3.1.10:
     touch "^3.1.0"
     undefsafe "^2.0.5"
 
-nopt@~2.1.1:
-  version "2.1.2"
-  resolved "https://registry.yarnpkg.com/nopt/-/nopt-2.1.2.tgz#6cccd977b80132a07731d6e8ce58c2c8303cf9af"
-  integrity sha512-x8vXm7BZ2jE1Txrxh/hO74HTuYZQEbo8edoRcANgdZ4+PCV+pbjd/xdummkmjjC7LU5EjPzlu8zEq/oxWylnKA==
-  dependencies:
-    abbrev "1"
-
 normalize-path@^3.0.0, normalize-path@~3.0.0:
   version "3.0.0"
   resolved "https://registry.yarnpkg.com/normalize-path/-/normalize-path-3.0.0.tgz#0dcd69ff23a1c9b11fd0978316644a0388216a65"