Commit Graph

44 Commits

Author SHA1 Message Date
Lorenzo Venerandi
1eb4f54f5c feat: implement in-memory caching for dashboard data and background warmup task 2026-03-10 11:00:22 +01:00
Lorenzo Venerandi
a9aeb00279 feat: add ban_override column to ip_stats and update attacker IP filtering logic 2026-03-08 12:26:13 +01:00
Lorenzo Venerandi
6dc9cfe502 feat: enhance database retention logic to preserve suspicious access logs and linked IPs 2026-03-03 19:16:27 +01:00
Lorenzo Venerandi
36df289a5e feat: enhance database retention logic to use configurable retention period and delete stale IPs 2026-03-03 19:03:40 +01:00
Lorenzo Venerandi
e1c1b4d953 chore: update Dockerfile and requirements for Python 3.13, enhance error handling in config and tracker modules 2026-03-01 21:52:27 +01:00
BlessedRebuS
049e70972b fixed user different_user_agents statement 2026-03-01 18:56:07 +01:00
Lorenzo Venerandi
214c83a852 fix: Adjust different user agents count for attacker and good crawler scores 2026-03-01 17:45:46 +01:00
Lorenzo Venerandi
43d3b96364 fix: Improve stale IP flagging logic to support forced rescan 2026-03-01 17:36:15 +01:00
Lorenzo Venerandi
ef467b0fd6 fix: enable task to run when loaded for flag-stale-ips 2026-03-01 16:58:10 +01:00
Lorenzo Venerandi
4d4d93ad80 linted code 2026-02-22 18:45:56 +01:00
Lorenzo Venerandi
b559fd6a84 feat: add task to flag stale IPs for reevaluation 2026-02-22 18:41:50 +01:00
Lorenzo Venerandi
9a271cbe68 feat: enhance IP analysis by adding robots.txt parsing and optimizing HTTP method scoring 2026-02-22 18:34:22 +01:00
Lorenzo Venerandi
b500b22041 code linting 2026-02-22 16:27:59 +01:00
Lorenzo Venerandi
8ce8b6b40a feat: implement IP ban and rate-limiting logic in database with migration scripts 2026-02-22 16:23:52 +01:00
Lorenzo Venerandi
db848e7ecb feat: add need_reevaluation column to IpStats and update related logic 2026-02-22 16:05:39 +01:00
carnivuth
d8220b2429 added parameter in config file to disable backup job 2026-02-22 16:01:39 +01:00
Lorenzo Venerandi
152682de2c code linted 2026-02-18 00:02:44 +01:00
Lorenzo Venerandi
846fba631f refactor: streamline memory management by removing unused variables and enhancing cleanup logic 2026-02-17 18:09:35 +01:00
Lorenzo Venerandi
b94cd38b61 refactor: optimize database access and implement retention task 2026-02-17 18:04:09 +01:00
Lorenzo Venerandi
507e3f2667 refactor: remove unused tracking variables and streamline memory stats logging 2026-02-17 17:59:03 +01:00
Lorenzo Venerandi
03491d55cc refactor: enhance tracker integration and memory cleanup logic 2026-02-17 15:17:54 +01:00
carnivuth
396b9b1710 linted code 2026-02-15 15:10:41 +01:00
carnivuth
815da4300b fixed bug on non existent database dump directory 2026-02-15 15:10:27 +01:00
Patrick Di Fazio
771174c6a9 added raw request handling, enanched attack detection for GET and POSTS, templatized suspicioius activity to fetch from wordlists.json, aligned helm to load new wordlist config, added migration scripts from 1.0.0 to new krawl versions, removed old and unused functions, added test scripts 2026-02-08 16:02:18 +01:00
carnivuth
789d77c784 linted code 2026-02-05 17:57:38 +01:00
carnivuth
7621932602 added task dump krawl data and adjusted configuration files 2026-02-05 17:26:06 +01:00
BlessedRebuS
01547dbf8f modified doc 2026-02-02 23:01:25 +01:00
Patrick Di Fazio
3cdb3a360b Merge branch 'dev' into feat/blocklist-api 2026-02-02 22:38:44 +01:00
carnivuth
35c8a90013 addd exports path configuration variable and default values, restored old filename 2026-02-02 14:19:40 +01:00
BlessedRebuS
863fac251d modified dashboard, added ip-api data fetch 2026-02-01 22:43:12 +01:00
carnivuth
09b986f1b0 changed workflow from live computation to scheduled job to support previous structure 2026-01-30 16:13:45 +01:00
Lorenzo Venerandi
d3caa99ecc Fix/export malicious ip file creation (#62)
* Fix: update EXPORTS_DIR path to ensure correct directory structure

* Fix: remove unused imports and honeypot access check from export task
2026-01-29 13:54:07 +01:00
Lorenzo Venerandi
a778262701 Fix: update EXPORTS_DIR path to ensure correct directory structure (#61) 2026-01-29 12:52:48 +01:00
Lorenzo Venerandi
e93bcb959a Doc/updated documentation (#60)
* added documentation, updated repo pointer in the dashboard, added dashboard link highlighting and mionor fixes

* added doc

* added logo to dashboard

* Fixed dashboard attack chart

* Enhance fake data generation with varied request counts for better visualization

* Add automatic migrations and support for latitude/longitude in IP stats

* Update Helm chart version to 0.2.2 and add timezone configuration option

---------

Co-authored-by: BlessedRebuS <patrick.difa@gmail.com>
2026-01-29 11:55:06 +01:00
Lorenzo Venerandi
39d9d62247 Feat/attack map improvement (#58)
* Enhance geolocation functionality and improve unenriched IP retrieval logic

* Refactor test_insert_fake_ips.py to enhance geolocation data handling and improve IP data structure

* Refactor code for improved readability and consistency in database and geolocation utilities
2026-01-27 17:32:38 +01:00
Lorenzo Venerandi
5aca684df9 Feat/attack map improvement (#57)
* feat: enhance IP reputation management with city data and geolocation integration

* feat: enhance dashboard with city coordinates and improved marker handling

* feat: update chart version to 0.2.1 in Chart.yaml, README.md, and values.yaml

* feat: update logo format and size in README.md

* feat: improve location display logic in dashboard for attackers and IPs
2026-01-27 16:56:34 +01:00
Lorenzo Venerandi
8c76f6c847 Feat/deployment update (#56)
* feat: update analyzer thresholds and add crawl configuration options

* feat: update Helm chart version and add README for installation instructions

* feat: update installation instructions in README and add Docker support

* feat: update deployment manifests and configuration for improved service handling and analyzer settings

* feat: add API endpoint for paginated IP retrieval and enhance dashboard visualization with category filters

* feat: update configuration for Krawl service to use external config file

* feat: refactor code for improved readability and consistency across multiple files

* feat: remove Flake8, Pylint, and test steps from PR checks workflow
2026-01-26 12:36:22 +01:00
BlessedRebuS
14d616fae3 added ip logging memory improvements, added local ip and public ip exlusion 2026-01-24 23:28:10 +01:00
Lorenzo Venerandi
4450d3a4e3 Linted code iwht black tool 2026-01-23 22:00:21 +01:00
carnivuth
9d9a718aec feat:removed manual timezone management, delegate timezone configuration to execution environment
removed code that manages timezone setup from config file, krawl now
obeys to the environment configuration
2026-01-17 18:06:09 +01:00
Phillip Tarrant
541b5d0f1b Optimize scheduled tasks to reduce unnecessary processing
- Add conditional execution to export-malicious-ips task: only runs
    when honeypot was accessed in last 5 minutes
  - Add since_minutes parameter to get_access_logs() for time filtering
  - Optimize analyze-ips task to only process IPs with activity in the
    last minute, fetching full history per-IP instead of all logs
  - Exclude RFC1918 private addresses and non-routable IPs from IP
    reputation enrichment (10.x, 172.16-31.x, 192.168.x, 127.x, 169.254.x)
2026-01-15 13:30:35 -06:00
Phillip Tarrant
f7b9ee54e3 Fix SQLite "database is locked" errors in fetch_ip_rep task
- Add missing session.commit() and cleanup to update_ip_rep_infos() **bugfix**
  - Enable SQLite WAL mode and 30s busy timeout for better concurrency **race condition prevention**
  - Add get_unenriched_ips() method to only fetch IPs needing enrichment **don't enrich what's already done*
  - Rewrite fetch_ip_rep task to process only unenriched IPs (limit 50) **API kindness**
  - Change task frequency from every 1 minute to every 5 minutes **API kindness**
  - Add request timeout (10s) and proper error handling **can update to longer if needed**
2026-01-10 14:59:15 -06:00
Leonardo Bambini
77196952d1 made ip analysis and ip rep info fetch a scheduled task 2026-01-10 14:53:31 +01:00
Phillip Tarrant
7d9f0616b7 Add background task to export suspicious IPs to text file
- Implement export-malicious-ips task that queries distinct IPs flagged
    as is_suspicious from database and writes to exports/malicious_ips.txt
  - Add exports volume mount to docker-compose.yaml for host persistence
  - Update entrypoint.sh to fix ownership of exports directory for krawl user
  - Update Dockerfile to create /app/exports directory during build

Other tasks can be added by creating them in the tasks dir using the same setup as this task.
All tasks *MUST* include a TASK_CONFIG dict and a main method in the file to work correctly.
2026-01-05 11:54:02 -06:00