Malin/krawl.es
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
93 Commits 1 Branch 0 Tags
554bd486daec0ff74ed9f07bea739b46db56a783
Commit Graph

93 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Patrick Di Fazio
554bd486da Merge pull request #39 from luismco/logo 
Adds SVG logo
 2026-01-14 13:51:22 +01:00
Luís Oliveira
f5e403a5f4 Rename krawl.svg to krawl-svg.svg 2026-01-14 12:19:30 +00:00
Luís Oliveira
d39d33c337 Adds SVG logo 2026-01-14 12:19:01 +00:00
Patrick Di Fazio
323aa0d15d Merge pull request #36 from BlessedRebuS/bug/db-locked-fix 
Bug/db locked fix
 2026-01-10 22:31:03 +01:00
Phillip Tarrant
72a213387e fixing requirements file to versioned pull for request 2026-01-10 15:17:16 -06:00
Phillip Tarrant
66d970ef92 Merge pull request #35 from BlessedRebuS/fix/requirements 
added requests
 2026-01-10 15:15:22 -06:00
Phillip Tarrant
0e0639ba8b adding requests to requirements 2026-01-10 15:14:17 -06:00
Phillip Tarrant
f7b9ee54e3 Fix SQLite "database is locked" errors in fetch_ip_rep task 
- Add missing session.commit() and cleanup to update_ip_rep_infos() **bugfix**
  - Enable SQLite WAL mode and 30s busy timeout for better concurrency **race condition prevention**
  - Add get_unenriched_ips() method to only fetch IPs needing enrichment **don't enrich what's already done*
  - Rewrite fetch_ip_rep task to process only unenriched IPs (limit 50) **API kindness**
  - Change task frequency from every 1 minute to every 5 minutes **API kindness**
  - Add request timeout (10s) and proper error handling **can update to longer if needed**
 2026-01-10 14:59:15 -06:00
Patrick Di Fazio
5d25fb9d6b added requests 2026-01-10 21:37:16 +01:00
Phillip Tarrant
edaafb0263 Merge pull request #34 from BlessedRebuS/feat/scoring-alogorithm 
Feat/scoring alogorithm
 2026-01-10 13:30:18 -06:00
Patrick Di Fazio
c65dca850c restored port 2026-01-10 20:06:33 +01:00
Patrick Di Fazio
c2dbcf588c added iprep to the dashboard, fixed bugs 2026-01-10 20:00:33 +01:00
Leonardo Bambini
77196952d1 made ip analysis and ip rep info fetch a scheduled task 2026-01-10 14:53:31 +01:00
Patrick Di Fazio
5a57c0774f Merge pull request #30 from BlessedRebuS/feat/background-tasks 
Feat/background tasks
 2026-01-09 20:39:38 +01:00
Patrick Di Fazio
431a42aa0a removed exports 2026-01-09 20:39:25 +01:00
Patrick Di Fazio
be7ba1f820 added download button 2026-01-09 20:37:20 +01:00
Phillip Tarrant
0b276b1a47 Merge branch 'dev' into feat/background-tasks 2026-01-09 09:39:56 -06:00
Phillip Tarrant
e1c49f1805 Merge pull request #31 from BlessedRebuS/fix/scoring-algorithm 
fixed categorization visualization, fixed date in the dashboard, fixe…
 2026-01-08 13:41:54 -06:00
Patrick Di Fazio
b61461d028 fixed categorization visualization, fixed date in the dashboard, fixed attack regex detection 2026-01-08 19:20:22 +01:00
Leonardo Bambini
4f42b946f3 added ip rep fetch + bug fix 2026-01-07 22:56:01 +01:00
Phillip Tarrant
edb288a271 Fixed some print statements to leverage logging, pulled in most recent dev edits, added exports to gitignore 2026-01-07 12:33:43 -06:00
Phillip Tarrant
5fe1984365 Merge branch 'dev' into feat/background-tasks 2026-01-07 11:51:27 -06:00
Patrick Di Fazio
e8a79ef970 Merge pull request #29 from BlessedRebuS/feat/scoring-alogorithm 
Feat/scoring alogorithm
 2026-01-07 18:50:44 +01:00
Phillip Tarrant
b6c3710ca3 Merge branch 'dev' into feat/scoring-alogorithm 2026-01-07 11:43:58 -06:00
Patrick Di Fazio
7690841029 added categorization visualization and itmeline 2026-01-07 18:24:43 +01:00
Patrick Di Fazio
02aed9e65a added drop down menu and scoring graph to the dashboard 2026-01-06 18:50:36 +01:00
Phillip Tarrant
7d9f0616b7 Add background task to export suspicious IPs to text file 
- Implement export-malicious-ips task that queries distinct IPs flagged
    as is_suspicious from database and writes to exports/malicious_ips.txt
  - Add exports volume mount to docker-compose.yaml for host persistence
  - Update entrypoint.sh to fix ownership of exports directory for krawl user
  - Update Dockerfile to create /app/exports directory during build

Other tasks can be added by creating them in the tasks dir using the same setup as this task.
All tasks *MUST* include a TASK_CONFIG dict and a main method in the file to work correctly.
 2026-01-05 11:54:02 -06:00
Patrick Di Fazio
190d74e1a7 modified krawl template for single page visualization 2026-01-05 17:27:27 +01:00
Patrick Di Fazio
4478c60956 added krawl homepage to templates 2026-01-05 17:07:10 +01:00
Patrick Di Fazio
bd8c326918 tuned weights 2026-01-05 16:54:43 +01:00
Leonardo Bambini
bf02fdb351 modified default analyzer values 2026-01-05 10:01:51 +01:00
Leonardo Bambini
ff98a77e1a parametrized into config.yaml + bug fix 2026-01-04 22:20:10 +01:00
Leonardo Bambini
48f38cb28e added scoring system + db model modifications 2026-01-04 19:12:23 +01:00
Patrick Di Fazio
604eba6c82 Merge pull request #27 from BlessedRebuS/chore/fix-merge-conflicts 
Sync Main and Dev with All Feature Branches
 2026-01-04 17:10:57 +01:00
Phillip Tarrant
07efa5bd40 Merge branch 'feat/config-yaml' into chore/fix-merge-conflicts 2026-01-03 14:45:32 -06:00
Phillip Tarrant
cde279960a adding feat/sql 2026-01-03 14:42:14 -06:00
Phillip Tarrant
7bd9d5055c fixing merge issues with main 2026-01-03 14:40:31 -06:00
Phillip Tarrant
4d1fd5491d Merge pull request #26 from BlessedRebuS/feat/add-deception-features 
Feat/add deception features
 2026-01-03 14:01:02 -06:00
Phillip Tarrant
4c490e30cb fixing dashboard to ensure starts with forward slash, put back the server_header option to allow pinning 2026-01-03 13:56:16 -06:00
Patrick Di Fazio
4a1d1cf7be added random SQL errors, random server errors, XSS baits 2026-01-03 17:16:37 +01:00
Patrick Di Fazio
5f8bb73546 added random SQL errors, random server errors, XSS baits 2026-01-03 17:14:58 +01:00
Phillip Tarrant
349c149335 Add logs directory bind mount with entrypoint permission fix 
- Add ./logs:/app/logs volume mount to docker-compose.yaml for log access
  - Create entrypoint.sh script that fixes directory ownership at startup
  - Install gosu in Dockerfile for secure privilege dropping
  - Use ENTRYPOINT to run permission fix as root, then drop to krawl user

  This ensures bind-mounted directories have correct permissions even when
  Docker creates them as root on the host.
 2026-01-02 13:52:51 -06:00
Phillip Tarrant
d458eb471d Migrate configuration from environment variables to YAML file 
- Add YAML-based configuration loaded from config.yaml (CONFIG_LOCATION env var)
  - Add PyYAML dependency and install requirements in Dockerfile
  - Replace Config.from_env() with get_config() singleton pattern
  - Remove server_header from config (now randomized from wordlists only)
  - Update docker-compose.yaml to mount config.yaml read-only
  - Update Helm chart: restructure values.yaml, generate config.yaml in ConfigMap
  - Update Kubernetes manifests: ConfigMap now contains config.yaml, deployments mount it
  - Remove Helm secret.yaml (dashboard path now auto-generated in config.yaml)
 2026-01-02 13:39:54 -06:00
Phillip Tarrant
5a00e374e6 Merge pull request #21 from BlessedRebuS/fix/add-kubernetes-db-config 
added db config for kubernetes and helm
 2025-12-30 06:03:47 -06:00
Patrick Di Fazio
c55b1375ad added db config for kubernetes and helm 2025-12-30 12:12:42 +01:00
Patrick Di Fazio
bf73bc7e2c Update README with demo and dashboard information 
Removed old sections and reorganized demo and dashboard links.
 2025-12-30 00:59:36 +01:00
Patrick Di Fazio
5ba02d3d0c Update README.md 2025-12-30 00:59:31 +01:00
Patrick Di Fazio
5a808c330c Update README.md 2025-12-30 00:59:15 +01:00
Patrick Di Fazio
0b1e9537d2 Added demo 2025-12-30 00:59:15 +01:00
Patrick Di Fazio
852a15976f Added demo 2025-12-30 00:59:15 +01:00