added iptables and nftables integration

2026-02-23 01:23:49 +01:00
parent 69e82fc030
commit 8c81dccc3b
6 changed files with 490 additions and 2 deletions
--- a/README.md
+++ b/README.md
@@ -148,8 +148,8 @@ curl https://your-krawl-instance/<DASHBOARD-PATH>/api/download/malicious_ips.txt
 This file enables automatic blocking of malicious traffic across various platforms. You can use it to update firewall rules on:
 * [OPNsense and pfSense](https://www.allthingstech.ch/using-opnsense-and-ip-blocklists-to-block-malicious-traffic)
 * [RouterOS](https://rentry.co/krawl-routeros)
-* IPtables
-* [Fail2Ban](plugins/fail2ban/fail2ban.md)
+* [IPtables](plugins/iptables/README.md) and [Nftables](plugins/nftables/README.md)
+* [Fail2Ban](plugins/fail2ban/README.md)

 ## IP Reputation
 Krawl [uses tasks that analyze recent traffic to build and continuously update an IP reputation](src/tasks/analyze_ips.py) score. It runs periodically and evaluates each active IP address based on multiple behavioral indicators to classify it as an attacker, crawler, or regular user. Thresholds are fully customizable.
--- a/plugins/fail2ban/fail2ban.md
+++ b/plugins/fail2ban/fail2ban.md
--- a/plugins/iptables/README.md
+++ b/plugins/iptables/README.md
@@ -0,0 +1,302 @@
+# Iptables + Krawl Integration
+
+## Overview
+
+This guide explains how to integrate **iptables** with Krawl to automatically block detected malicious IPs at the firewall level. The iptables integration fetches the malicious IP list directly from Krawl's API and applies firewall rules.
+
+## Architecture
+
+```
+Krawl detects malicious IPs
+         ↓
+Stores in database
+         ↓
+API endpoint
+         ↓
+Cron job fetches list
+         ↓
+Iptables firewall blocks IPs
+         ↓
+All traffic from banned IPs dropped
+```
+
+## Prerequisites
+
+- Linux system with iptables installed (typically pre-installed)
+- Krawl running with API accessible
+- Root/sudo access
+- Curl or wget for HTTP requests
+- Cron for scheduling (or systemd timer as alternative)
+
+## Installation & Setup
+
+### 1. Create the [krawl-iptables.sh](krawl-iptables.sh) script
+
+```bash
+#!/bin/bash
+
+curl -s https://your-krawl-instance/your-dashboard-path/api/get_banlist?fwtype=iptables | while read ip; do 
+  iptables -C INPUT -s "$ip" -j DROP || iptables -A INPUT -s "$ip" -j DROP; 
+done
+```
+
+Make it executable:
+```bash
+sudo chmod +x ./krawl-iptables.sh
+```
+
+### 2. Test the Script
+
+```bash
+sudo ./krawl-iptables.sh
+```
+
+### 3. Schedule with Cron
+
+Edit root crontab:
+```bash
+sudo crontab -e
+```
+
+Add this line to sync IPs every hour:
+
+```bash
+0 * * * * /path/to/krawl-iptables.sh
+```
+
+## How It Works
+
+### When the Script Runs
+
+1. **Fetch IPs** from Krawl API (`/api/get_banlist?fwtype=iptables`)
+2. **Add new DROP rules** for each IP
+3. **Rules are applied immediately** at kernel level
+
+## Monitoring
+
+### Check Active Rules
+
+View all KRAWL-BAN rules:
+```bash
+sudo iptables -L KRAWL-BAN -n
+```
+
+Count blocked IPs:
+```bash
+sudo iptables -L KRAWL-BAN -n | tail -n +3 | wc -l
+```
+
+### Check Script Logs
+
+```bash
+sudo tail -f /var/log/krawl-iptables-sync.log
+```
+
+### Monitor in Real-Time
+
+Watch dropped packets (requires kernel logging):
+```bash
+sudo tail -f /var/log/syslog | grep "IN=.*OUT="
+```
+
+## Management Commands
+
+### Manually Block an IP
+
+```bash
+sudo iptables -A KRAWL-BAN -s 192.168.1.100 -j DROP
+```
+
+### Manually Unblock an IP
+
+```bash
+sudo iptables -D KRAWL-BAN -s 192.168.1.100 -j DROP
+```
+
+### List All Blocked IPs
+
+```bash
+sudo iptables -L KRAWL-BAN -n | grep DROP
+```
+
+### Clear All Rules
+
+```bash
+sudo iptables -F KRAWL-BAN
+```
+
+### Disable the Chain (Temporarily)
+
+```bash
+sudo iptables -D INPUT -j KRAWL-BAN
+```
+
+### Re-enable the Chain
+
+```bash
+sudo iptables -I INPUT -j KRAWL-BAN
+```
+
+### View Statistics
+
+```bash
+sudo iptables -L KRAWL-BAN -n -v
+```
+
+## Persistent Rules (Survive Reboot)
+
+### Save Current Rules
+
+```bash
+sudo iptables-save > /etc/iptables/rules.v4
+```
+
+### Restore on Boot
+
+Install iptables-persistent:
+```bash
+sudo apt-get install iptables-persistent
+```
+
+During installation, choose "Yes" to save current IPv4 and IPv6 rules.
+
+To update later:
+```bash
+sudo iptables-save > /etc/iptables/rules.v4
+sudo systemctl restart iptables
+```
+
+## Performance Considerations
+
+### For Your Setup
+
+- **Minimal overhead** — Iptables rules are processed at kernel level (very fast)
+- **No logging I/O** — Blocked IPs are dropped before application sees them
+- **Scales to thousands** — Iptables can efficiently handle 10,000+ rules
+
+### Optimization Tips
+
+1. **Use a custom chain** — Isolates Krawl rules from other firewall rules
+2. **Schedule appropriately** — Every hour is usually sufficient; adjust based on threat level
+3. **Monitor rule count** — Check periodically to ensure the script is working
+4. **Consider IPSET** — For 10,000+ IPs, use ipset instead (more efficient)
+
+### Using IPSET (Advanced)
+
+For large-scale deployments, ipset is more efficient than individual iptables rules:
+
+```bash
+# Create ipset
+sudo ipset create krawl-ban hash:ip
+
+# Add IPs to ipset
+while read ip; do
+  sudo ipset add krawl-ban "$ip"
+done
+
+# Single iptables rule references the ipset
+sudo iptables -I INPUT -m set --match-set krawl-ban src -j DROP
+```
+
+## Troubleshooting
+
+### Script Says "Failed to fetch IP list"
+
+Check API connectivity:
+```bash
+curl http://your-krawl-instance/api/get_banlist?fwtype=iptables
+```
+
+Verify:
+- Krawl is running
+- API URL is correct
+- Firewall allows outbound HTTPS/HTTP
+- No authentication required
+
+### Iptables Rules Not Persisting After Reboot
+
+Install and configure iptables-persistent:
+```bash
+sudo apt-get install iptables-persistent
+sudo iptables-save > /etc/iptables/rules.v4
+```
+
+### Script Runs but No Rules Added
+
+Check if chain exists:
+```bash
+sudo iptables -L KRAWL-BAN -n 2>&1 | head -1
+```
+
+Check logs for errors:
+```bash
+sudo grep ERROR /var/log/krawl-iptables-sync.log
+```
+
+Verify IP format in Krawl API response:
+```bash
+curl http://your-krawl-instance/api/get_banlist?fwtype=iptables | head -10
+```
+
+### Blocked Legitimate Traffic
+
+Check what IPs are blocked:
+```bash
+sudo iptables -L KRAWL-BAN -n | grep -E [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}
+```
+
+Unblock an IP:
+```bash
+sudo iptables -D KRAWL-BAN -s 203.0.113.50 -j DROP
+```
+
+Report false positive to Krawl administrators.
+
+## Security Best Practices
+
+1. **Limit API access** — Restrict `/api/get_banlist` to trusted networks (if internal use)
+2. **Use HTTPS** — Fetch from HTTPS endpoint if available
+3. **Verify TLS certificates** — Add `-k` only if necessary, not by default
+4. **Rate limit cron jobs** — Don't run too frequently to avoid DoS
+5. **Monitor sync logs** — Alert on repeated failures
+6. **Backup rules** — Periodically backup `/etc/iptables/rules.v4`
+
+## Integration with Krawl Workflow
+
+Combined with fail2ban and iptables:
+
+```
+Real-time events (fail2ban)
+         ↓
+Immediate IP bans (temporary)
+         ↓
+Hourly sync (iptables cron)
+         ↓
+Permanent block until next rotation
+         ↓
+30-day cleanup cycle
+```
+
+## Manual Integration Example
+
+Instead of cron, manually fetch and block:
+
+```bash
+# Fetch malicious IPs
+curl -s http://your-krawl-instance/api/get_banlist?fwtype=iptables > /tmp/malicious_ips.txt
+
+# Read and block each IP
+while read ip; do
+  sudo iptables -A KRAWL-BAN -s "$ip" -j DROP
+done < /tmp/malicious_ips.txt
+
+# Save rules
+sudo iptables-save > /etc/iptables/rules.v4
+```
+
+## References
+
+- [Iptables Man Page](https://linux.die.net/man/8/iptables)
+- [Iptables Essentials](https://www.digitalocean.com/community/tutorials/iptables-essentials-common-firewall-rules-and-commands)
+- [Ipset Documentation](https://ipset.netfilter.org/)
+- [Linux Firewall Administration Guide](https://www.kernel.org/doc/html/latest/networking/nf_conntrack-sysctl.html)
--- a/plugins/iptables/krawl-iptables.sh
+++ b/plugins/iptables/krawl-iptables.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+# This script fetches a list of malicious IPs from your Krawl instance and adds them to the iptables firewall to block incoming traffic from those IPs.
+
+curl -s https://your-krawl-instance/api/get_banlist?fwtype=iptables | while read ip; do 
+  iptables -C INPUT -s "$ip" -j DROP || iptables -A INPUT -s "$ip" -j DROP; 
+done
--- a/plugins/nftables/README.md
+++ b/plugins/nftables/README.md
@@ -0,0 +1,161 @@
+# Nftables + Krawl Integration
+
+## Overview
+
+This guide explains how to integrate **nftables** with Krawl to automatically block detected malicious IPs at the firewall level. Nftables is the modern replacement for iptables on newer Linux systems and provides more efficient IP set-based blocking.
+
+## Architecture
+
+```
+Krawl detects malicious IPs
+         ↓
+Stores in database
+         ↓
+API endpoint
+         ↓
+Cron job fetches list
+         ↓
+Nftables firewall blocks IPs
+         ↓
+All traffic from banned IPs dropped
+```
+
+## Prerequisites
+
+- Modern Linux system with nftables installed (Ubuntu 22+, Debian 12+, RHEL 9+)
+- Krawl running with API accessible
+- Root/sudo access
+- Curl for HTTP requests
+- Cron for scheduling
+
+## When to Use Nftables
+
+Check if your system uses nftables:
+
+```bash
+sudo nft list tables
+```
+
+If this shows tables, you're using nftables. If you get command not found, use iptables instead.
+
+## Installation & Setup
+
+### 1. Create the [krawl-nftables.sh](krawl-nftables.sh) script
+
+```bash
+#!/bin/bash
+
+
+curl -s https://your-krawl-instance/your-dashboard-path/api/get_banlist?fwtype=iptables > /tmp/ips_to_ban.txt
+
+
+sudo nft add set inet filter krawl_ban { type ipv4_addr \; } 2>/dev/null || true
+
+
+while read -r ip; do
+  [[ -z "$ip" ]] && continue
+  sudo nft add element inet filter krawl_ban { "$ip" }
+done < /tmp/ips_to_ban.txt
+
+
+sudo nft add rule inet filter input ip saddr @krawl_ban counter drop 2>/dev/null || true
+
+
+rm -f /tmp/ips_to_ban.txt
+```
+
+Make it executable:
+```bash
+sudo chmod +x ./krawl-nftables.sh
+```
+
+### 2. Test the Script
+
+```bash
+sudo ./krawl-nftables.sh
+```
+
+### 3. Schedule with Cron
+
+Edit root crontab:
+```bash
+sudo crontab -e
+```
+
+Add this line to sync IPs every hour:
+
+```bash
+0 * * * * /path/to/krawl-nftables.sh
+```
+
+## Monitoring
+
+### Check the Blocking Set
+
+View blocked IPs:
+```bash
+sudo nft list set inet filter krawl_ban
+```
+
+Count blocked IPs:
+```bash
+sudo nft list set inet filter krawl_ban | grep "elements" | wc -w
+```
+
+### Check Active Rules
+
+View all rules in the filter table:
+```bash
+sudo nft list table inet filter
+```
+
+Find Krawl-specific rules:
+```bash
+sudo nft list chain inet filter input | grep krawl_ban
+```
+
+### Monitor in Real-Time
+
+Watch packets being dropped:
+```bash
+sudo nft list set inet filter krawl_ban -a
+```
+
+## Management Commands
+
+### Manually Block an IP
+
+```bash
+sudo nft add element inet filter krawl_ban { 192.168.1.100 }
+```
+
+### Manually Unblock an IP
+
+```bash
+sudo nft delete element inet filter krawl_ban { 192.168.1.100 }
+```
+
+### List All Blocked IPs
+
+```bash
+sudo nft list set inet filter krawl_ban
+```
+
+### Clear All Blocked IPs
+
+```bash
+sudo nft flush set inet filter krawl_ban
+```
+
+### Delete the Rule
+
+```bash
+sudo nft delete rule inet filter input handle <handle>
+```
+
+## References
+
+- [Nftables Official Documentation](https://wiki.nftables.org/)
+- [Nftables Quick Reference](https://wiki.nftables.org/wiki-nftables/index.php/Quick_reference-nftables_in_10_minutes)
+- [Linux Kernel Netfilter Guide](https://www.kernel.org/doc/html/latest/networking/netfilter/)
+- [Nftables Man Page](https://man.archlinux.org/man/nft.8)
--- a/plugins/nftables/krawl-nftables.sh
+++ b/plugins/nftables/krawl-nftables.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# Fetch malicious IPs to temporary file
+curl -s https://your-krawl-instance/your-dashboard-path/api/get_banlist?fwtype=iptables > /tmp/ips_to_ban.txt
+
+# Create the set if it doesn't exist
+sudo nft add set inet filter krawl_ban { type ipv4_addr \; } 2>/dev/null || true
+
+# Add IPs to the set
+while read -r ip; do
+  [[ -z "$ip" ]] && continue
+  sudo nft add element inet filter krawl_ban { "$ip" }
+done < /tmp/ips_to_ban.txt
+
+# Create the rule if it doesn't exist
+sudo nft add rule inet filter input ip saddr @krawl_ban counter drop 2>/dev/null || true
+
+# Cleanup
+rm -f /tmp/ips_to_ban.txt