From d3e8a8ab8b9fe7361319c540d71b07789e1f2ba9 Mon Sep 17 00:00:00 2001
From: fab <fabrizio.salmi@gmail.com>
Date: Sat, 21 Dec 2024 01:50:43 +0100
Subject: [PATCH] Update README.md

---
 README.md | 74 +++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 19 deletions(-)

diff --git a/README.md b/README.md
index 0d2a84b..66f18cf 100644
--- a/README.md
+++ b/README.md
@@ -1,12 +1,19 @@
-# 🔒 Patterns – Automated OWASP CRS for Caddy, Nginx, and Apache  
+# 🔒 Patterns – Automated OWASP CRS and Bad Bot Detection for Caddy, Nginx, and Apache  
 Automate the scraping of **OWASP Core Rule Set (CRS)** patterns and convert them into **Caddy, Nginx, and Apache** WAF configurations.  
 
-> This project helps protect web servers against common web attacks like **SQL Injection (SQLi)**, **XSS**, **RCE**, **LFI**, and more – automatically updated and deployed!  
+Additionally, **Bad Bot/User-Agent detection** is integrated to block malicious web crawlers and scrapers.  
+
+> This project helps protect web servers against common web attacks like **SQL Injection (SQLi)**, **XSS**, **RCE**, **LFI**, and malicious bots – automatically updated and deployed!  
+
+---
 
 ## 🚀 Project Overview  
 - **🎯 Goal**: Automate OWASP CRS rule collection and generate WAF configs for **Caddy, Nginx, and Apache**.  
 - **⚡ Automation**: GitHub Actions fetch rules **daily** and push updated configurations to the repository.  
-- **📄 Output**: WAF `.conf` files categorized by attack type (SQLi, XSS, LFI) for all supported web servers.  
+- **🤖 Bad Bot Blocking**: Block harmful bots by generating WAF rules from **public bot lists**.  
+- **📄 Output**: WAF `.conf` files categorized by attack type (SQLi, XSS, LFI) and bot lists.  
+
+---
 
 ## 📂 Project Structure  
 ```
@@ -19,12 +26,15 @@ patterns/
 ├── owasp2caddy.py          # 🔄 Convert OWASP JSON to Caddy WAF configs
 ├── owasp2nginx.py          # 🔄 Convert OWASP JSON to Nginx WAF configs
 ├── owasp2apache.py         # 🔄 Convert OWASP JSON to Apache ModSecurity configs
+├── badbots.py              # 🔄 Generate WAF configs to block bad bots
 ├── owasp_rules.json        # 📊 Fetched OWASP rules (raw)
 ├── requirements.txt        # 🔄 Required tools
 └── .github/workflows/      # 🤖 GitHub Actions for automation
     └── update_patterns.yml
 ```
 
+---
+
 ## 🛠️ How It Works  
 ### 🔹 1. Scraping OWASP Rules  
 - `owasp.py` scrapes the latest OWASP CRS patterns from GitHub.  
@@ -35,9 +45,16 @@ patterns/
 - **`owasp2nginx.py`** – Converts OWASP patterns into Nginx WAF rules.  
 - **`owasp2apache.py`** – Converts OWASP rules into Apache **ModSecurity** configurations.  
 
-### 🔹 3. Automation (GitHub Actions)  
-- GitHub Actions fetch new rules **daily at midnight**.  
-- Automatically commits and pushes new `.conf` files for all three platforms (Caddy, Nginx, Apache).  
+### 🔹 3. Bad Bot/User-Agent Detection  
+- `badbots.py` fetches **public bot lists** to block malicious crawlers.  
+- Fallback lists ensure bot detection works even if the main source fails.  
+- Generates `.conf` files for Caddy, Nginx, and Apache.  
+
+### 🔹 4. Automation (GitHub Actions)  
+- GitHub Actions fetch new rules and bot lists **daily at midnight**.  
+- Automatically commits and pushes updated `.conf` files for all three platforms (Caddy, Nginx, Apache).  
+
+---
 
 ## ⚙️ Installation  
 **1. Clone the Repository:**  
@@ -57,8 +74,11 @@ python owasp.py
 python owasp2caddy.py
 python owasp2nginx.py
 python owasp2apache.py
+python badbots.py
 ```
 
+---
+
 ## 🚀 Usage (Web Server Integration)  
 ### 🔹 1. Caddy WAF Integration  
 ```bash
@@ -73,6 +93,8 @@ Reload Caddy:
 caddy reload
 ```
 
+---
+
 ### 🔹 2. Nginx WAF Integration  
 ```bash
 sudo cp waf_patterns/nginx/*.conf /etc/nginx/waf/
@@ -91,6 +113,8 @@ Reload Nginx:
 sudo nginx -s reload
 ```
 
+---
+
 ### 🔹 3. Apache ModSecurity Integration  
 ```bash
 sudo cp waf_patterns/apache/*.conf /etc/modsecurity.d/
@@ -104,25 +128,31 @@ Restart Apache:
 sudo systemctl restart apache2
 ```
 
+---
+
+## 🧩 Example Output (Bot Blocker – Nginx WAF)  
+**Bot Blocking (waf_patterns/nginx/bots.conf)**:  
+```nginx
+map $http_user_agent $bad_bot {
+    "~*AhrefsBot" 1;
+    "~*SemrushBot" 1;
+    "~*MJ12bot" 1;
+    default 0;
+}
+if ($bad_bot) {
+    return 403;
+}
+```
+
+---
+
 ## 🤖 Automation (GitHub Workflow)  
 The GitHub Action (`.github/workflows/update_patterns.yml`) automates updates:  
 - 🕛 **Runs Daily at Midnight (UTC)**  
 - 🎯 **Manual Trigger Available** (from GitHub Actions tab)  
 - 🚀 **Pushes Updated WAF Files** to `waf_patterns/`  
 
-To enable:  
-- Ensure the workflow is active in your repository.  
-- Updated patterns will automatically sync to the repo and reflect in your WAF setup.  
-
-
-## 🧩 Example Output (ModSecurity – Apache WAF)  
-**SQL Injection Blocking (waf_patterns/apache/sql.conf)**:  
-```apache
-SecRuleEngine On
-
-SecRule REQUEST_URI "union.*select|insert.*into|delete.*from|drop table" "id:1000,phase:1,deny,status:403,log,msg:'SQLi attack detected'"
-SecRule REQUEST_URI "alter table|truncate.*|--" "id:1001,phase:1,deny,status:403,log,msg:'SQLi attack detected'"
-```
+---
 
 ## 🔧 Contributing  
 1. Fork the repository.  
@@ -130,10 +160,14 @@ SecRule REQUEST_URI "alter table|truncate.*|--" "id:1001,phase:1,deny,status:403
 3. Commit and push changes.  
 4. Open a pull request (PR).  
 
+---
+
 ## 📄 License  
 This project is licensed under the **MIT License**.  
 See the [LICENSE](LICENSE) file for details.  
 
+---
+
 ## 🌐 Resources  
 - [OWASP CRS GitHub](https://github.com/coreruleset/coreruleset)  
 - [Caddy Web Server](https://caddyserver.com/)  
@@ -141,5 +175,7 @@ See the [LICENSE](LICENSE) file for details.
 - [Apache ModSecurity](https://modsecurity.org/)  
 - [MIT License](https://opensource.org/licenses/MIT)  
 
+---
+
 ## 🚨 Issues  
 If you encounter any issues, please open a ticket in the [Issues Tab](https://github.com/your-username/patterns/issues).