Update README.md

2025-12-17 09:45:34 +00:00 · 2024-12-21 01:50:43 +01:00 · 2024-12-21 01:50:43 +01:00 · d3e8a8ab8b
commit d3e8a8ab8b
parent 8d55569a01
1 changed files with 55 additions and 19 deletions
--- a/README.md
+++ b/README.md
@ -1,12 +1,19 @@
-# 🔒 Patterns – Automated OWASP CRS for Caddy, Nginx, and Apache  
+# 🔒 Patterns – Automated OWASP CRS and Bad Bot Detection for Caddy, Nginx, and Apache  
 Automate the scraping of **OWASP Core Rule Set (CRS)** patterns and convert them into **Caddy, Nginx, and Apache** WAF configurations.  

-> This project helps protect web servers against common web attacks like **SQL Injection (SQLi)**, **XSS**, **RCE**, **LFI**, and more – automatically updated and deployed!  
+Additionally, **Bad Bot/User-Agent detection** is integrated to block malicious web crawlers and scrapers.  
+
+> This project helps protect web servers against common web attacks like **SQL Injection (SQLi)**, **XSS**, **RCE**, **LFI**, and malicious bots – automatically updated and deployed!  
+
+---

 ## 🚀 Project Overview  
 - **🎯 Goal**: Automate OWASP CRS rule collection and generate WAF configs for **Caddy, Nginx, and Apache**.  
 - **⚡ Automation**: GitHub Actions fetch rules **daily** and push updated configurations to the repository.  
- **📄 Output**: WAF `.conf` files categorized by attack type (SQLi, XSS, LFI) for all supported web servers.  
+- **🤖 Bad Bot Blocking**: Block harmful bots by generating WAF rules from **public bot lists**.  
+- **📄 Output**: WAF `.conf` files categorized by attack type (SQLi, XSS, LFI) and bot lists.  
+
+---

 ## 📂 Project Structure  
 ```
@ -19,12 +26,15 @@ patterns/
 ├── owasp2caddy.py          # 🔄 Convert OWASP JSON to Caddy WAF configs
 ├── owasp2nginx.py          # 🔄 Convert OWASP JSON to Nginx WAF configs
 ├── owasp2apache.py         # 🔄 Convert OWASP JSON to Apache ModSecurity configs
+├── badbots.py              # 🔄 Generate WAF configs to block bad bots
 ├── owasp_rules.json        # 📊 Fetched OWASP rules (raw)
 ├── requirements.txt        # 🔄 Required tools
 └── .github/workflows/      # 🤖 GitHub Actions for automation
    └── update_patterns.yml
 ```

+---
+
 ## 🛠️ How It Works  
 ### 🔹 1. Scraping OWASP Rules  
 - `owasp.py` scrapes the latest OWASP CRS patterns from GitHub.  
@ -35,9 +45,16 @@ patterns/
 - **`owasp2nginx.py`** – Converts OWASP patterns into Nginx WAF rules.  
 - **`owasp2apache.py`** – Converts OWASP rules into Apache **ModSecurity** configurations.  

-### 🔹 3. Automation (GitHub Actions)  
- GitHub Actions fetch new rules **daily at midnight**.  
- Automatically commits and pushes new `.conf` files for all three platforms (Caddy, Nginx, Apache).  
+### 🔹 3. Bad Bot/User-Agent Detection  
+- `badbots.py` fetches **public bot lists** to block malicious crawlers.  
+- Fallback lists ensure bot detection works even if the main source fails.  
+- Generates `.conf` files for Caddy, Nginx, and Apache.  
+
+### 🔹 4. Automation (GitHub Actions)  
+- GitHub Actions fetch new rules and bot lists **daily at midnight**.  
+- Automatically commits and pushes updated `.conf` files for all three platforms (Caddy, Nginx, Apache).  
+
+---

 ## ⚙️ Installation  
 **1. Clone the Repository:**  
@ -57,8 +74,11 @@ python owasp.py
 python owasp2caddy.py
 python owasp2nginx.py
 python owasp2apache.py
+python badbots.py
 ```

+---
+
 ## 🚀 Usage (Web Server Integration)  
 ### 🔹 1. Caddy WAF Integration  
 ```bash
@ -73,6 +93,8 @@ Reload Caddy:
 caddy reload
 ```

+---
+
 ### 🔹 2. Nginx WAF Integration  
 ```bash
 sudo cp waf_patterns/nginx/*.conf /etc/nginx/waf/
@ -91,6 +113,8 @@ Reload Nginx:
 sudo nginx -s reload
 ```

+---
+
 ### 🔹 3. Apache ModSecurity Integration  
 ```bash
 sudo cp waf_patterns/apache/*.conf /etc/modsecurity.d/
@ -104,25 +128,31 @@ Restart Apache:
 sudo systemctl restart apache2
 ```

+---
+
+## 🧩 Example Output (Bot Blocker – Nginx WAF)  
+**Bot Blocking (waf_patterns/nginx/bots.conf)**:  
+```nginx
+map $http_user_agent $bad_bot {
+    "~*AhrefsBot" 1;
+    "~*SemrushBot" 1;
+    "~*MJ12bot" 1;
+    default 0;
+}
+if ($bad_bot) {
+    return 403;
+}
+```
+
+---
+
 ## 🤖 Automation (GitHub Workflow)  
 The GitHub Action (`.github/workflows/update_patterns.yml`) automates updates:  
 - 🕛 **Runs Daily at Midnight (UTC)**  
 - 🎯 **Manual Trigger Available** (from GitHub Actions tab)  
 - 🚀 **Pushes Updated WAF Files** to `waf_patterns/`  

-To enable:  
- Ensure the workflow is active in your repository.  
- Updated patterns will automatically sync to the repo and reflect in your WAF setup.  
-
-
-## 🧩 Example Output (ModSecurity – Apache WAF)  
-**SQL Injection Blocking (waf_patterns/apache/sql.conf)**:  
-```apache
-SecRuleEngine On
-
-SecRule REQUEST_URI "union.*select|insert.*into|delete.*from|drop table" "id:1000,phase:1,deny,status:403,log,msg:'SQLi attack detected'"
-SecRule REQUEST_URI "alter table|truncate.*|--" "id:1001,phase:1,deny,status:403,log,msg:'SQLi attack detected'"
-```
+---

 ## 🔧 Contributing  
 1. Fork the repository.  
@ -130,10 +160,14 @@ SecRule REQUEST_URI "alter table|truncate.*|--" "id:1001,phase:1,deny,status:403
 3. Commit and push changes.  
 4. Open a pull request (PR).  

+---
+
 ## 📄 License  
 This project is licensed under the **MIT License**.  
 See the [LICENSE](LICENSE) file for details.  

+---
+
 ## 🌐 Resources  
 - [OWASP CRS GitHub](https://github.com/coreruleset/coreruleset)  
 - [Caddy Web Server](https://caddyserver.com/)  
@ -141,5 +175,7 @@ See the [LICENSE](LICENSE) file for details.
 - [Apache ModSecurity](https://modsecurity.org/)  
 - [MIT License](https://opensource.org/licenses/MIT)  

+---
+
 ## 🚨 Issues  
 If you encounter any issues, please open a ticket in the [Issues Tab](https://github.com/your-username/patterns/issues).