Patterns: OWASP CRS and Bad Bot Detection rules for web servers

Automate the scraping of OWASP Core Rule Set (CRS) patterns and convert them into Apache, Nginx, Caddy, Traefik and HaProxy WAF configurations. Additionally, Bad Bot/User-Agent detection is integrated to block malicious web crawlers and scrapers.

Note

This project helps protect web servers against common web attacks like SQL Injection (SQLi), XSS, RCE, LFI, and malicious bots – automatically updated and deployed!

---

🚀 Project Overview

• 🎯 Goal: Automate OWASP CRS rule collection and generate WAF configs for Caddy, Nginx, and Apache.
• ⚡ Automation: GitHub Actions fetch rules daily and push updated configurations to the repository.
• 🤖 Bad Bot Blocking: Block harmful bots by generating WAF rules from public bot lists.
• 📄 Output: WAF .conf files categorized by attack type (SQLi, XSS, LFI) and bot lists.

---

📂 Project Structure

patterns/
├── waf_patterns/           # 🔧 Generated WAF config files
│   ├── caddy/              # Caddy WAF configs
│   ├── nginx/              # Nginx WAF configs
│   └── apache/             # Apache WAF configs (ModSecurity)
├── owasp.py                # 🕵️ OWASP scraper (fetch CRS rules)
├── owasp2caddy.py          # 🔄 Convert OWASP JSON to Caddy WAF configs
├── owasp2nginx.py          # 🔄 Convert OWASP JSON to Nginx WAF configs
├── owasp2apache.py         # 🔄 Convert OWASP JSON to Apache ModSecurity configs
├── badbots.py              # 🔄 Generate WAF configs to block bad bots
├── owasp_rules.json        # 📊 Fetched OWASP rules (raw)
├── requirements.txt        # 🔄 Required tools
└── .github/workflows/      # 🤖 GitHub Actions for automation
    └── update_patterns.yml

---

🛠️ How It Works

🔹 1. Scraping OWASP Rules

• owasp.py scrapes the latest OWASP CRS patterns from GitHub.
• Pulls attack patterns for SQLi, XSS, RCE, LFI from OWASP CRS .conf files.

🔹 2. Multi-Platform WAF Config Generation

• owasp2caddy.py – Generates Caddy WAF configs using OWASP patterns.
• owasp2nginx.py – Converts OWASP patterns into Nginx WAF rules.
• owasp2apache.py – Converts OWASP rules into Apache ModSecurity configurations.

🔹 3. Bad Bot/User-Agent Detection

• badbots.py fetches public bot lists to block malicious crawlers.
• Fallback lists ensure bot detection works even if the main source fails.
• Generates .conf files for Caddy, Nginx, and Apache.

🔹 4. Automation (GitHub Actions)

• GitHub Actions fetch new rules and bot lists daily at midnight.
• Automatically commits and pushes updated .conf files for all three platforms (Caddy, Nginx, Apache).

---

⚙️ Installation

1. Clone the Repository:

git clone https://github.com/your-username/patterns.git  
cd patterns

2. Install Dependencies:

pip install -r requirements.txt

3. Run Manually (Optional):

python owasp.py
python owasp2caddy.py
python owasp2nginx.py
python owasp2apache.py
python badbots.py

---

🚀 Usage (Web Server Integration)

🔹 1. Caddy WAF Integration

sudo cp waf_patterns/caddy/*.conf /etc/caddy/

Add to Caddyfile:

import waf_patterns/caddy/*.conf

Reload Caddy:

caddy reload

---

🔹 2. Nginx WAF Integration

sudo cp waf_patterns/nginx/*.conf /etc/nginx/waf/

Modify Nginx config:

server {
    include /etc/nginx/waf/*.conf;
    location / {
        # Other configs...
    }
}

Reload Nginx:

sudo nginx -s reload

---

🔹 3. Apache ModSecurity Integration

sudo cp waf_patterns/apache/*.conf /etc/modsecurity.d/

Add to Apache config:

IncludeOptional /etc/modsecurity.d/*.conf

Restart Apache:

sudo systemctl restart apache2

---

🧩 Example Output (Bot Blocker – Nginx WAF)

Bot Blocking (waf_patterns/nginx/bots.conf):

map $http_user_agent $bad_bot {
    "~*AhrefsBot" 1;
    "~*SemrushBot" 1;
    "~*MJ12bot" 1;
    default 0;
}
if ($bad_bot) {
    return 403;
}

---

🤖 Automation (GitHub Workflow)

The GitHub Action (.github/workflows/update_patterns.yml) automates updates:

• 🕛 Runs Daily at Midnight (UTC)
• 🎯 Manual Trigger Available (from GitHub Actions tab)
• 🚀 Pushes Updated WAF Files to waf_patterns/

---

🔧 Contributing

1. Fork the repository.
2. Create a feature branch (feature/new-patterns).
3. Commit and push changes.
4. Open a pull request (PR).

---

📄 License

This project is licensed under the MIT License.
See the LICENSE file for details.

---

🌐 Resources

• OWASP CRS GitHub
• Caddy Web Server
• Nginx
• Apache ModSecurity
• MIT License

---

🚨 Issues

If you encounter any issues, please open a ticket in the Issues Tab.