External/patterns
1
0
Fork 0
mirror of https://github.com/fabriziosalmi/patterns.git synced 2025-12-17 09:45:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
patterns/README.md
fab 472b26aef4
Update README.md
2025-01-03 13:24:40 +01:00

12 KiB
Raw Blame History

🔒 Patterns: OWASP CRS and Bad Bot Detection for Web Servers

Automate the scraping of OWASP Core Rule Set (CRS) patterns and convert them into Apache, Nginx, Traefik, and HAProxy WAF configurations.
Additionally, Bad Bot/User-Agent detection is integrated to block malicious web crawlers and scrapers.

🚀 Protect your servers against SQL Injection (SQLi), XSS, RCE, LFI, and malicious bots – with automated daily updates.

📌 Project Highlights

  • 🛡️ OWASP CRS Protection – Leverages OWASP Core Rule Set for web application firewall (WAF) defense.
  • 🤖 Bad Bot Blocking – Blocks known malicious bots using public bot lists.
  • ⚙️ Multi-Web Server Support – Generates WAF configs for Apache, Nginx, Traefik, and HAProxy.
  • 🔄 Automatic Updates – GitHub Actions fetch new rules daily and push updated configs.
  • 🧩 Scalable and Modular – Easily extendable to support other web servers or load balancers.

🌐 Supported Web Servers

  • 🔵 Nginx
  • 🟠 Apache (ModSecurity)
  • 🟣 Traefik
  • 🔴 HAProxy

Note

If you are using Caddy check the caddy-waf project.

📂 Project Structure

patterns/
├── waf_patterns/           # 🔧 Generated WAF config files
│   ├── nginx/              # Nginx WAF configs
│   ├── apache/             # Apache WAF configs (ModSecurity)
│   ├── traefik/            # Traefik WAF configs
│   └── haproxy/            # HAProxy WAF configs
│── import_apache_waf.py
│── import_haproxy_waf.py
│── import_nginx_waf.py
│── import_traefik_waf.py
├── owasp.py                # 🕵️ OWASP scraper (fetch CRS rules)
├── owasp2nginx.py          # 🔄 Convert OWASP JSON to Nginx WAF configs
├── owasp2apache.py         # 🔄 Convert OWASP JSON to Apache ModSecurity configs
├── owasp2haproxy.py        # 🔄 Convert OWASP JSON to HAProxy WAF configs
├── badbots.py              # 🤖 Generate WAF configs to block bad bots
├── requirements.txt        # 📄 Required dependencies
└── .github/workflows/      # 🤖 GitHub Actions for automation
    └── update_patterns.yml

🛠️ How It Works

🔹 1. Scraping OWASP Rules

  • owasp.py scrapes the latest OWASP CRS patterns from GitHub.
  • Extracts SQLi, XSS, RCE, LFI patterns from OWASP CRS .conf files.

🔹 2. Generating WAF Configs for Each Platform

  • owasp2nginx.py – Generates Nginx WAF configurations.
  • owasp2apache.py – Outputs Apache ModSecurity rules.
  • owasp2traefik.py – Creates Traefik WAF rules.
  • owasp2haproxy.py – Builds HAProxy ACL files.

🔹 3. Bad Bot/User-Agent Detection

  • badbots.py fetches public bot lists and generates bot-blocking configs.
  • Supports fallback lists to ensure reliable detection.

Example

fab@Mac patterns % python3 owasp.py
2025-01-03 13:22:52,744 - INFO - Latest matching tag: refs/tags/v4.0.0-rc2
2025-01-03 13:22:52,988 - INFO - Fetching REQUEST-901-INITIALIZATION.conf...
2025-01-03 13:22:53,164 - INFO - Fetching REQUEST-905-COMMON-EXCEPTIONS.conf...
2025-01-03 13:22:53,335 - INFO - Fetching REQUEST-911-METHOD-ENFORCEMENT.conf...
2025-01-03 13:22:53,513 - INFO - Fetching REQUEST-913-SCANNER-DETECTION.conf...
2025-01-03 13:22:53,683 - INFO - Fetching REQUEST-920-PROTOCOL-ENFORCEMENT.conf...
2025-01-03 13:22:53,892 - INFO - Fetching REQUEST-921-PROTOCOL-ATTACK.conf...
2025-01-03 13:22:54,057 - INFO - Fetching REQUEST-922-MULTIPART-ATTACK.conf...
2025-01-03 13:22:54,218 - INFO - Fetching REQUEST-930-APPLICATION-ATTACK-LFI.conf...
2025-01-03 13:22:54,388 - INFO - Fetching REQUEST-931-APPLICATION-ATTACK-RFI.conf...
2025-01-03 13:22:54,563 - INFO - Fetching REQUEST-932-APPLICATION-ATTACK-RCE.conf...
2025-01-03 13:22:54,784 - INFO - Fetching REQUEST-933-APPLICATION-ATTACK-PHP.conf...
2025-01-03 13:22:54,947 - INFO - Fetching REQUEST-934-APPLICATION-ATTACK-GENERIC.conf...
2025-01-03 13:22:55,150 - INFO - Fetching REQUEST-941-APPLICATION-ATTACK-XSS.conf...
2025-01-03 13:22:55,328 - INFO - Fetching REQUEST-942-APPLICATION-ATTACK-SQLI.conf...
2025-01-03 13:22:55,560 - INFO - Fetching REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf...
2025-01-03 13:22:55,750 - INFO - Fetching REQUEST-944-APPLICATION-ATTACK-JAVA.conf...
2025-01-03 13:22:55,922 - INFO - Fetching REQUEST-949-BLOCKING-EVALUATION.conf...
2025-01-03 13:22:56,106 - INFO - Fetching RESPONSE-950-DATA-LEAKAGES.conf...
2025-01-03 13:22:56,530 - INFO - Fetching RESPONSE-951-DATA-LEAKAGES-SQL.conf...
2025-01-03 13:22:56,705 - INFO - Fetching RESPONSE-952-DATA-LEAKAGES-JAVA.conf...
2025-01-03 13:22:57,088 - INFO - Fetching RESPONSE-953-DATA-LEAKAGES-PHP.conf...
2025-01-03 13:22:57,279 - INFO - Fetching RESPONSE-954-DATA-LEAKAGES-IIS.conf...
2025-01-03 13:22:57,454 - INFO - Fetching RESPONSE-955-WEB-SHELLS.conf...
2025-01-03 13:22:57,669 - INFO - Fetching RESPONSE-959-BLOCKING-EVALUATION.conf...
2025-01-03 13:22:57,842 - INFO - Fetching RESPONSE-980-CORRELATION.conf...
2025-01-03 13:22:58,006 - INFO - Fetched 646 rules.
2025-01-03 13:22:58,013 - INFO - Rules saved to owasp_rules.json.
2025-01-03 13:22:58,014 - INFO - All rules fetched and saved successfully.

fab@Mac patterns % python3 owasp2apache.py
2025-01-03 13:23:02,973 - INFO - [*] Loading OWASP rules...
2025-01-03 13:23:02,974 - INFO - [*] Generating Apache WAF configs from 646 rules...
2025-01-03 13:23:02,975 - INFO - [+] Generated waf_patterns/apache/initialization.conf (30 patterns)
2025-01-03 13:23:02,975 - INFO - [+] Generated waf_patterns/apache/exceptions.conf (5 patterns)
2025-01-03 13:23:02,975 - INFO - [+] Generated waf_patterns/apache/enforcement.conf (112 patterns)
2025-01-03 13:23:02,975 - INFO - [+] Generated waf_patterns/apache/detection.conf (9 patterns)
2025-01-03 13:23:02,976 - INFO - [+] Generated waf_patterns/apache/attack.conf (31 patterns)
2025-01-03 13:23:02,976 - INFO - [+] Generated waf_patterns/apache/lfi.conf (13 patterns)
2025-01-03 13:23:02,976 - INFO - [+] Generated waf_patterns/apache/rfi.conf (15 patterns)
2025-01-03 13:23:02,976 - INFO - [+] Generated waf_patterns/apache/rce.conf (54 patterns)
2025-01-03 13:23:02,977 - INFO - [+] Generated waf_patterns/apache/php.conf (39 patterns)
2025-01-03 13:23:02,977 - INFO - [+] Generated waf_patterns/apache/generic.conf (18 patterns)
2025-01-03 13:23:02,977 - INFO - [+] Generated waf_patterns/apache/xss.conf (43 patterns)
2025-01-03 13:23:02,977 - INFO - [+] Generated waf_patterns/apache/sqli.conf (73 patterns)
2025-01-03 13:23:02,977 - INFO - [+] Generated waf_patterns/apache/fixation.conf (14 patterns)
2025-01-03 13:23:02,977 - INFO - [+] Generated waf_patterns/apache/java.conf (34 patterns)
2025-01-03 13:23:02,978 - INFO - [+] Generated waf_patterns/apache/evaluation.conf (54 patterns)
2025-01-03 13:23:02,978 - INFO - [+] Generated waf_patterns/apache/leakages.conf (11 patterns)
2025-01-03 13:23:02,978 - INFO - [+] Generated waf_patterns/apache/sql.conf (25 patterns)
2025-01-03 13:23:02,978 - INFO - [+] Generated waf_patterns/apache/iis.conf (13 patterns)
2025-01-03 13:23:02,978 - INFO - [+] Generated waf_patterns/apache/shells.conf (34 patterns)
2025-01-03 13:23:02,978 - INFO - [+] Generated waf_patterns/apache/correlation.conf (19 patterns)
2025-01-03 13:23:02,978 - INFO - [✔] Apache ModSecurity configurations generated successfully.

fab@Mac patterns % python3 owasp2nginx.py
2025-01-03 13:23:06,134 - INFO - [*] Loading OWASP rules...
2025-01-03 13:23:06,135 - INFO - [*] Generating Nginx WAF configs from 646 rules...
2025-01-03 13:23:06,135 - INFO - [+] Generated waf_patterns/nginx/initialization.conf (30 patterns)
2025-01-03 13:23:06,135 - INFO - [+] Generated waf_patterns/nginx/exceptions.conf (5 patterns)
2025-01-03 13:23:06,136 - INFO - [+] Generated waf_patterns/nginx/enforcement.conf (112 patterns)
2025-01-03 13:23:06,136 - INFO - [+] Generated waf_patterns/nginx/detection.conf (9 patterns)
2025-01-03 13:23:06,136 - INFO - [+] Generated waf_patterns/nginx/attack.conf (31 patterns)
2025-01-03 13:23:06,136 - INFO - [+] Generated waf_patterns/nginx/lfi.conf (13 patterns)
2025-01-03 13:23:06,136 - INFO - [+] Generated waf_patterns/nginx/rfi.conf (15 patterns)
2025-01-03 13:23:06,136 - INFO - [+] Generated waf_patterns/nginx/rce.conf (54 patterns)
2025-01-03 13:23:06,137 - INFO - [+] Generated waf_patterns/nginx/php.conf (39 patterns)
2025-01-03 13:23:06,137 - INFO - [+] Generated waf_patterns/nginx/generic.conf (18 patterns)
2025-01-03 13:23:06,137 - INFO - [+] Generated waf_patterns/nginx/xss.conf (43 patterns)
2025-01-03 13:23:06,137 - INFO - [+] Generated waf_patterns/nginx/sqli.conf (73 patterns)
2025-01-03 13:23:06,137 - INFO - [+] Generated waf_patterns/nginx/fixation.conf (14 patterns)
2025-01-03 13:23:06,137 - INFO - [+] Generated waf_patterns/nginx/java.conf (34 patterns)
2025-01-03 13:23:06,137 - INFO - [+] Generated waf_patterns/nginx/evaluation.conf (54 patterns)
2025-01-03 13:23:06,138 - INFO - [+] Generated waf_patterns/nginx/leakages.conf (11 patterns)
2025-01-03 13:23:06,138 - INFO - [+] Generated waf_patterns/nginx/sql.conf (25 patterns)
2025-01-03 13:23:06,138 - INFO - [+] Generated waf_patterns/nginx/iis.conf (13 patterns)
2025-01-03 13:23:06,138 - INFO - [+] Generated waf_patterns/nginx/shells.conf (34 patterns)
2025-01-03 13:23:06,138 - INFO - [+] Generated waf_patterns/nginx/correlation.conf (19 patterns)
2025-01-03 13:23:06,138 - INFO - [✔] Nginx WAF configurations generated successfully.

fab@Mac patterns % python3 owasp2haproxy.py
2025-01-03 13:23:08,952 - INFO - [*] Loading OWASP rules...
2025-01-03 13:23:08,953 - INFO - [*] Generating HAProxy WAF configs from 646 rules...
2025-01-03 13:23:08,953 - INFO - [+] Created or verified directory: waf_patterns/haproxy
2025-01-03 13:23:08,954 - INFO - [+] HAProxy WAF rules generated at waf_patterns/haproxy/waf.acl
2025-01-03 13:23:08,954 - INFO - [✔] HAProxy WAF configurations generated successfully.
fab@Mac patterns % python3 owasp2traefik.py
[+] Traefik WAF rules generated at waf_patterns/traefik/middleware.toml

⚙️ Installation

1. Clone the Repository:

git clone https://github.com/fabriziosalmi/patterns.git  
cd patterns

2. Install Dependencies:

pip install -r requirements.txt

3. Run Manually (Optional):

python owasp.py
python owasp2nginx.py
python owasp2apache.py
python owasp2haproxy.py
python owasp2traefik.py
python badbots.py

🚀 Usage (Web Server Integration)

🔹 1. Nginx WAF Integration

sudo python3 import_nginx_waf.py

🔹 2. Apache WAF Integration

sudo python3 import_apache_waf.py

🔹 3. Traefik WAF Integration

sudo python3 import_traefik_waf.py

🔹 4. HAProxy WAF Integration

sudo python3 import_haproxy_waf.py

🔧 Example Output (Bot Blocker – Nginx)

map $http_user_agent $bad_bot {
    "~*AhrefsBot" 1;
    "~*SemrushBot" 1;
    "~*MJ12bot" 1;
    default 0;
}
if ($bad_bot) {
    return 403;
}

🤖 Automation (GitHub Workflow)

  • 🕛 Daily Updates – GitHub Actions fetch the latest OWASP CRS rules every day.
  • 🔄 Auto Deployment – Pushes new .conf files directly to waf_patterns/.
  • 🎯 Manual Trigger – Updates can also be triggered manually.

🤝 Contributing

  1. Fork the repository.
  2. Create a feature branch (feature/new-patterns).
  3. Commit and push changes.
  4. Open a Pull Request.

📄 License

This project is licensed under the MIT License.
See the LICENSE file for details.

📞 Need Help?

🌐 Resources