External/patterns
1
0
Fork 0
mirror of https://github.com/fabriziosalmi/patterns.git synced 2025-12-17 09:45:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #16 from fabriziosalmi/copilot/improve-documentation-quality

Browse Source 
docs: Synchronize documentation with codebase and expand contribution guides
This commit is contained in:
fab 2025-11-15 20:46:44 +01:00 committed by GitHub
commit 87f18886b7
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 647 additions and 45 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
.github/ISSUE_TEMPLATE/bug_report.md vendored
View File

@ -12,27 +12,25 @@ A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
1. Run command '...'
2. With configuration '...'
3. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Error messages/logs**
If applicable, add error messages or log output to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
```
Paste error messages or logs here
```
**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
**Environment (please complete the following information):**
- OS: [e.g. Ubuntu 22.04, macOS 13, Windows 11]
- Python Version: [e.g. 3.11.0]
- Web Server: [e.g. Nginx 1.22, Apache 2.4, Traefik 2.9, HAProxy 2.6]
- Installation Method: [e.g. built from source, downloaded pre-generated configs]
**Additional context**
Add any other context about the problem here.

10
.github/workflows/update_patterns.yml vendored
View File

@ -54,13 +54,8 @@ jobs:
- name: 🔄 Convert OWASP to HAProxy WAF
run: python json2haproxy.py
- name: 🔄 Generate Bad Bot Blockers (Placeholder - Provide badbots.py)
run: |
# Placeholder: Replace this with your actual badbots.py script.
# Assuming badbots.py generates files in waf_patterns/
# Example (if badbots.py creates nginx/bots.conf):
# python badbots.py
echo "Placeholder for badbots.py execution"
- name: 🔄 Generate Bad Bot Blockers
run: python badbots.py
- name: 🚀 Commit and Push Changes (if any)
run: |
@ -98,6 +93,7 @@ jobs:
- name: 🚀 Create GitHub Release (if previous steps succeeded)
id: create_release
if: success() # Only create release if previous steps were successful
uses: actions/create-release@v1
env:

110
CONTRIBUTING.md
View File

@ -1 +1,109 @@
You can send a PR any time.
# Contributing to Patterns
Thank you for your interest in contributing to the Patterns project! We appreciate your help in making this project better.
## How to Contribute
### Reporting Issues
If you find a bug or have a suggestion for improvement:
1. Check if the issue already exists in the [Issues](https://github.com/fabriziosalmi/patterns/issues) section.
2. If not, create a new issue with a clear title and description.
3. Include relevant details such as:
- Steps to reproduce (for bugs)
- Expected vs. actual behavior
- Your environment (OS, Python version, web server type)
### Submitting Pull Requests
We welcome pull requests! Here's how to submit one:
1. **Fork the Repository**
```bash
git clone https://github.com/YOUR_USERNAME/patterns.git
cd patterns
```
2. **Create a Feature Branch**
Use descriptive branch names following this convention:
- `feature/description` - For new features
- `fix/description` - For bug fixes
- `docs/description` - For documentation changes
- `refactor/description` - For code refactoring
Example:
```bash
git checkout -b feature/add-caddy-support
```
3. **Make Your Changes**
- Write clear, concise commit messages
- Follow the existing code style and conventions
- Add comments where necessary
- Update documentation if you're changing functionality
4. **Test Your Changes**
Before submitting, ensure your code works correctly:
```bash
# Install dependencies
pip install -r requirements.txt
# Test the OWASP scraper
python owasp2json.py
# Test the converters
python json2nginx.py
python json2apache.py
python json2traefik.py
python json2haproxy.py
# Test bad bot generation
python badbots.py
```
For web server specific testing, check the respective workflow files in `.github/workflows/`.
5. **Commit and Push**
```bash
git add .
git commit -m "feat: add support for Caddy web server"
git push origin feature/add-caddy-support
```
6. **Open a Pull Request**
- Go to the original repository on GitHub
- Click "New Pull Request"
- Select your branch
- Provide a clear title and description of your changes
- Reference any related issues
## Code Style Guidelines
- Use Python 3.11 or higher
- Follow PEP 8 style guidelines
- Use meaningful variable and function names
- Add docstrings to functions and classes
- Keep functions focused and modular
- Handle errors gracefully with try-except blocks
## Adding Support for New Web Servers
If you want to add support for a new web server:
1. Create a new converter script: `json2WEBSERVER.py`
2. Create output directory: `waf_patterns/WEBSERVER/`
3. Add README.md with integration instructions
4. Update the main README.md to include the new web server
5. Update the GitHub Actions workflow to include the new converter
6. Add example configurations
## Questions?
If you have questions about contributing, feel free to:
- Open an issue for discussion
- Contact the maintainers
Thank you for contributing!

48
README.md
View File

@ -36,14 +36,15 @@ patterns/
│ ├── apache/ # Apache WAF configs (ModSecurity)
│ ├── traefik/ # Traefik WAF configs
│ └── haproxy/ # HAProxy WAF configs
│── import_apache_waf.py
│── import_haproxy_waf.py
│── import_nginx_waf.py
│── import_traefik_waf.py
├── owasp.py # 🕵️ OWASP scraper (fetch CRS rules)
├── owasp2nginx.py # 🔄 Convert OWASP JSON to Nginx WAF configs
├── owasp2apache.py # 🔄 Convert OWASP JSON to Apache ModSecurity configs
├── owasp2haproxy.py # 🔄 Convert OWASP JSON to HAProxy WAF configs
├── import_apache_waf.py # 📥 Import Apache WAF configurations
├── import_haproxy_waf.py # 📥 Import HAProxy WAF configurations
├── import_nginx_waf.py # 📥 Import Nginx WAF configurations
├── import_traefik_waf.py # 📥 Import Traefik WAF configurations
├── owasp2json.py # 🕵️ OWASP scraper (fetch CRS rules)
├── json2nginx.py # 🔄 Convert OWASP JSON to Nginx WAF configs
├── json2apache.py # 🔄 Convert OWASP JSON to Apache ModSecurity configs
├── json2traefik.py # 🔄 Convert OWASP JSON to Traefik WAF configs
├── json2haproxy.py # 🔄 Convert OWASP JSON to HAProxy WAF configs
├── badbots.py # 🤖 Generate WAF configs to block bad bots
├── requirements.txt # 📄 Required dependencies
└── .github/workflows/ # 🤖 GitHub Actions for automation
@ -54,14 +55,14 @@ patterns/
## 🛠️ How It Works
### 🔹 1. Scraping OWASP Rules
- **`owasp.py`** scrapes the latest OWASP CRS patterns from GitHub.
- **`owasp2json.py`** scrapes the latest OWASP CRS patterns from GitHub.
- Extracts **SQLi, XSS, RCE, LFI** patterns from OWASP CRS `.conf` files.
### 🔹 2. Generating WAF Configs for Each Platform
- **`owasp2nginx.py`** – Generates **Nginx WAF** configurations.
- **`owasp2apache.py`** – Outputs **Apache ModSecurity** rules.
- **`owasp2traefik.py`** – Creates **Traefik WAF** rules.
- **`owasp2haproxy.py`** – Builds **HAProxy ACL** files.
- **`json2nginx.py`** – Generates **Nginx WAF** configurations.
- **`json2apache.py`** – Outputs **Apache ModSecurity** rules.
- **`json2traefik.py`** – Creates **Traefik WAF** rules.
- **`json2haproxy.py`** – Builds **HAProxy ACL** files.
### 🔹 3. Bad Bot/User-Agent Detection
- **`badbots.py`** fetches public bot lists and generates bot-blocking configs.
@ -71,6 +72,13 @@ patterns/
## ⚙️ Installation
### Prerequisites
Before installing, ensure you have the following:
- **Python 3.11 or higher** (the project uses Python 3.11 as specified in the GitHub Actions workflow)
- **pip** (Python package installer)
- **git** (for cloning the repository)
### Option 1: Download Pre-Generated Configurations
You can download the latest pre-generated WAF configurations directly from the [GitHub Releases](https://github.com/fabriziosalmi/patterns/releases) page.
@ -94,11 +102,11 @@ pip install -r requirements.txt
**3. Run Manually (Optional):**
```bash
python owasp.py
python owasp2nginx.py
python owasp2apache.py
python owasp2haproxy.py
python owasp2traefik.py
python owasp2json.py
python json2nginx.py
python json2apache.py
python json2haproxy.py
python json2traefik.py
python badbots.py
```
@ -169,9 +177,9 @@ See the [LICENSE](LICENSE) file for details.
---
## Others projects
## Other Projects
If You like my projects, you may also like these ones:
If you like this project, you may also like these:
- [caddy-waf](https://github.com/fabriziosalmi/caddy-waf) Caddy WAF (Regex Rules, IP and DNS filtering, Rate Limiting, GeoIP, Tor, Anomaly Detection)
- [blacklists](https://github.com/fabriziosalmi/blacklists) Hourly updated domains blacklist 🚫

63
SECURITY.md
View File

@ -2,12 +2,71 @@
## Supported Versions
We actively support the current version of this project. The WAF patterns are updated daily via automated GitHub Actions.
| Version | Supported |
| ------- | ------------------ |
| current | :white_check_mark: |
| current (main branch) | :white_check_mark: |
| latest release | :white_check_mark: |
## Reporting a Vulnerability
Please open an issue to report a vulnerability.
We take security vulnerabilities seriously. If you discover a security issue, please follow these steps:
### For Non-Critical Issues
For general security concerns or minor issues:
1. Open an issue in the [Issues](https://github.com/fabriziosalmi/patterns/issues) section
2. Use the label "security" if available
3. Provide a clear description of the issue
### For Critical Vulnerabilities
For critical security vulnerabilities (e.g., in the WAF patterns themselves):
1. **DO NOT** open a public issue
2. Email the maintainer directly at: fabrizio.salmi@gmail.com
3. Include:
- Description of the vulnerability
- Steps to reproduce
- Potential impact
- Suggested fix (if available)
### What to Include
When reporting a vulnerability, please include:
- Type of vulnerability (e.g., regex bypass, pattern detection issue)
- Affected web server(s) (Nginx, Apache, Traefik, HAProxy)
- Attack pattern that bypasses detection
- Suggested regex or pattern improvement
- Any proof-of-concept code (if applicable)
### Response Time
- We aim to acknowledge vulnerability reports within **48 hours**
- Critical vulnerabilities will be addressed in the next daily update
- Less critical issues will be prioritized based on severity
### After Reporting
Once you report a vulnerability:
1. We will acknowledge receipt
2. We will investigate and validate the issue
3. We will work on a fix and test it
4. We will deploy the fix in the next update
5. We will credit you in the release notes (unless you prefer to remain anonymous)
## Security Best Practices
When using the WAF patterns from this project:
- Always test new rules in a staging environment first
- Monitor your logs for false positives
- Keep your web server and WAF software up to date
- Review the OWASP CRS documentation for additional hardening
- Consider layering multiple security controls (WAF + rate limiting + IPS, etc.)
## Scope
This security policy covers:
- WAF pattern generation logic
- Regex patterns for attack detection
- GitHub Actions workflow security
- Dependencies listed in requirements.txt
Thank you for helping keep this project secure!

116
waf_patterns/apache/README.md
View File

@ -1 +1,117 @@
# Apache ModSecurity WAF Configuration
This directory contains Apache ModSecurity WAF configuration files generated from OWASP CRS rules.
You can include these files in your existing Apache configuration to enhance security.
## Prerequisites
- Apache HTTP Server (2.4 or higher)
- ModSecurity module installed and enabled
- Core Rule Set (CRS) base configuration
## Installation
### Ubuntu/Debian
```bash
sudo apt-get update
sudo apt-get install libapache2-mod-security2
sudo a2enmod security2
sudo systemctl restart apache2
```
### CentOS/RHEL
```bash
sudo yum install mod_security
sudo systemctl restart httpd
```
## Usage
1. Copy the generated configuration files to your Apache configuration directory:
```bash
sudo cp waf_patterns/apache/*.conf /etc/apache2/modsecurity.d/
# or for CentOS/RHEL:
# sudo cp waf_patterns/apache/*.conf /etc/httpd/modsecurity.d/
```
2. Include the configuration files in your Apache configuration.
Edit `/etc/apache2/mods-enabled/security2.conf` (Ubuntu/Debian) or `/etc/httpd/conf.d/mod_security.conf` (CentOS/RHEL):
```apache
<IfModule security2_module>
Include /etc/apache2/modsecurity.d/*.conf
</IfModule>
```
3. Test the configuration:
```bash
# Ubuntu/Debian
sudo apache2ctl configtest
# CentOS/RHEL
sudo httpd -t
```
4. Reload Apache to apply the changes:
```bash
# Ubuntu/Debian
sudo systemctl reload apache2
# CentOS/RHEL
sudo systemctl reload httpd
```
## Configuration Details
The generated rules include:
- **SQL Injection (SQLi)** detection patterns
- **Cross-Site Scripting (XSS)** prevention rules
- **Remote Code Execution (RCE)** blocking
- **Local File Inclusion (LFI)** protection
- **Bad Bot/User-Agent** blocking
## Customization
You can adjust the severity and actions for each rule by modifying the configuration files.
Common actions include:
- `deny` - Block the request
- `log` - Log the event
- `status:403` - Return HTTP 403 Forbidden
## Troubleshooting
### Check ModSecurity is loaded
```bash
# Ubuntu/Debian
apache2ctl -M | grep security
# CentOS/RHEL
httpd -M | grep security
```
### View ModSecurity logs
```bash
# Ubuntu/Debian
sudo tail -f /var/log/apache2/modsec_audit.log
# CentOS/RHEL
sudo tail -f /var/log/httpd/modsec_audit.log
```
### Test with a sample attack
```bash
curl "http://yourserver.com/?id=1' OR '1'='1"
# Should return 403 Forbidden if WAF is working
```
## Notes
- Rules are updated daily via GitHub Actions
- Blocked requests return a `403 Forbidden` response by default
- Review the ModSecurity documentation for advanced configuration options
## Resources
- [ModSecurity Documentation](https://github.com/SpiderLabs/ModSecurity)
- [OWASP CRS](https://coreruleset.org/)
- [Apache ModSecurity Module](https://modsecurity.org/)

182
waf_patterns/haproxy/README.md
View File

@ -1 +1,183 @@
# HAProxy WAF Configuration
This directory contains HAProxy WAF configuration files generated from OWASP CRS rules.
You can include these ACL (Access Control List) files in your HAProxy configuration to enhance security.
## Prerequisites
- HAProxy 2.0 or higher
- Basic understanding of HAProxy ACLs and rules
## Configuration Files
The generated files include:
- ACL files with pattern matching rules
- Request filtering configurations
- Bad bot/User-Agent blocking lists
## Usage
1. Copy the generated ACL files to your HAProxy configuration directory:
```bash
sudo cp waf_patterns/haproxy/*.acl /etc/haproxy/
```
2. Include the ACL files in your HAProxy configuration.
Edit `/etc/haproxy/haproxy.cfg`:
```haproxy
frontend http-in
bind *:80
# Load WAF ACL files
acl is_sql_injection path_reg -i -f /etc/haproxy/sqli_patterns.acl
acl is_xss_attack path_reg -i -f /etc/haproxy/xss_patterns.acl
acl is_bad_bot hdr_reg(User-Agent) -i -f /etc/haproxy/bad_bots.acl
# Block malicious requests
http-request deny if is_sql_injection
http-request deny if is_xss_attack
http-request deny if is_bad_bot
# Default backend
default_backend web_servers
backend web_servers
balance roundrobin
server web1 10.0.0.1:80 check
server web2 10.0.0.2:80 check
```
3. Test the configuration:
```bash
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
```
4. Reload HAProxy to apply the changes:
```bash
sudo systemctl reload haproxy
# or
sudo service haproxy reload
```
## Advanced Configuration
### Logging Blocked Requests
Add logging for better visibility:
```haproxy
frontend http-in
bind *:80
# ... ACL definitions ...
# Log blocked requests
http-request capture req.hdr(User-Agent) len 200
http-request deny deny_status 403 if is_sql_injection
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
```
### Custom Error Pages
Return custom error pages for blocked requests:
```haproxy
frontend http-in
bind *:80
# ... ACL definitions ...
# Return custom error page
http-request deny deny_status 403 if is_sql_injection
errorfile 403 /etc/haproxy/errors/403.http
```
### Rate Limiting
Combine with rate limiting for additional protection:
```haproxy
frontend http-in
bind *:80
# Track request rate
stick-table type ip size 100k expire 30s store http_req_rate(10s)
http-request track-sc0 src
# Deny if rate limit exceeded
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
# ... WAF ACLs ...
```
## Testing
### Test SQL Injection Protection
```bash
curl "http://yourserver.com/?id=1' OR '1'='1"
# Should return 403 Forbidden
```
### Test XSS Protection
```bash
curl "http://yourserver.com/?q=<script>alert('xss')</script>"
# Should return 403 Forbidden
```
### Test Bad Bot Blocking
```bash
curl -H "User-Agent: AhrefsBot" http://yourserver.com
# Should return 403 Forbidden
```
## Monitoring
### Check HAProxy Stats
```bash
# Enable stats in haproxy.cfg
listen stats
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
```
Visit `http://yourserver:8404/stats` to view statistics.
### View Logs
```bash
sudo tail -f /var/log/haproxy.log
```
## Performance Considerations
- ACL pattern matching is highly efficient in HAProxy
- Use regular expressions sparingly for better performance
- Consider using stick tables for rate limiting
- Monitor CPU and memory usage under load
- Test thoroughly before deploying to production
## Configuration Details
The ACL files protect against:
- **SQL Injection (SQLi)** - Common SQL injection patterns
- **Cross-Site Scripting (XSS)** - JavaScript injection attempts
- **Remote Code Execution (RCE)** - Command injection patterns
- **Local File Inclusion (LFI)** - Path traversal attempts
- **Bad Bots** - Known malicious crawlers and scrapers
## Notes
- Rules are updated daily via GitHub Actions
- Blocked requests return `403 Forbidden` by default
- ACLs are case-insensitive (`-i` flag)
- Regular expressions are used for pattern matching (`-f` for file-based ACLs)
- Compatible with HAProxy 2.0 and higher
## Resources
- [HAProxy Documentation](https://www.haproxy.org/#docs)
- [HAProxy ACL Guide](https://www.haproxy.com/documentation/hapee/latest/onepage/#7)
- [OWASP CRS](https://coreruleset.org/)
- [HAProxy Configuration Manual](http://cbonte.github.io/haproxy-dconv/)

135
waf_patterns/traefik/README.md
View File

@ -1 +1,136 @@
# Traefik WAF Configuration
This directory contains Traefik WAF configuration files generated from OWASP CRS rules.
You can use these middleware configurations to enhance security in your Traefik setup.
## Prerequisites
- Traefik v2.x or higher
- Basic understanding of Traefik middleware
## Configuration Files
The generated configuration includes:
- Middleware definitions for request filtering
- Regular expression patterns for attack detection
- Bad bot/User-Agent blocking rules
## Usage
### Option 1: File Provider (Recommended)
1. Copy the generated configuration files to your Traefik configuration directory:
```bash
cp waf_patterns/traefik/*.toml /etc/traefik/dynamic/
# or to your custom config directory
```
2. Configure Traefik to load dynamic configuration from files.
In your `traefik.yml` or `traefik.toml`:
```yaml
providers:
file:
directory: "/etc/traefik/dynamic"
watch: true
```
3. Apply the middleware to your routes by referencing it in your service configuration:
```yaml
http:
routers:
my-router:
rule: "Host(`example.com`)"
service: my-service
middlewares:
- waf-middleware
```
### Option 2: Docker Labels
If you're using Docker, you can apply the middleware via labels:
```yaml
services:
my-service:
image: my-app:latest
labels:
- "traefik.enable=true"
- "traefik.http.routers.my-router.rule=Host(`example.com`)"
- "traefik.http.routers.my-router.middlewares=waf-middleware@file"
```
### Option 3: Kubernetes IngressRoute
For Kubernetes deployments:
```yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: waf-middleware
spec:
plugin:
# Reference your WAF plugin configuration here
```
## Configuration Details
The middleware includes protection against:
- **SQL Injection (SQLi)** attacks
- **Cross-Site Scripting (XSS)** attempts
- **Remote Code Execution (RCE)** patterns
- **Local File Inclusion (LFI)** attempts
- **Malicious bots and crawlers**
## Testing
Test the WAF is working by sending a malicious request:
```bash
curl -H "User-Agent: AhrefsBot" http://yourserver.com
# Should be blocked if bot protection is working
curl "http://yourserver.com/?id=1' OR '1'='1"
# Should be blocked if SQLi protection is working
```
## Monitoring
Monitor blocked requests in Traefik logs:
```bash
# Docker
docker logs traefik 2>&1 | grep -i "blocked\|forbidden"
# Standard installation
tail -f /var/log/traefik/access.log | grep -i "403"
```
## Customization
You can customize the middleware behavior by:
1. Editing the generated `.toml` files
2. Adjusting regex patterns for your specific needs
3. Modifying response codes and error pages
4. Adding custom headers for blocked requests
## Performance Considerations
- Regular expression matching can impact performance under high load
- Consider using caching middleware in combination with WAF
- Monitor CPU usage and adjust rules if needed
- Use Traefik's built-in rate limiting for additional protection
## Notes
- Rules are updated daily via GitHub Actions
- Blocked requests typically return `403 Forbidden` or `400 Bad Request`
- Middleware is applied at the router level
- Compatible with other Traefik middlewares (chain them as needed)
## Resources
- [Traefik Documentation](https://doc.traefik.io/traefik/)
- [Traefik Middleware](https://doc.traefik.io/traefik/middlewares/overview/)
- [OWASP CRS](https://coreruleset.org/)