- Error Handling:
- Added error handling for file operations, JSON parsing, and invalid rule structures.
- Logs warnings for invalid rules instead of crashing.
- Path Handling:
- Used pathlib.Path for better path manipulation and readability.
Made paths configurable via environment variables.
- Logging:
- Replaced print() with Python's logging module for more flexible and structured logging.
- Input Validation:
- Added checks for missing keys in the input JSON file.
- Rule Formatting:
- Ensured proper formatting of HAProxy ACL rules.
- Output Directory Permissions:
- Ensured the output directory is created with parents=True to handle nested directories.
- Code Structure:
- Encapsulated the main logic in a main() function for better organization.
- Added docstrings to functions for clarity.
- Error Handling: Added try-except blocks to handle file operations, subprocess commands, and permission issues. Logs detailed error messages for debugging.
- Path Handling: Used pathlib.Path for better path manipulation and readability. Made paths configurable via environment variables.
- File Permissions: Ensured the target directory is created with parents=True to handle nested directories. Checked if files already exist in the target directory to avoid unnecessary overwrites.
- Logging: Added more detailed logging for better transparency and debugging.
- Subprocess Security: Added checks for apachectl and systemctl commands to ensure compatibility with supported systems.
- Input Validation: Validated the existence of .conf files before copying them.
- Code Structure: Encapsulated the main logic in a main() function for better organization.
- Added docstrings to functions for clarity.
- Error Handling: Added error handling for file operations, JSON parsing, and invalid rule structures. Logs warnings for invalid rules instead of crashing.
- Unique Rule IDs: Each rule is assigned a unique id to avoid collisions in ModSecurity.
- Path Handling: Used pathlib.Path for better path manipulation and readability.
- Logging: Replaced print() with Python's logging module for more flexible and structured logging.
- Input Validation: Added checks for missing keys in the input JSON file.
- Template for Rules: Used a template string (MODSEC_RULE_TEMPLATE) for consistent rule formatting.
- Output Directory Permissions: Ensured the output directory is created with parents=True to handle nested directories.
- Introduced concurrency with ThreadPoolExecutor for faster bot list fetching.
- Added rate limiting and exponential backoff for resilient fetch operations.
- Expanded bot list sources to include new datasets for improved coverage.
- Implemented GitHub token authentication for rate-limited endpoints.
- Refactored file generation to use pathlib for better path management.
- Replaced os.path with Path for consistency and readability.
- Modularized retry logic and enhanced error handling for robustness.
- Ensured bot lists are aggregated and deduplicated across sources.
- Improved logging for better visibility into fetch and write operations.
- Added exponential backoff and retry logic for GitHub API requests to handle rate limits and transient errors.
- Introduced SHA verification for fetched blobs to ensure data integrity.
- Implemented optional GitHub token support for authenticated requests.
- Improved handling of the latest matching tag by dynamically sorting and selecting the newest available version.
- Reorganized functions for better modularity and readability.
- Added error handling for blob fetching and decoding with base64 processing.
- Directory creation for output files now ensured, improving save reliability.
Initially tried to add all files ending in .conf in correct dir to a
string in {a,b,c,d} form for cp, but ran into a char limit for cp so
sadly went with the slower for loop method
