feat: Integrate tqdm for progress tracking during bot list fetching
feat: Add regex filters to exclude IP addresses and domains from bot lists
refactor: Remove IP and domain-specific sources from BOT_LIST_SOURCES
refactor: Update parse_bot_list to skip lines matching IP or domain regex
refactor: Improve logging for better debugging and clarity
refactor: Use ThreadPoolExecutor with tqdm for concurrent fetching with progress
docs: Add comments and docstrings for better code understanding
chore: Ensure output directories exist before generating WAF configurations
- Introduced concurrency with ThreadPoolExecutor for faster bot list fetching.
- Added rate limiting and exponential backoff for resilient fetch operations.
- Expanded bot list sources to include new datasets for improved coverage.
- Implemented GitHub token authentication for rate-limited endpoints.
- Refactored file generation to use pathlib for better path management.
- Replaced os.path with Path for consistency and readability.
- Modularized retry logic and enhanced error handling for robustness.
- Ensured bot lists are aggregated and deduplicated across sources.
- Improved logging for better visibility into fetch and write operations.
