fab f347ac2817 Update badbots.py ...

- Introduced concurrency with ThreadPoolExecutor for faster bot list fetching.
- Added rate limiting and exponential backoff for resilient fetch operations.
- Expanded bot list sources to include new datasets for improved coverage.
- Implemented GitHub token authentication for rate-limited endpoints.
- Refactored file generation to use pathlib for better path management.
- Replaced os.path with Path for consistency and readability.
- Modularized retry logic and enhanced error handling for robustness.
- Ensured bot lists are aggregated and deduplicated across sources.
- Improved logging for better visibility into fetch and write operations.

2024-12-30 00:08:12 +01:00

5.9 KiB

Raw Blame History

View Raw