5 Commits

Author	SHA1	Message	Date
fab	9f1266f81f	Update badbots.py ... parsing fixes	2025-01-01 12:56:12 +01:00
fab	f347ac2817	Update badbots.py ... - Introduced concurrency with ThreadPoolExecutor for faster bot list fetching. - Added rate limiting and exponential backoff for resilient fetch operations. - Expanded bot list sources to include new datasets for improved coverage. - Implemented GitHub token authentication for rate-limited endpoints. - Refactored file generation to use pathlib for better path management. - Replaced os.path with Path for consistency and readability. - Modularized retry logic and enhanced error handling for robustness. - Ensured bot lists are aggregated and deduplicated across sources. - Improved logging for better visibility into fetch and write operations.	2024-12-30 00:08:12 +01:00
fab	2763bbc13b	Update badbots.py ... Fix for fallback #1 URL which lands nowhere (404).	2024-12-21 11:29:45 +01:00
fab	df34f762c3	Update badbots.py	2024-12-21 01:57:55 +01:00
fab	c9ef0751e1	Create badbots.py	2024-12-21 01:46:33 +01:00