feat: rate-limit good bots via object cache, add curl/ClaudeBot tiers, whitelist cache warmer

2026-07-06 09:44:41 +02:00
parent 1b9504e5f9
commit 714064b7c7
4 changed files with 11 additions and 7 deletions
@@ -10,7 +10,6 @@ firefox/19.0
 firefox/38.0
 firefox/18.0
 wget
-curl
 libwww-perl
 WinHttp
 okhttp
@@ -229,7 +228,6 @@ Xaldon
 Zeus
 ZmEu
 Zyborg
-crawle
 igdeSpyder
 Robot
 Aport
@@ -373,7 +371,7 @@ PolycomVVX
 ^The Knowledge AI
 SputnikBot
 od-database-crawler
-Hype%20Machine
+HypeMachine
 The Hype Machine Engine
 Apache-HttpClient
 Goodzer