It’s almost necessary at this point. At least some form of AI scraper prevention.
I had to take my public repos down a couple days ago, individuals and belligerents using botnets make blocking scrapers via normal means (user-agent/CIDR block) ineffective. So things like CloudFlare or Anubis are becoming necessary.
Ironic that they have their repos hiding behind Cloudflare, then.
It’s almost necessary at this point. At least some form of AI scraper prevention.
I had to take my public repos down a couple days ago, individuals and belligerents using botnets make blocking scrapers via normal means (user-agent/CIDR block) ineffective. So things like CloudFlare or Anubis are becoming necessary.
well it’s anti–LLM crawlers