If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve.
Yes, there are sites being DDoSed by scrapers for LLMs.
> If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve.
This isn't about one request per week or per month. There were reports from many sites that they're being hit by scrapers that request from many different IP addresses, one request each.
Not sure why you believe massive repeated scraping isn't a problem. It's not like there is just one single actor out there, and ignoring robits.txt seems to be the norm nowadays.
It is very real and the reason why Anubis has been created in the first place. It is not plain hostility towards LLMs, it is *first and foremost* a DDoS protection against their scrapers.
https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
There are already a few dozens of thousands of scrapers right now trying to get even more training data.
It will only get worse. We all want more training data. I want more training data. You want more training data.
We all want the most up to date data there is. So, yeah, it will only get worse as time goes on.