Just follow the golden rule: don’t ever load any site more aggressively than you would want yours to be.
This isn’t hard stuff, and these AI companies have grossly inefficient and obnoxious scrapers.
As a site owner those pisses me off as a matter of decency on the web, but as an engineer doing distributed data collection I’m offended by how shitty and inefficient their crawlers are.
They do. Marc Andreeson said as much in his "techno-optimist manifesto," that any hesitation or slowdown in AI development or adoption is equivalent to mass murder.
It’s a waste of bandwidth and CPU on their end as well, “the bitter lesson” isn’t “keep duplicating the same training data”.
I’m glad DeepSeek is showing how inefficient and dogshit most of the frontier model engineer is - how much VC is getting burned literally redownloading a copy of the entire web daily when like <1% of it is new data.
I get they have no shame economically, that they are deluded and greedy. But bad engineering is another class of sin!