Fighting the AI Scraperbot Scourge | Hacker News Reader

Fighting the AI Scraperbot Scourge

lwn.net

54 pointspabs310 comments

Interesting how he brings up AI companies paying botnets to scrape sites, I hadn't heard of that before. It's extremely malicious behavior because as he mentions, it's literally a DDoS attack.

> But, increasingly, the scraperbot traffic does not fit that pattern. Instead, traffic will come from literally millions of IP addresses, where no specific address is responsible for more than two or three hits over the course of a week. Watching the traffic on the site, one can easily see scraping efforts that are fetching a sorted list of URLs in an obvious sequence, but the same IP address will not appear twice in that sequence. The specific addresses involved come from all over the globe, with no evident pattern.

> In other words, this scraping is being done by botnets, quite likely bought in underground markets and consisting of compromised machines. There really is not any other explanation that fits the observed patterns. Once upon a time, compromised systems were put to work mining cryptocurrency; now, it seems, there is more money to be had in repeatedly scraping the same web pages. When one of these botnets goes nuts, the result is indistinguishable from a distributed denial-of-service (DDOS) attack — it is a distributed denial-of-service attack. Should anybody be in doubt about the moral integrity of the people running these systems, a look at the techniques they use should make the situation abundantly clear.

So far I have not seen any of the operators proposing what seems to me to be the obvious solution: sell this data set in a machine readable format for cheap to the biggest scrapers.

It would have to be aggregated data that's constantly updated if it were to be anywhere competitive to just simply scraping the page.

Plus, the "biggest scrapers" have shown their unwillingness to pay for content.

Yes, presumably the approach would be to publish both a current dump of all of the databases every week or month, as well as a daily diff. Even better if it were just given away for free. The challenge would be around monetization and licensing. Even if you only published e.g. the kernel mailing list archive in this way, I could see someone like OpenAI turning around and suing to claim that you had given them copyright over that content rather than just providing the content itself.

Lots of sites have been trying to do this, but they're fighting a losing battle because the AI companies can get the same data for free.

They're probably using residential proxy services. You can pay for a proxy service that redirects each individual connection through a random proxy in their pool, and all of them are in netblocks marked as residential. They're not all botnets - people can also get paid for sharing their connection. They're expensive, on the order of $1/GB.

I think that in the next 12 months AI companies will shut these scrapers off. The big AI companies have enough data already, and there won't be funding for any new smaller companies.

Good post, but it looks like there is no solution to this? At what point do you give up the web and invent a different protocol? Is there actually a solution to this?

Maybe the botnets can start fighting each other, eliminate the drifters to maximise inputs for slopclips

Wonder if meta et al buy these kind of services, they already steal so it’s not a huge step

Ye ai/ml was a neat piece of tech; it is intellectually interesting to try figure out what it learns and how it learns; and there are maybe valid uses, but capitalism and worse has utterly consumed it.

I don’t have the words to quite describe it all, the farce of it. Then you get people that talk about alignment, how can anything “aligned” be built on such a foundation.

What’s the end goal, a techbronocracy worshiping a fake god

Reply on news.ycombinator.com