Hacker News Reader

Comment by FeepingCreature | Hacker News Reader

To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.

View on HN

> "FOSS sites damaged by extreme incompetence and unaccountability."

Given this info, the natural next question is “who is doing the harm?”

The answer is “AI companies”. Most people would now view the situation as having a lot to do with AI companies.

As a matter of fact, yes. As a matter of cause, no. Being an AI company doesn't make these companies especially incompetent; rather, this is the normal tech company level of incompetence, and being an AI company causes it to externalize via crawling.

Bing used to do the same thing. (It might still do it, I just haven't heard about it in a while.)

It’s not incompetence. These large AI companies have (or can hire) the competency to engineer proper crawlers. This is deliberate due to lack of accountability and “who’s gonna stop them?”

> has nothing to do with AI

Surely has very little to do with "intelligence".

I think there’s room to argue that that’s a matter of perspective. The intent of the attacker is not that important to the people being attacked.

In the same way a drunk driver isn't deliberately trying to run over pedestrians, I suppose. I think gross negligence is in many ways worse than malice. A malicious actor can at least be somewhat reasoned with.

Correct, they could - but they are not. This is about the unaccountability, and if I was charitable (which I'm not) I'd add also incompetence, of the techbros leading the AI giants. Are we still expecting anything like "ethics"? I hope the few engineers reading HN will still have some, but the higher you go the foreign the concept gets.

Sure, I just think focusing it on AI companies misses the reason this happens. It's not an "AI company problem", it's a "tech company problem". It just happens that AI companies are the tech companies that externalize their incompetence with crawlers at this point in time.

The only way to stop a bad AI with a gun is a good AI with a gun?

> The crawlers could just as well be search engine startups.

And yet they are not. So what does that tell you?

Yes, any crawler or some traffic peak can do this if your infrastructure is not well engineered

You're getting it the wrong way. It's: Any crawler that's not well engineered, that doesn't follow robots.txt, that fakes its User Agent, that doesn't allow you to contact hem, that fetches content an indiscriminate number of times, repeatedly, all day long... can do this to your infrastructure unless you're a giant.

What these crawlers are doing is akin to DDoS attacks.

Please do explain how you'd engineer a site to deal with barrage of poorly written scrapers descending upon it. After you've done geo-ip routing, implemented various levels of caching, separated read/write traffic and bought an ever increasing amount of bandwidth, what is there left to do?

You could also get CloudFlare, or some other CDN, but depending on your size that might not be within your budget. I don't get why the rest of the internet should subsidize these AI companies. They're not profitable and live of venture capital and increase the operation costs of everyone else.

Reply on news.ycombinator.com