Hacker News Reader

Comment by ivanjermakov | Hacker News Reader

Also, isn't this illegal? Either Google respected rate limits (which I doubt) or it was a form of a DDoS attack.

It's only illegal if you poor people do it. For us rich corporations, we do what we want

Google is the kingmaker and basically a monopoly on search. If you're going to light up dollars to share with a bot, let it be with google, who on a lucky day might decide you are king (because you let them index your site) I would presume and assume google obeys the robots.txt mandate as well.

But I would agree it's a very outstanding and real problem that is YC-worthy - sharing structured webpage data with trusted partners in a generic and efficient way. I've heard about various AI companies that perform such data scraping and structuring with AI, forget the name - this is many notches in sophistication above a Selenium-headless type driver. If only html were made into a model-view-controller neatly and users were let to bring their own views & controllers.

Are there specific laws that deal with rate limits? Honest question - I get that something too fast could be considered DDoS, but so long as it’s below a certain threshold wouldn’t it be okay (not sure how said threshold would be determined)?

In the US, CFAA prohibits causing "damage", which includes "impairment to the integrity or availability" of data or systems. But as with many other things in law, it boils down to the court trying to assess your intent, whether you could've reasonably anticipated the outcome, and what that outcome ended up being.

There's no law that says "you can't send more than n packets per hour".

I guess they wanted to be crawled because the Google crawler is easy to identify/block/rate limit.

It was fine because we let them.

Google respected rate limits. Reddit wanted the crawl.

Google's entire business model is based on behavior that is now illegal.

Reply on news.ycombinator.com