Hacker News Reader

> I don't like working on projects that are actively pitting me against someone though.

I'm surprised you didn't also didn't mention the ethical problems with trying to take something (search results) without permission.

Your new project sounds awesome, though. : )

View on HN

Google doesn't exactly tie itself in knots that they built a few hundred billion dollars on top of "We're going to crawl the entire Internet and datamine it for our own purposes. Nobody will agree to this, so we won't ask them. Instead, we will offer easy ways to opt out after having achieved hegemonic control of Internet navigation."

Though to be fair, that's how all the pre-Google search engines worked too.

There have been easy ways to opt out since day one. Google has always respected robots.txt.

Meh. I didn't have any qualms with that, per-se. As others have pointed out I think there are ethical concerns with what Google is doing itself.

Also it's indicative of something if people are building businesses to scrape something they could offer as a pay-for API that many people would gladly / happily pay lots of money for. The arguments for NOT doing that are ridiculous because people will get the data regardless.

Reply on news.ycombinator.com