Hacker News Reader

Comment by namdnay | Hacker News Reader

Websites can choose whether to be scraped or not, and google respects that choice

To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.

By creating an account and using their webmaster tools giving up even more of your data?

No you just have to put a noindex header on requests to your website

So they still can use up my resources?

How can they know without looking? Again, you could also use the webmasters tools to block it...

Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.

How dare they!

> robots.txt can accomplish the same thing

Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.

It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.

You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?

No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.

Reply on news.ycombinator.com