Hacker News Reader

Comment by yhager | Hacker News Reader

Not completely by the specification, but I think this one works as expected.

    user-agent: *
    disallow: /

    user-agent: Googlebot
    allow: /

View on HN

I think that userbinator's point is that the DO conditionally allow scraping which makes their position even more tenuous IMO.

By a single search engine which probably provides the vast majority of their traffic.

While I don't necessarily agree with the concept of only allowing google to index your site, comparing a search engine which feeds you business to a company reselling your data with no attribution is not really fair in my opinion.

Neither of the two parties mentioned is likely to resell the data directly, both are likely to to create a derivative work which they will exploit commercially as will Google. What part of the ToS make one OK while and the other forbidden? How does this work when a smart scraper can just pull from the Google or archive.org cache?

Not sure it's a good idea to block all search engines but Google.

Probably not, but if they don't want to be scraped, the first place they should notify about this is robots.txt. I was just stating an example that you can allow some bots and not others if you like.

Of course, forging your user agent, disobeying robots.txt or scraping after you were told not to, is wrong ethically.

Reply on news.ycombinator.com