Hacker News Reader

> Web site owners chose to make it available to Google.

Strong disagree. Since robots.txt is optional and the default is "crawl me as you please", website owners don't "choose to make it available", they just don't choose to make it non-available.

View on HN

That's a functionally meaningless distinction. If you setup a web server that responds to requests, then you're choosing to make content available because your server can choose to not respond to requests. The entire protocol includes mechanisms to negotiate access.

Granting access and granting right to redistribute (even just title + snippet) and use your content commercially are two completely different things.

And yet it is legal to produce and redistribute summaries as sufficiently transformative derivative works, and this has been court tested[1]. Of course in Australia we passed rather specific laws to the contrary, because lo and behold Rupert Murdoch wanted money and gosh darn it our government was going to give it to him[2].

[1] https://www.practicalecommerce.com/Search-Engines-Indexing-a...

[2] https://www.alrc.gov.au/publication/copyright-and-the-digita...

This is a meaningless simplification. In this framework "robots.txt" has no role, because your server "can choose" not to respond. Heck, even DDOS is fine, because "protocol"

Reply on news.ycombinator.com