Hacker News Reader

Comment by mthreat | Hacker News Reader

Has Rupert heard of /robots.txt?

Are you saying that if your content doesn't have a robots.txt, then everyone is allowed to copy the content of the site with no restrictions whatsoever, and that copyright law doesn't apply?

I'm not a big fan of Murdoch, but it's an interesting discussion that should go beyond the obvious technical arguments.

Imagine the world with strict copyright. Search engines would be illegal!

I guess I'll ask again, differently. What do you mean by strict? Are you saying that today, there is an implicit understanding among a class of people (judges, CEOs of web 2.0 companies...?) to not apply the law strictly? That doesn't sound very likely.

If Google started copying a full news site verbatim and offer its content when people search for news, that would definitely be wrong, robots.txt or not.

Strict copyright includes fair use exceptions and so on. Where fair use stops and infringment starts is an interesting discussion for me.

You get moderated downwards for that comment? What is this, Slashdot?

Uh, Google doesn't do just "anything" with the content, they index, summarize, and link back to the original article. There's no obvious copyright violation there.

The GP was hyberbole that doesn't add anything to the debate. Ironically that's exactly what happens all the time on Slashdot.

Yes, I got modded down. I didn't list my credentials on the topic, so maybe I should have. I did get sued and went through significant $$$ expenses to defend frivolous lawsuits, so anyone who blankly states "oh, use robots.txt and don't worry about a thing" sounds like a beginner in the intricacies of the law to me.

That's kind of a red herring, because Murdoch wants his stuff to be copied by google regardless of how that argument is resolved.

It's not a particularly onerous requirement and far cheaper than calling your lawyers.

If you're making the argument that the company that is automatically spidering your content should stop then your case is somewhat undermined by failing to take the simple steps to prevent aforementioned spidering.

  Are you saying that if your content doesn't have a
  robots.txt, then everyone is allowed to copy the content
  of the site with no restrictions whatsoever

No, he is saying that if your content doesn't have a robots.txt, then everyone is allowed to index the content of the site with no restrictions whatsoever. After all, you are publishing it to the world. The fact that technology and automation are involved is irrelevant: the process is identical to me telling you that paper newspaper X had an article on Y in it's Jan 23rd, 2005 edition.

The point is not that he wants to cut Google off, it's that he wants leverage.

Simply updating robots.txt won't effect Google at all and will blow what little leverage he has left.

This is exactly the first thing that comes to my mind when I read about "indexing" in the article.

Reply on news.ycombinator.com