Hacker News Reader

Comment by j_baker | Hacker News Reader

This may be a dumb move from a legal perspective. Court cases have alluded that robots.txt files may count as technological measures in DMCA cases[1]. Granted, that's far from guaranteed. But I certainly wouldn't want to be the one to go to court over it.

[1] http://www.groklaw.net/article.php?story=20070819090725314&#...

View on HN

In Germany, same situation. There was a lawsuit against google's image search. The highest court (BGH) has decided that not taking technical measures against having images indexed counts as an agreement. Therefore, ignoring robots.txt is potentially dangerous under german jurisdiction. http://kuerzer.de/bghthumb

AT does many, many illegal things. They copy and redistribute content without permission of the owner-- indeed, that is the entire point of the Archive Team.

AT does many, many illegal things. They copy and redistribute content without permission of the owner

I don't know if AT does or does not do anything illegal but what you have outlined is not illegal per se. archive.org and Google do it too.

But Google do allow an opt-out and follows what AT consider outdated methods of data protection. Doesn't seem that AT is the same.

The old it isn't illegal because bob and joe do it too defense. The behavior of others don't change the legality of your behavior.

While I think the AT stance amounts to arrogance more than anything, enforcing compliance with robots.txt with law is absurd and unjust. To the best of my knowledge it's not a part of the HTTP standards or codified in any law.

Don't be naive. People get sued over things that aren't codified in any law or part of the HTTP standard every day.

This is a bit of a legal grey area. The kind of thing that BigCos with expensive lawyers like to sue over. Would you really be willing to pay lawyers to settle it in court once and for all?

I think you misunderstood. I wasn't commenting on the likelihood, I was commenting on the justice of it. I'm not speaking as a practitioner of law (I am not one), I'm speaking as a subject of the law and it's equally my responsibility to respond to injustice.

Suing is not about what is lawful or not; suing is part of the civil court, not court of law. Anyone can sue anybody for any reason and the judge has the responsibility to decide if any wrongs were performed, and if they deserve compensation. So it isn't about what is lawful, it is about what bullshit they can convince the judge of, so getting a clued in judge is what you need.

The point you're missing here is that as there isn't a legal standard for this, robots.txt has arguably become a de-facto standard. It's widely understood that the things you mark as off-limits to crawlers in robots.txt should not be indexed or reproduced in automated fashion.

Think of the potential alternatives. One is everything can be crawled with no way for the site owner to say "no, please don't crawl this."

The other major one is to legally bar all crawling without express permission.

The current de facto world - crawling is OK unless robots.txt says otherwise - is pretty nice. If we want that to be a legal defense in court ("You didn't put up a robots.txt, so my indexing was legal, so you can't sue me."), which seems useful, then the necessary flipside of that is that violating robots.txt exposes the crawler to liability. That's a tradeoff I'm perfectly willing to accept to allow the web, and necessary services such as indexers and crawlers, to work while still allowing publishers to have some reasonable control over their content distribution.

I seem to remember one of the writers at Search Engine Land presenting a nice description of the robots.txt request in contract negotiation terms. Something like this:

Archiver: "Are there any limits on what I can archive or index from your site?" (translation: GET /robots.txt)

Site: "Nope." (translation: 404 Not Found)

Site: "Yep, here they are." (translation: 200 OK followed by restrictions in robots.txt format)

So, by asking for /robots.txt, the crawler can be construed as asking permission to index, and the response setting up the terms of indexing. That seems like a really useful defense and sane compromise in this age of "indexing so you can drive search users to our content is copyright infringement."

[EDIT: fix formatting]

I guess the first alternative seems like the more sane one to me, as far as legality goes. Website-crawling policies don't seem like the kind of thing that rises to the level where it's worth involving courts and laws, so I'd leave it to technological mechanisms plus voluntary compliance with non-technological mechanisms. But I suppose I have a pretty high bar for what problems are severe enough to require a government solution.

And pragmatically, the vast majority of the non-robots.txt-respecting crawls I see are coming from countries that won't enforce such laws anyway, so enforcing them in western countries seems like a downside (more entanglement between the internet and various countries' national laws) with little upside (won't stop many crawls).

'Law' is not just written legislation handed down from legislatures and other 'sovereigns'. It's also made on the ground, by custom and consensus, and then recognized as wise and just by the courts in the case of disputes – at least in common-law jurisdictions like most of the English-speaking world:

http://en.wikipedia.org/wiki/Common_law

Robots.txt has a lot of custom and implied authority behind it by now – and as others have noted, some recognition by courts as well.

Reply on news.ycombinator.com