Robots.txt is a suicide note | Hacker News Reader

Robots.txt is a suicide note

archiveteam.org

115 pointspanza91 comments

"Archive Team interprets ROBOTS.TXT as damage and temporary madness, and works around it. Everyone should. If you don't want people to have your data, don't put it online."

Open articleView on HN

This is composed from equals parts of insight and daftness, though not entirely for the right reason.

The daftness: maybe the claim is true that robots.txt was only a stop-gap measure back when web servers sucked, however the de facto modern use for it goes far beyond that, and ignoring that standard is likely to piss off lots of people.

The insight: for crawlers, relying on robots.txt to prevent getting stuck indexing infinite hierarchies of data is a bad idea. It should be able to figure that much out for itself, so it doesn't explode when faced with sites that don't exclude such hierarchies using robots.txt.

For servers, relying on a client hint to ensure reliability is daft. It should have some form of rate limiting built in, as that's the only sensible design. This seems the only marginally sensible use of robots.txt from a server standpoint. Using it for any form of security (e.g. preventing DB scraping) is daft, and a more robust mechanism should be employed there too.

however the de facto modern use for it goes far beyond that, and ignoring that standard is likely to piss off lots of people

I think they're saying is that their archive is going to ignore it, and your website should just eliminate it.

We'll have to assume that archiveteam.org are by now real experts in guessing which buttons are going to be pissing people off. :-)

Archiveteam member here. Robots.txt is useful for a site to avoid having search-engine spiders to blindly wander down an infinite hallway.

Archiveteam projects are different because we run closely-monitored, highly targeted crawls. If a site has an infinite hallway in it, we'll notice that and exclude it while taking steps to replicate enough of it to retain its valuable attributes. We also do our best to avoid retrieving content more times than necessary. If you operate a database, we try to get enough pages to replicate a good portion of the underlying data.

You do try to contact the "targets" sometimes though?

Sure. We ask them to give us database dumps, when they refuse, for some crazy reason, we just download their website.

I can't say I approve of the "if the crazy fools won't give us what we want, we'll just take it some other way" attitude.

As the "some crazy reason" might be that the database contains data that is not publicly addressable for a reason and sanitising the data-set could be non-zero effort operation, letting you crawl the public pages and do some work at your side would be the sensible route from most points of view. Heck, even if there is definitely not anything not already displayed to the public in the DB, letting you go ahead and get the information the hard way is likely to be preferable to lifting a finger to send you a DB dump.

Sort of like a cat's attitude to playing fetch: You want it? You go get it. I've got more important things to do.

"Archiveteam projects are different because we run closely-monitored, highly targeted crawls."

Maybe, but does that make robots.txt useless? I don't think so.

Your app may do the right thing but who's to say another app isn't? Why should someone ignore a tool because Archiveteam claims to do the right thing?

The file has more uses than just blocking a particular URL. What about adding a crawler delay for a site on a shared server that gets slammed sometimes? Or what about blocking a particular user-agent that misbehaves?

If someone shoots themselves in the foot with their robots.txt, it's their foot and their gun.

Fair enough. I've always seen my robots.txt files as a friendly way of saying "you probably don't need to worry about this stuff" to crawlers. That's not your use case, and I'm happy to have my "Disallowed" content to be crawled, I expect poorly written bots to do so anyway. If I actually want to prevent browsing/crawling of something, I wont ask politely I'm just going to throw up 403s.

The great things about 'robots.txt' are (1) it's the simplest thing that could possibly work; and (2) the default assumption in the absence of webmaster effort is 'allow'.

(2) is immensely valuable. Without it, search engines and the largest archive of web content, the Internet Archive (where I work on web archiving), could not exist at their current scales, as a practical matter.

There's a place for ArchiveTeam's style of in-your-face, adversarial archiving... but if it were the dominant approach, the backlash from publishers and the law could result in prevailing conventions that are much worse than robots.txt, such as a default-deny/always-ask-permission-first regime. Search and archiving activities would have to be surreptitious, or limited to those with much deeper pockets for obscuring their actions, requesting/buying permission, or legal defenses.

So, Jason, be careful what you wish for.

I disagree with this post almost as strongly as I agree with it.

Robots.txt is a suicide note. It's utter short-sighted hubris to say "this is MY information and I don't want you spidering it". Are you volunteering to maintain that information forever? Are you promising to never go out of business? Never be ordered to remove it by the government? Never be bought out by Oracle?

Right now there seems to be a lot of confusion over the morality of information. People are possessed by the strange idea that you, mister content provider, own that content and have an inalienable right to control it any way you can get away with. But someday you will die, and your company will die, just like Geocities, Google Video, and the Library of Alexandria. Society should have a right to keep that information after you're gone.

Of course, the law disagrees. And without the efforts of criminals like geohot, the iPhone DevTeam, The Nomad, Muslix64 and, yes, The Archive Team, people of the future will have no way to access the information we've locked up through our own paranoia. You don't have to cast your mind to a thousand years in the future - it's happening right now. Vast swathes of data are disappearing as DRM servers go dark only a few years after they appear (thanks, MSN Music, Yahoo Music Store).

I believe that we owe it to our descendants to give them access to their history. I believe it's not our decision whether the things we make are too valuable or too uncomfortable to be preserved. And I believe that robots.txt is a suicide note, a product of the diseased minds that think our short-term desire for control outweighs our legacy.

But I don't know what the fuck the article's talking about. It seems to be making a bunch of points that don't matter. Use robots.txt to prevent technical problems if you like, I don't care. Just don't use it to stop people from crawling your content or you're shitting on the future.

While I'm sure the "content provider" would disagree, in many cases it's perfectly legal to make a copy of their content - the laws in place only prevent redistribution for a certain time period.

Personally, I'd be ecstatic if there was some organization set up that is manufacturing high-quality archive copies of books, music, film and the like, and storing them with the date of initial publication, as well as the date they will enter public domain (where it is known - not the case for living authors) in various locales. Then have a site live-tracking the release of that content.

There's a site that keeps getting linked every year that posts things that would be public domain this year were it not for US copyright extensions, and something like that with accurate PD status and archive copies would be marvelous.

The rationale is weak. Some data is simply not worth indexing, and not worth serving up to bots. The flipside is: your crawler doesn't need to fetch everything on my site, and I'd be happy to ban all non-conforming bots site-wide.

It's not just about the functionality, but also a show of good faith and basic respect. If you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all.

robots.txt isn't perfect. Ideally a web server would be configured to deny bots access to restricted content via some sort of dnsbl mechanism (or CPAN/whatever module.) Or do both and ban the non-conforming site-wide.

The above notwithstanding, I'm voting for this article. It doesn't betray the usual cowardice by hiding the assertion behind the presumptuous Why.

"Some data is simply not worth indexing, and not worth serving up to bots."

The broader points being made in the article are that the value of information is determined by the visitor, and that the burden of keeping the site up and running should fall on the host.

By that reasoning, the presence of a ROBOTS.TXT signifies "damage" or "temporary madness", and should therefore be ignored.

Also, keep in mind where this article came from - the Archive Team are bloody-minded about preserving information.

OK, wait.

If the data has value to the user, shouldn't the user be paying the host for the cost of making that data available (or perhaps considerably more, if it has a lot of value to the user)?

Excellent. How much are you paying ycombinator for your account?

If the answer is "nothing", then I guess you've just told me that all your comments are useless, and I should ignore them.

I'm committing the classic mistake of talking about the karma system here, but what, precisely, is wrong with my statement?

Drivingmenuts says the user should pay for hosting data. HN is hosting his data, but he doesn't pay for it. Is there some error in my logic, here, some flaw in my conclusion?

the value of information is determined by the visitor

That may be so in a sense, unless the visitor has no means of making the call -- or does so poorly. I promise you that every robots.txt ignoring bot makes terrible judgments in that regard. (I'm not trying to overstate anything here; those that abide it make, on average, slightly less terrible judgments.)

But it's not just the visitor who gets to make judgment calls. The value of serving those pages is something the host can decide. Belligerent drunks who abuse the staff aren't allowed in the coffee shop, and if they do it often enough they're not allowed back in when they sober up either.

If it's public, it's public. Don't put private stuff up in public.

It's not about that. Explicitly so. Did you read the thread you're responding to?

the Archive Team are bloody-minded about preserving information

They won't do a good job of that if they encourage too many people to make IP blocks of their machines.

AT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.

Do all of them ignore robots.txt?

Well, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to:

  User-agent: * 
  Disallow: /

Does that count?

1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/

Here's a concrete example of a site not worth archiving - back in the early days of the web (1994?) a friend of mine wrote a 4 function calculator. Each button was a hyperlink, so "34-12" meant going to the pages "?", "?3", "?34", "?34-", "?34-1", and "?34-12". That's an infinite number of web pages which should not be indexed.

I remember some anti-spam vigilante code that told legitimate bots to go away with robots.txt, told humans what the game was about, and attacked e-mail address scrapers by presenting an infinite number of fake e-mail addresses. "Click here for 1000 more e-mail addresses that will never bother a human!".

I love it!

Not all of them were fake. Some of them were spam traps which never appeared anywhere else. Some of those spam traps were also coded so you could track down when they were served to connect the spiders to the spewers.

At least, that's the way I used to roll.

Ah very good! I was trying to think of a short example that wasn't contrived with this very behaviour.

If you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all.

Quite. My robots.txt reads:

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /scratch/

I do not believe, as the archiveteam.org seem to think I do, that I have either stopped bots interfering with CGI scripts or spidering semi-private, temporary content. Indeed, my server logs show clearly enough that bots do these things all the time. Instead, the bot has violated my site's policy and its IP address will be considered for an iptables ban. It is a matter of which bots I consider fair game for rough treatment.

This is a common attitude to robots.txt, I think, and it does not reflect well on archiveteam.org's competence that they seem to be unaware of it.

Yeah what this really needs is a bot block filter service where bots that do not conform to expectations can be bulk banned. It is too much effort to do this manually but a larger service can work.

This may be a dumb move from a legal perspective. Court cases have alluded that robots.txt files may count as technological measures in DMCA cases[1]. Granted, that's far from guaranteed. But I certainly wouldn't want to be the one to go to court over it.

[1] http://www.groklaw.net/article.php?story=20070819090725314&#...

While I think the AT stance amounts to arrogance more than anything, enforcing compliance with robots.txt with law is absurd and unjust. To the best of my knowledge it's not a part of the HTTP standards or codified in any law.

Don't be naive. People get sued over things that aren't codified in any law or part of the HTTP standard every day.

This is a bit of a legal grey area. The kind of thing that BigCos with expensive lawyers like to sue over. Would you really be willing to pay lawyers to settle it in court once and for all?

I think you misunderstood. I wasn't commenting on the likelihood, I was commenting on the justice of it. I'm not speaking as a practitioner of law (I am not one), I'm speaking as a subject of the law and it's equally my responsibility to respond to injustice.

Suing is not about what is lawful or not; suing is part of the civil court, not court of law. Anyone can sue anybody for any reason and the judge has the responsibility to decide if any wrongs were performed, and if they deserve compensation. So it isn't about what is lawful, it is about what bullshit they can convince the judge of, so getting a clued in judge is what you need.

The point you're missing here is that as there isn't a legal standard for this, robots.txt has arguably become a de-facto standard. It's widely understood that the things you mark as off-limits to crawlers in robots.txt should not be indexed or reproduced in automated fashion.

Think of the potential alternatives. One is everything can be crawled with no way for the site owner to say "no, please don't crawl this."

The other major one is to legally bar all crawling without express permission.

The current de facto world - crawling is OK unless robots.txt says otherwise - is pretty nice. If we want that to be a legal defense in court ("You didn't put up a robots.txt, so my indexing was legal, so you can't sue me."), which seems useful, then the necessary flipside of that is that violating robots.txt exposes the crawler to liability. That's a tradeoff I'm perfectly willing to accept to allow the web, and necessary services such as indexers and crawlers, to work while still allowing publishers to have some reasonable control over their content distribution.

I seem to remember one of the writers at Search Engine Land presenting a nice description of the robots.txt request in contract negotiation terms. Something like this:

Archiver: "Are there any limits on what I can archive or index from your site?" (translation: GET /robots.txt)

Site: "Nope." (translation: 404 Not Found)

Site: "Yep, here they are." (translation: 200 OK followed by restrictions in robots.txt format)

So, by asking for /robots.txt, the crawler can be construed as asking permission to index, and the response setting up the terms of indexing. That seems like a really useful defense and sane compromise in this age of "indexing so you can drive search users to our content is copyright infringement."

[EDIT: fix formatting]

I guess the first alternative seems like the more sane one to me, as far as legality goes. Website-crawling policies don't seem like the kind of thing that rises to the level where it's worth involving courts and laws, so I'd leave it to technological mechanisms plus voluntary compliance with non-technological mechanisms. But I suppose I have a pretty high bar for what problems are severe enough to require a government solution.

And pragmatically, the vast majority of the non-robots.txt-respecting crawls I see are coming from countries that won't enforce such laws anyway, so enforcing them in western countries seems like a downside (more entanglement between the internet and various countries' national laws) with little upside (won't stop many crawls).

'Law' is not just written legislation handed down from legislatures and other 'sovereigns'. It's also made on the ground, by custom and consensus, and then recognized as wise and just by the courts in the case of disputes – at least in common-law jurisdictions like most of the English-speaking world:

http://en.wikipedia.org/wiki/Common_law

Robots.txt has a lot of custom and implied authority behind it by now – and as others have noted, some recognition by courts as well.

AT does many, many illegal things. They copy and redistribute content without permission of the owner-- indeed, that is the entire point of the Archive Team.

AT does many, many illegal things. They copy and redistribute content without permission of the owner

I don't know if AT does or does not do anything illegal but what you have outlined is not illegal per se. archive.org and Google do it too.

But Google do allow an opt-out and follows what AT consider outdated methods of data protection. Doesn't seem that AT is the same.

The old it isn't illegal because bob and joe do it too defense. The behavior of others don't change the legality of your behavior.

In Germany, same situation. There was a lawsuit against google's image search. The highest court (BGH) has decided that not taking technical measures against having images indexed counts as an agreement. Therefore, ignoring robots.txt is potentially dangerous under german jurisdiction. http://kuerzer.de/bghthumb

This is a childish argument based upon an attitude "don't use robots.txt because it interferes with what we do and what we do is aw3s0m3 l337". This attitude is also prevailing in the archiveteam's comments here. I doubt their actions can be taken seriously.

I wonder how this made into 88 votes here on HN..

The Archiveteam.org favicon is a hand making a rude gesture. I think that sums up many peoples opinion to this story.

It certainly is an indicator of how seriously you should take this organisation.

Am I missing something? It looks to me like it's making a 'stop', with the hand held up and flat.

Their attitude can be summed up as "it's on the internet, it's ours to take". Ok, oversimplified, but that's the essence, no?

So, dear archiveteam, please remember that when I put a server on the internet, it's a voluntary and public service, and putting 'Disallow:' lines in the robots.txt means that I set some rules. It's just rude to ignore those rules, whatever you motivations are.

You have no right to access my content, just as you have no right to walk into my house. If I invite you, please behave.

The analogies here always end up all over the map, but the walking-into-a-house one seems pretty off. The service is set up for anonymous public access, run by an automated process designed to service requests, and faces a public thoroughfare. If we have to make IRL analogies, that sounds closer to a vending machine or other kiosk on the side of a road. In that case, you probably wouldn't have much faith in whether people will follow any instructions you tape to the machine.

Sorry, but this is terrible advice. Yes, you should make sure your site won't break if it's slammed by a large crawl. Yes you should hide destructive actions behind posts, not gets. But, robots.txt is insanely useful. If I didn't have a robots.txt file, google/bing/yahoo would index countless repetitive non-important files and my site would suffer in search engine ranking. In our case, we host GPX/KML files and textual cuesheets for driving and biking. If that stuff is indexed, our sites' relevant keywords are "left", "right" and GPS timestamp fragments like "0z01".

So, use it wisely, but don't abandon its usage altogether.

robots.txt is simple and effective.

I do not want certain bots, especially so-called "archives" to automatically download all my content. And that's what robots.txt is for and works well.

The article is just stupid, sorry. There is not one real knowledgable argument.

that's what robots.txt is for and works well

Robots.txt doesn't stop anyone doing anything, it is simply a policy. Bots either respect it or they do not. Archiveteam.org have indicated that they wish to join the side of the spammers and incompetent spider authors.

are you confusing archive.org and archiveteam?

Thanks, fixed. Actually I was not, that was a typo.

I agree. archiveteam.org have stated they will ignore robots.txt. This makes me disrespect their organization. Well, that and their ridiculous website.

Wow. What arrogance.

A Good reason to honeypot if you aren't already.

It's expected. It's polite. Respect the site owner's published policy or expect to get IP banned like any other script kiddie because when a site admin see you ripping content, he isn't thinking "yay! archive team is here to do a free backup!" he thinks you are stealing his shit.

I love the tone of this article, I was smiling the whole time. Especially here-

the onslaught of some social media hoo-hah

edit: just clicked through a few pages- whoever does the writing at Archiveteam is fantastic!

Jason Scott, very distinctive voice. Maker of the BBS Documentary, etc. Engaging public speaker too; lots of good examples on his wikipedia page: http://en.wikipedia.org/wiki/Jason_Scott_Sadofsky

Point of order: A bunch of us[1] maintain the wiki. Jason just does the big, visible things, and is thus big and visible.

http://ascii.textfiles.com/ is his personal blog, and is as profanity-ridden as you would expect.

1: http://archiveteam.org/index.php?title=IRC_Channel

I wonder if honeypots that auto-block rogue crawlers have occurred to these yoyos.

We typically use wget running on consumer machines. This isn't like the spider farm of a typical search engine, with hundreds of machines consuming gigabytes/second of bandwidth. AT hits are specific and targeted.

BTW, robots.txt disables access to versions of pages already archived on the Wayback Machine. I encountered this when looking for old technotes on developer.apple.com.

The wayback machine is the only reason I use robots.txt:

    User-agent: ia_archiver
    Disallow: /

There is no good reason to let old (and possibly embarrassing) website designs distract from the current version of your website.

Few users prefer the archived versions; only when they need information that has disappeared (or is temporarily unavailable) do they try the Wayback Machine.

Many, many web designers are delighted to be able to see their old creations – or even recover past work from the Archive when all other copies are lost to organizational upheaval and system problems.

Finally, note that by blocking archiving, you're opting out of history. Those who look back to see the web of today won't see your contributions. Alternatives, competitors, and other projects that are open to archiving will be seen and vividly remembered. Being invisible to the future may be OK for your projects, but many others prefer to part of the shared memory.

(FYI: I work on web archiving at the Internet Archive.)

I feel like creating a honey pot for bad bots now. Put an exclude line in ROBOTS.TXT and then include that URL in my pages and when a bot hits it anyways, ban the IP.

How would you know if the robot is autonomous or not? They could just use a typical browser user-agent and you wouldn't know...

Hide the link from users?

I seem to remember there being a thread on HN about spam bot scraping for email addresses, and some of them are sophisticated enough that they can even determine if an element has been hidden through rules applied via an external CSS file.

Link a 1 pixel image and stick it in the bottle left hand corner? While someone could click on it, it is extremely unlikely.

IIRC, technically you can have a 0 pixel image in some formats (gif) though I imagine some browsers won't like that.

Using a 1 pixel transparent GIF works - also worth putting in a warning for screen readers not to follow the link.

FWIW I've been using robots.txt honeypots on two sites for the past few years to deter scrapers. Approx 1,600 IP addresses have been banned. Most of these are for data centers.

Fuck you guy, that's pretty pricky. Is bandwidth that expensive for you?

That's uncalled for. If he owns the data and pays for the connectivity, he should be free to do as he pleases just like I can exclude you from my home or business without justification.

I like humans browsing my sites but I certainly don't want scrapers copying my work wholesale, claiming it as their own and ranking ahead of me in SERPs.

I'm not alone :-) see http://incredibill.blogspot.com/

This is what Project Honeypot does http://www.projecthoneypot.org/faq.php#f

Sounds like risking the consequences of blackhat SEO with little advantage; give one version to a typical search crawler, the other version to non-official crawler. The search engine assumes it's a bait-and-switch and automatically blacklists the website.

That is not what zbowling has been describing.

I don't really understand why they compare the Robots.txt file to a suicide note. Could anyone explain this?

ArchiveTeam mostly flashmob archives sites that are going to die an rm -rf / death, or atleast kill their users content.

Like Geocities, Yahoo! Video, Google Video, Friendster

No, it is about not being willing to waste bandwidth and server capacity on unworthy projects (no person will ever search for my site through baidu but it still being indexed).

Google and archive.org is one thing, I will be happy to support them.

So a little while ago we had a story which was essentially: "promote your startup / website by causing maximum outrage! Outrage is good! YAY PISSING PEOPLE OFF!"

Now we get a non-story which is essentially designed to piss off the people of HN. Looking forward to more of the same given it works.

Well, sure, robots.txt is not the best solution, but it works and helps a lot when you got msnbot or yandexbot that takes more than half of the requests of your mediawiki (differences between revisions), your gitweb (commitdiffs) or your phpBB installation and kills the performance...

Bored of having our machine killed by those bots, we use some robots.txt.

Sure, there are other solutions (proper blocking), but this one works perfectly fine and avoids having to modify 3rd party applications that we are running for an open-source development team.

The only thing I have ever used robots.txt for is to stop from leaking pagerank. I have a folder called /redirect/ and i exclude that folder in my robots.txt. I then link to sites like this /redirect/?l=www.mysite.com

Anything I don't want archived, I put behind a login wall.

Some parameter are useful, but they are not part of the standard.

e.g. Crawl-Delay, prevent DDOS from YAHOO! Slurp

Is Yahoo! Slurp still active? I thought they turned it off when they switched to using Bing search results on Yahoo!

I don't think so. I still receive a lot of requests from Yahoo! Slurp / Yahoo! Slurp China this month.

I saw a few hits from Slurp with Y!/USA IP just 2 days ago in my logs.

Dear "archiveteam", I pay by the MB. How do I opt out of your shit? KTHXBYE

Don't accept user-generated content. Failing that, never go down, and never announce that you're shutting down.

Or, if you do, hand over the data in a nicely archivable form, and prevent the need for crawlers.

shrug

Things like /search?q= are in my robots.txt because crawling those pages is just a waste of everyone's time and resources.

Then you should block their crawler :)

Reply on news.ycombinator.com