Robots.txt is a suicide note (2011)

Copying my comment from a previous discussion of ignoring robots.txt, below. I actually don’t care if someone ignores my robots.txt, as long as their crawler is well run. But the smug attitude is annoying when so many crawlers are not.

————

We have a faceted search that creates billions of unique URLs by combinations of the facets. As such, we block all crawlers from it in robots.txt, which saves us AND them from a bunch of pointless indexing load. But a stealth bot has been crawling all these URLs for weeks. Thus wasting a shitload of our resources AND a shitload of their resources too. Whoever it is, they thought they were being so clever by ignoring our robots.txt. Instead they have been wasting money for weeks. Our block was there for a reason.

Not sure the emotive language is warranted. Message appears to be “if you use robots.txt AND archive sites honor it AND you are dumb enough to delete your data without a backup THEN you won’t have a way to recover and you’ll be sorry”.

It also presumes that dealing with automated traffic is a solved problem, which with the volumes of LLM scraping going on, is simply not true for more hobbyist setups.

This is a screed that does not address a single point of the actual philosophical issue.

The issue is a debate between what the expectations are for content posted on the public internet. There is the viewpoint that it should be totally machine operable and programmatic and if you want it to be private you should gate it behind authentication, that the semantic web is an important concept and violating it is a breach of protocol. There's also the argument that it's your content, no one has a right to it, and you should be able to license its use anyway you want. There is a trade off between the implications of the two.

I think this is kind of misguided - it ignores the main reason sites use robots.txt, which is to exclude irrelevant/old/non-human-readable pages that nevertheless need to remain online from being indexed by search engines - but it's an interesting look at Archive Team's rationale.

I absolutely will use a robots.txt on my personal sites, which will include a tarpit.

This has nothing to do with keeping my webserver from crashing, and has more to do with crawlers using content to train AI.

Anything I actually want to keep as a legacy, I’ll store with permanent.org

Any time I think about robots.txt, I think about a quote from Pirates of the Caribbean. [1] "The only rules that really matter are these: what a man can do and what a man can't do." except that I replace man with bot. Everything should be designed to handle pirates, given the hostile nature of the internet.

To me, robots.txt is a friendly way to say, "Hey bots, this is what I allow. Stay in these lanes including crawl-delay [2] and I won't block you." Step outside and I can put you on an exercise wheel. I know very few support crawl-delay since it is not part of the standard but that is not my problem. Blocking bots or making them waste a lot of cycles or get dummy data or wildly reordering packets or adding random packet loss or slowing them to 2KB/s is more fun for me than playing Doom.

[1] - https://www.youtube.com/watch?v=B4zwh26kP8o [video][2 mins]

[2] - https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directi...

given this is from a group determined to copy and archive your data with or without your permission, their opinions on the usefulness of ROBOTS.TXT seem kind of irrelevant. of course they aren't going to respect it. they see themselves as 'rogue digital archivists', and being edgy and legally rather grey is part of their self-image. they're going to back it up, regardless of who says they can't.

for the rest of the net, ROBOTS.TXT is still often used for limiting the blast radius of search engines and bot crawl-delays and other "we know you're going to download this, please respect these provisions" type situations, as a sort of gentlemen's agreement. the site operator won't blackhole your net-ranges if you abide their terms. that's a reasonably useful thing to have.

I see old stuff like this and it starts to become clear why the web is in tatters today. It may not be respected, but unless you have a really silly config (I'm hard-pressed to even guess what you could do short of a weird redirect loop), it won't be doing any harm.

> What this situation does, in fact, is cause many more problems than it solves - catastrophic failures on a website are ensured total destruction with the addition of ROBOTS.TXT.

Of course an archival pedant [1] will tell you it's a bad idea (because it makes their archival process less effective)—but this is one of those "maybe you should think for yourself and not just implement what some rando says on the internet" moments.

If you're using version control, running backups, and not treating your production env like a home computer (i.e., you're aware of the ephemeral nature of a disk on a VPS), you're fine.

[1] Archivists are great (and should be supported), but when you turn it into a crusade, you get foolish, generalized takes like this wiki.

Is a person not allowed to put up a "no trespassing" sign on their land unless they have a reason that makes sense to would-be trespassers?

I know that ignoring a robots.txt file doesn't carry the same legal consequences as trespassing on physical land, but it's still going against the expressed wishes of the site owner.

Sure, you can argue that the site owner should restrict access using other gates, just as you might argue a land owner should put up a fence.

But isn't this a weird version of Chesterton's Fence, where a person decides that they will trespass beyond the fenced area because they can see no reason why the area should be fenced?

Not having things archived because you explicitly opted out of crawling is a feature, not a bug

Otherwise you can whitelist a specific crawler in robots.txt

This wiki page was created in 2011, in case you're wondering how long they've held this position

Related:

ROBOTS.TXT is a suicide note - https://news.ycombinator.com/item?id=13376870 - Jan 2017 (30 comments)

Robots.txt is a suicide note - https://news.ycombinator.com/item?id=2531219 - May 2011 (91 comments)

If you don't obey someone's robots.txt, your bot will end up in their honeypot: be prepared for zip bombs, or generated infinite recursions and whatnot. You better have good countercountermeasures.

robots.txt is helping you identify which parts of the website the author believes are of interest for search indexing or AI training or whatever.

fetching robots.txt and behaving in a conforming manner can open doors for you. If I spot a bot like that in my logs, I might whitelist them, and feed them a different robots.txt.

I think it is not so right. It can be useful to exclude some kinds of dynamic files, files with redundant pieces in URLs (e.g. a query string for files that do not require it; if you could reasonably do it (which in many cases you unfortunately can't), you might make it never crawl URLs with a query string), to set crawl delays, etc. (You might also want to e.g. use other ways of mirroring some files; e.g. for a version control repository, you do not need to crawl the web pages and you can just clone the repository instead. This way, you will only need to clone the new and changed files and not all of them.)

Robots.txt should not be used for preventing automated access in general or for disallowing mirrors to be made.

Someone else wrote "I actually don't care if someone ignores my robots.txt, as long as their crawler is well run." I mostly agree with this, although whoever wrote the crawler does not know everything (but neither does the server operator).

In writing the specification for the crawling policy file for Scorpion protocol, I had tried to make some things more clear and avoid some problems, although it is not perfect.

robots.txt is the digital equivalent of "one piece per person" on an unwatched Halloween bowl.

The people who wouldn't don't need the sign, the people who want to do it anyway.

If you don't want crawling, there are other ways to prevent / slow down crawling than asking nicely.

Set up a tarpit, put it in the robots.txt as a global exclusion, watch hilarity ensue for all the crawlers that ignore the exclusion.

Whatever we think of archive.org's position, modern AI companies have clearly taken the same basic position. And are willing to devote a lot more resources to vacuuming up the internet than crawlers did back in 2011.

See https://news.ycombinator.com/item?id=43476337 for a random example of a discussion about this.

My personal position is that robots.txt is useless when faced with companies who have no sense of shame about abusing the resources of others. And if it is useless, there isn't much of a point in having it. Just make sure that nothing public facing is going to be too expensive for your server. But that's like saying that the solution to thieves is to not carry money around. Yes, it is a reasonable precaution. But it doesn't leave me feeling any better about the thieves.

The real question is that why haven't we moved robots.txt to DNS TXT records? This way the consent to scrape would be crystal clear even before they connect to the server.

Ugh. Yeah, this misses the point: not everyone wants their content archived. Of course, there are no feasible technical means to prevent this from happening, so robots.txt is a friendly way of saying "hey, don't save this stuff." Just because theres no technical reason you can't archive doesn't mean that you shouldn't respect someone's wishes.

It's a bit like going to a clothing optional beach with a big camera and taking a bunch of photos. Is what you're doing legal? In most countries, yes. Are you an asshole for doing it? Also yes.

robots.txt takes as assumptions that are well-meant, and respectful to the site intentions, major players, that try to index/mirror sites while avoiding overwhelming it and accessing only what is supposed to be freely accessed. Using a visible user-agent, having a clearly defined IP block for doing those scans, and a method of scanning goes in the same direction of cooperating with the site owner to both get visibility while not affecting (a lot) functionality.

But that doesn't mean that there aren't bad players, that ignore the robots.txt, give random user agent strings, or connects from IPs from all the world to avoid being blocked.

LLMs has changed a bit the landscape, mostly because far more players want to get everything or have automated tools to search your information on specific requests. But that doesn't rule out that still exist well-behaved players.

It's a shame that most companies that ignore robots.txt also operate horribly behaved crawlers. I feel like they forfeit the moral high ground to render judgement.

Every website operator should be required to learn _rate-limiting_ as well as how to create a robots.txt file

What matters is rate not "user-agent"

User-agent can be fake but rate is real

Renaming my robots.txt to reeeebots.txt and writing a justification line by line on why XYZ shouldn't be archived is now on my todo. Along with adding a tarpit.

Robots.txt was adopted on the trust that people working would respect others like themselves and follow the directives. Robots.txt worked for a while,

the archiveteam statements in the article are sure to win special attention, i think this could be footgunning, and .IF archiveteam .THEN script.exe pleasantries.

Interestingly, not having a robots.txt pointing to a sitemap is considered an SEO error by tools like SEM Rush.

Its mostly for search engines to figure out how to crawl your website. Use it sparingly.

(2011)

I have more complaints about this shitty article than it is worth. At least it's clearly a human screed, not LLM generated.

Just say you won't honor it and move on.

> Precisely one reason comes to mind to have ROBOTS.TXT, and it is, incidentally, stupid - to prevent robots from triggering processes on the website that should not be run automatically

Counter-point: I have a blog I don't want to appear on search engines because it has private stuff on it. 25 years ago I added two lines to robots.txt file, and I've never seen it show up on any search engine ever since.

I'm not pretending nobody has indexed my blog and kept a copy of the results. I'm just saying the blog I started in college doesn't show up when you search for my name on Google, which is all I care about.

I mean the main reason is that robots.txt is pointless these days.

When it was introduced, the web was largely collaborative project within the academic realm. A system based on the honor system worked for the most part.

These days the web is adversarial through and through. A robots.txt file seems like an anachronistic, almost quaint museum piece, reminding us of what once was, while we stoop head first into tech feudalism.