Robots.txt Disallow: 20 Years of Mistakes To Avoid

How is IA supposed to distinguish a new website from a sincere wish to delete old stuff? A change in domain registration data means nothing; I have a domain that I registered for an association in my name, and which I then sold to them (for a symbolic price), but it was only an administrative issue - the site was the same.

IA is on iffy territory w.r.t. copyright as it is; if they stop respecting robots.txt, they could get into a world of hurt.

Internet Archive isn't wiping out historical content. It's just unavailable/hidden for the time being. (As long as there is a restrictive robots.txt available).

As I pointed out elsewhere in the thread, making analogies to physical media is just as flawed as the "you wouldn't steal a car" anti-piracy campaigns.

A physical library is either getting their newspapers by asking/paying the newspaper company to deliver them, asking citizens to donate them, or collecting them from already delivered newspapers. If the IA was just piggybacking on user activity (by caching and storing things from a user's browser cache after they visit a page) then I'd have far less of a concern with them. If we're so attached to physical metaphors, this would be equivalent to the librarian running around outside the newspaper's printing room and snatching newspapers from the bundles as the company's employees loaded them onto trucks.

Things disappear from the internet, for good, all the time.

Also, if people become aware someone is naive they automatically are dicks to them? Regardless of wether that information is actually of interest to anyone, just because someone wants to take something down, they should not ever be able to?

Some people act and think like that that, yeah. But to accept this as the baseline of human behaviour is, well, not for me. This entitledness to watch the lives of others from the the dark may have been bred by reality TV or whatever; but it's more a personality flaw and an addiction, a useless misfiring of synapses become culture, than a cornerstone of an information age.

There are two sides to this argument. I argue both. All the fucking time.

If you're in the business of providing public content that's well-known, to the public, then allowing it to be archived makes a lot of sense.

If you're providing user-generated content I'd argue that the case for allowing archival is extended even more so. Sites that violate this, and Quora comes specifically to mind, are violating what many, myself included, consider to be part of the social contract of the Web.

On the other hand, if you're an individual, and you are posting your own content and ramblings, and circumstances change for whatever reason: you've got a job, you've lost a job, you're married, you're divorced, you're getting divorced, your child is at war in a foreign country, a foreign country is at war with yours, or you're just sick of the crap you wrote when you were young and arrogant and now and old and arrogant you wants it gone: I'm pretty willing to grant you that right.

If you've committed some terrible crime against humanity, or just a human, and have been fairly tried and convicted of it, I'd probably not give you the right to remove large bits of that information.

And yes, there are vast fields, deserts, tundras, plains, steppes, ice-fields, and oceans of grey about all of this.

Barbra Streisand got Streisanded because she is Streisand.

Ahmed's Falafel Hut likely wouldn't suffer the same fate. His Q-score is somewhat lower, and there's only so much real estate in the public consciousness.

I still don't see the relationship between deleting a blog post that you have authored and burning a library full of other people's works down.

Okay, I completely agree about newspapers, maybe other things as well. But you said people who ban IA from their site could be criticized, there are plenty of other comments along those lines, and I just don't see it. As advice, sure: if you post it on the internet, assume it will stick around forever, because that could happen. But still, there are personal public websites, if you know what I mean. They're not secret, they're not hidden, they are accessible to the public -- but they do not belong to the public, they are not like a public park or road. And sometimes, a website is more thinking aloud, or talking to oneself, than writing a book that then belongs to your "audience".

Civility is a good keyword, and while this may be a bit of a stretch, imagine sitting in public cafés and writing down what people say, and then criticizing people for lowering their voice and turning their back so you can't read their lips, even though you genuinely mean well, and just want to preserve daily public life for future historians. In general, this is what this attitude of "the internet" feeling entitled to whatever was ever posted anywhere feels to me. Maybe I just don't get it, but I really don't get it.

I think the question wether a private conversation should be recorded just because it's in public, just because you can, is kind of a no-brainer, but here are ones I don't have an answer to: Should an artist be allowed to make a performance and ask it to not be recorded? Should someone be able to hold a political speech and ask the same? For me the answers are kind yes, and no-ish... but what about political art? Are we allowed to try to influence people, and then try to erase the traces? Now that is tricky, and I may have ended up ranting myself into agreeing more with the IA "side" of the argument than I expected to. Because either something is personal, trivial in one way or another, or commercial and/or political. Personal things I think should be respected, but commercial and political things shouldn't be, they do belong to historians. Well, fuck.

[this is why I "blog" bit, actually -- because posting stuff online makes me think harder about them than I would otherwise, I don't even need an actual audience for that, just the possibility of one -- but that's also why I don't feel great about all of that floating around forever, it's all rather temporary in nature, a process.. and the person who wrote stuff a year ago does not exist anymore, so why should the name of this current person be attached to it?]