Donate to the Internet Archive

> archive.org's copy goes down.

From asking around they do retain the original data, but exclude it from public results following a robots.txt exclusion. It's something I do wish they would re-consider though as it limits the purpose.

Just for the record, we are an archive, just one with some policies (that I myself have been working to have tuned) that you might not like. I understand that you don't like the policy.

The issue is that archive.org exists in a quasi-legal gray area. Imagine that you decided to selectively archive comic strips that are accessible online and created a "Comics Library" site. I expect you'd get some cease and desist letters.

Archive.org has mostly gotten away with what they do based on the fact that they try to be comprehensive, they don't post ads or charge for access, AND that they won't display your site if you ask them (through robots.txt). Yes it's frustrating but barring some legal ruling that cements their right to archive and offer access to copyrighted works, I don't expect it to change.

A million times this.

So much content has been lost and sadly the Internet Archive is not an archive whilst this policy exists which evicts all historical content when a updated robots.txt is found.

I have been for quite some time been donating 1TB-2TB/month of bandwidth in support of Jason Scott and the ArchiveTeam. This has been my way of supporting the Archive.org project and would consider increasing the bandwidth donation in exchange for resolution to the robots.txt bit-rot.

I won't link to the leaderboard but for reference I'm currently working on the TwitPic project and am within the top 10; same username as HN.

As someone who once helped maintain the exclusion-mechanisms, my personal opinion is:

The retroactive-robots.txt policy made sense originally as a way of reducing risk from angry-rightsholders, while minimizing burdens on staff time, and had little downside when the history-of-the-web was short, and most domains were still under their original ownership. It was a toggle any webmaster could throw, with no support/maintenance effort required at IA.

It obviously sucks now, more than a decade after it was adopted as a quick fix, but would take some dedicated policy and technical design to gracefully replace. For example, many rightsholders may be relying on the old behavior. But, the IA hasn't yet been able to prioritize creation of a new scheme.

A new process could involve a policy where someone claiming to be the original site/content rightsholder asserts that as of some boundary date, for example when they ceded or sold the domain, later robots.txt should not affect earlier content. (Such a boundary could also be clearly-indicated in Wayback summary/calendar pages.) Then, presumption would flip to showing the earlier content, unless some other rightsholder (such as the current domain-holder) formally claims ("under penalty of perjury", etc) that it's their material and they wish the block to stay in place.

It'd be a bit in overall shape like the DMCA takedown/counter-notify procedure, as if the robots.txt was a sloppy takedown request. Squatters making a false claim to ownership of older content would be forced to go on record and take some risk. Ideally any dispute could then proceed between the other two parties, leaving the IA out of it. Again, this would be similar to how the DMCA tries to leave ISPs/caches/hosters out of takedown legal battles.

I doubt most new domain-owners are strongly or consciously trying to hide the past; it's usually an automatic choice made for other reasons. A few might be holding the history hostage to make ownership of the domain more valuable – "buy it to re-access the past". Some others might be embarrassed of the previous unaffiliated content – but a clearer Wayback UI indicating changes-of-management could help allay that concern. Unfortunately, for both good and bad reasons, there is no easy/reliable/canonical source of all domain-ownership info over time.

I don't know about mobile (I've only noticed a small banner on at the bottom of the page on WP), but it appears they're using the exact same style of banner that Wikimedia use? They even explicitly give thanks to Wikimedia in the footer of their fundraising banner. I think both services are great, so I don't really mind if either of them decide to optimize their conversion rates, just like any other startup would.

Their 2012 990PF: http://207.153.189.83/EINS/943242767/943242767_2012_09c9e75b...

http://www.eri-nonprofit-salaries.com/?FuseAction=NPO.Summar...

That's how I read BYTE. It's also a good source/reference for (some (specific)) WP articles.

https://archive.org/details/byte-magazine

http://en.wikipedia.org/w/index.php?title=CHIP-8&diff=585383...

http://en.wikipedia.org/w/index.php?title=Halt_and_Catch_Fir...

Thanks, we changed it from https://archive.org.

Except _internet_history_ is no-one's to control; once something is on the internet, it belongs to The People.

You can get your site taken down, if that's your worry. But like generally on the internet, your best recourse is to control your online presence at the source. Don't post regrettable things publicly, and you should be fine. If other people do it to you, your issue really would be with them; fortunately, if something terrible is done, you've the Archive to prove the damage and back you up.

It really is a different beast from your normal panopticon internet tracking. This is more akin to showing up in the newspaper, rather than a surveillance camera.

[0] https://archive.org/about/faqs.php#29