Hacker News Reader

Comment by LoSboccacc | Hacker News Reader

beware: robots.txt can retroactively clear archive.org data

Hopefully not anymore: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

To clear things up: robots.txt can retroactively hide content from the archive. If it's changed back to allowing the archive's crawler, content from before the ban can be accessed again.

another reason to use archive.is

What is the difference between these two?

archive.is does not crawl automatically, it must be pointed at a page by a user. While this makes it particularly useful for snapshotting frequently-changed pages, it is not a replacement for the proper Internet Archive.

perma.cc is another option as well.

archive.is is very nice, but they're a URL-shortener as well, so their links are utterly opaque strings of alphanumerics, whereas the Wayback Machine preserves both the full original URL and the date and time it was captured in the archival URL.

Considering the topic of discussion, how sure can you be that archive.is will still be around in a year? Three years? Ten?

As much as I tried, all I could find about it is that it's run by one guy in Czech Republic who's paying $2000/month out of pocket for hosting, and apparently dislikes Finland.

It's actually worse - http://blog.archive.is/post/151510917631/how-do-you-guys-kee... says it's $4k/mo.

http://archive.is/robots.txt doesn't seem too bad, it looks like you could slowly inhale everything... in theory. There are no sitemaps (they're there, but empty placeholders); you have to know the site name to be able to get a workable list.

The author hasn't ruled out/blocked archiving the snapshots, but apparently it's... big. http://blog.archive.is/post/154930531126/if-someone-was-will...

I think http://www.webcitation.org/ might be better in that regard since it's a consortium of "editors, publishers, libraries". See "How can I be assured that archived material remains accessible and that webcitation.org doesn't disappear in the future?" in their FAQ (http://www.webcitation.org/faq). Although from my perspective it seems to be more geared towards academic use.

Reply on news.ycombinator.com