Hacker News Reader

Hacker News

Comment by ez_mmk | Hacker News Reader

Full thread

ez_mmk

Can somebody put this on the Internet Archive?

View on HN

On its way via ArchiveBot:

http://archivebot.com/ https://wiki.archiveteam.org/index.php/ArchiveBot

I've grown wary of relying only on IA

  [lammy@popola#Webrips] du -hs www.rr-fallenflags.org 
   43G    www.rr-fallenflags.org
  [lammy@popola#Webrips] find www.rr-fallenflags.org | wc -l
    479221

ArchiveBot regularly grabs sites of that size and uploads them to IA, its not a big deal. The Imgur/Reddit archiving was at least an order of magnitude or two larger.

Of course it is always good to have multiple copies of a site, especially personal ones for things you care about.

Will that work? The site mentions that it's not indexed by search machines. So I guess it has a robots.txt (on my phone and did not check...) Would the archive respect that?

Annoyingly, Internet Archive doesn't respect robots.txt. I specifically excluded ia_archiver from my site which worked for a number of years until they decided to ignore it because robots.txt "do not necessarily serve our archival purposes." They do remove your site if you email them though.

https://www.digitaltrends.com/computing/internet-archive-rob...

Personally, I'm with the Internet Archive on this one. If they were to respect robots.txt, it wouldn't be long before a whole host of websites exclude the Internet Archive for dubious reasons such as lost advertising revenue, copyright concerns, exclusivity deals etc. I am curious to know if you've found the Internet Archive's activity to be exceptionally taxing on your servers, or whether you have another reason to wish to exclude them?

Mainly I just feel I should be in control of the sites I make. They're personal in nature that I don't mind sharing with the world, but if I want to change something or make them disappear completely it irks me that there's a website out there violating my express wishes.

How interesting! That's a completely different way of looking at the Web; I don't think I've thought about it like that before. I view the Web as a kind of library where you can add books as well as borrow them.

When I read something online, I sort of feel that it becomes a part of me as it informs me and shapes my perspectives. I like to think that I could re-read it if I ever forgot the details; as a result, I've downloaded quite a few websites. Common facts don't really apply here for me, as they'll be accessible as long as encyclopaedias exist, but personal anecdotes and niche catalogues are worth their metaphorical weight in gold.

Additionally, I download things that I one day hope to read, but think that the website might disappear before that time comes (due to the author not renewing a domain name etc.)

Reply on news.ycombinator.com