Hacker News Reader

Comment by mostlysimilar | Hacker News Reader

archive.org feels like an irreplaceable treasure, the Wayback Machine alone is a time capsule of our digital history.

I donate to them monthly and know a lot of other people do as well, so I don't worry much about their financial stability. I'm more worried about external pressures taking content down. I hope the data is backed up six ways to sunday, and that somewhere there's a plan to make it all accessible if Internet Archive can't continue to play the role it does.

View on HN

I've thought for a long time why I absolutely agree with this sentiment and the best I can come up with is that the Internet Archive feels to me like the embodiment of the old internet, not this marketing driven, data steal-and-sell, VC backed cyberpunk dystopia Internet that we have everywhere else.

It's the same sort of feeling as I why I enjoy this site, wikipedia, and so on. Bonus, IA is also a bit of an organizational mess, but rewards the adventurer with rich treasure.

(I was recently on a Korean history kick and came across not only 1, but several entirely different first hand books written by visitors in the late 19th and early 20th century, scanned in, freely accessible/downloadable, in a variety of formats, and with an excellent on-web reader. These books are so out of print I checked with three local counties for copies and none of them even have references to any of them in their catalog -- treasure!)

I'm very worried about its backups and what happens when the next Big One hits SF.

As far as I've ever been able to determine from talking to anyone at IA (e.g., Kahle, Scott), they don't really have any sort of backups that could actually be restored from in a disaster situation.

There is http://iabak.archiveteam.org, but it’s not exactly large.

If I'm reading that correctly, it would only cost a bit over 500 bucks a month to host that whole archive on BackBlaze B2.

Furthermore it would not be so hard to translate Archive.org items to IPFS objects, if there were an effort to pin a significant number of them to storage and network.

You're reading it correctly, but IABAK backs up 0.2% of the Internet Archive.

Since numbers are not 100% clear...

(50 petabytes * 0.2% = 100 terabytes)

[$0.005 ($/GB/Month) BackBlaze cost]

[(50 petabytes) / (1 gigabyte) = 50,000,000]

(50,000,000 * $0.005 = $250,000 US$)

—————

Meaning based on my numbers, that is $250,000 USD a month to host 50 petabytes of data on BackBlaze.

That's a fraction of the AWS bills for many startups arguably doing absolutely nothing

They have VC money to burn. Archive.org doesn't.

Also, those backups would be (relatively) cheap to keep, but not necessarily to restore.

I would guess restore wouldn't be a problem. AWS or whoever would do it for free given it is a non-profit (in case of a disaster only, of course).

The effort is the issue here. There was this comment back when IA.BAK was in design phase https://news.ycombinator.com/item?id=9148576 And then this is all there was to show for it: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

Yeah, I noticed that being a problem. My biggest problems with IPFS are hardly mentioned in their updates, and it's hard to tell if they have any interest.

I had constant issues with objects simply never (hours and many requests) being found, despite being pinned in several places; and the daemon sucked resources away from the system at an alarming rate back when I was trying properly.

With all that being said, maybe now is the time to look at it properly, the budgets are there. Maybe Juan, _prometheus, can find somebody to at least PoC this important application.

The IA is about 50 PB.

IABAK stores 100 TB, or 0.2% of it.

You can get an 8TB HDD for $150 right now. That's 6250 drives. That's about $1MM in drives, which doesn't sound that cost-prohibitive. Obviously that's not the whole cost since you need to pay for bandwidth, replication and other infrastructure like the host node, but it sounds like something that could be even hosted by a number of volunteers on r/datahoarder or r/homelab.

I also remember reading about Sia on HN, which is a dapp that pays hosts to store data and distributes it. Looking at the going rates on Sia ($1.45/TB/mo), that's $870k/yr. That's ~10% of the IA budget (which is only $10MM/yr, which sounds very efficient!) but shows that the order of magnitude is not that crazy.

BackBlaze B2 is $5/TB/month

Azure Archive is $2/TB/month ($1.68 if reserved)

AWS Glacier Deep Archive is $1/TB/month

GCP Cloud Storage Archive is $1.20/TB/month

Of course, there can be i/o and network charges, and different levels of redundancy (but possibly bulk discounts)...but the bare storage costs for for 50 PB per year would be roughly $600k - $3 MM/y.

The cloud business is a small fraction of Amazon's revenue but a large part of their profits. It's extremely profitable for them. That's why there is such a large discrepancy between (non bulk) HDD price and (non bulk) per month cost for archival.

Aren't the costs of getting that data out of the backup much larger than the cost of keeping it in the first place, to the point that when you actually need to restore a large backup, it turns out it was better to have been managing it yourself? That's the impression I got from various HN comments on the topic over the years.

The cost is only there if you transfer out of aws. Something like glacier will have a retrieval time on the order of hours or days.

Glacier Deep Archive does charge for retrievals at $0.02 per GB and additional $0.01 per 1000 such requests (both of which are $0.00 for Standard S3). PUT, LIST, DELETE are at $0.05 per 1000 requests, 10x the Standard S3 rates.

https://aws.amazon.com/s3/pricing/

AWS has Snowball and Snowmobile though only used former to reduce data transfer costs. Dont remember what other savings are in there. Like is there price reduction if use with Glacier or not.

Isn't that inbound only? Getting the data out again is also required.

You can export data via Snowball as well.

https://docs.aws.amazon.com/snowball/latest/ug/create-export...

Low cost to insert, low cost to keep it there, high cost to retrieve is exactly the combination you want when looking at disaster backup solutions, since you don't intend to retrieve the data frequently. Buy some earthquake insurance (I know, easier said than done) and only pay for 1/20 of the retrieval cost.

The majority of cloud costs of storage are bandwidth, so ignoring this makes the analysis meaningless.

I think at this scale we're no longer talking about buying drives, but securing a steady supply stream of them. So we're talking not $1MM, but a somewhat safe $1MM per year for it to be even worth considering.

> That's about $1MM

is that a "Million Million" eg. 10^12 ?

Some industries use M for thousand. For example, advertising uses CPM (cost per mille), which is the cost for 1000 clicks or views.

In this case, 6250*150 = 937500, almost a million.

Would it remain feasible as they scale up? Their content is growing faster and faster so the number of drives would have to rise every single month, probably by several dozen even.

It definitely stalled out. It needs a Windows version to get real traction IMO.

I wonder what the cost would be to build out a secondary backup site on a different continent.

Do they have distributed backup?

Surely a lot of people would be happy to donate unused space on their drives (I know I would) - especially since backup wouldn’t use much bandwidth...

Whatever happened to their copy in Canada?

https://blog.archive.org/2016/12/03/faqs-about-the-internet-...

Without going too much into it (not my place) it exists.

Thanks, I'm glad to hear it! It sounds like IA is aware of the need to have multiple approaches (and places) to backup as much of the data as possible.

> archive.org feels like an irreplaceable treasure

Yes I save some dead sites that I think are critically important resources to tarballs that I'm storing separately in case the Wayback Machine disappears.

That's useful for personal use. I think that the IPFS is designed for this purpose as well, but meant to be accessible to anyone.

Well the idea is I can provide it publicly (not sure how... don't know how they get the copyright to do it) myself in the future if needed.

While that may work for your personal usecase, there is a significant issue around trust, if you were to share these tarballs with anyone else. I wouldn't trust you to give me originals in the same way I trust the wayback machine. So even if there are backups lying around here and there, their utility is decreased a lot.

By your philosophy, just get off the internet. At some point you gotta trust HN, your keyboard driver, CPU firmware, motherboard, ISP, Google or whatever service you're using. I am not even mentioning your mailman and your amazon delivery driver.

Your stance is pretty extreme - please don't discourage people from making backups.

I don’t think their stance is extreme as you make it seem. Insert their username into the same list you just provided:

im3w1l, HN, your keyboard driver, CPU, firmware, motherboard, ISP, Google

One of these things is not like the others.

They have lots of copyrighted content. Such as virtually every commercial retro video game ever made it seems.

I personally think it’s great but surely companies aren’t too pleased? How does archive.org avoid being sued into oblivion?

It's a non-profit library and they don't publicly provide a way to download most of the copyrighted content, so rights holders aren't too concerned. I imagine most people, even copyright lawyers, personally support the free archiving of everything made in the current age as long as it doesn't detract from current business operations and the copyrights are respected for their duration.

It's very easy to download the copyrighted content. For example, here is a full NES ROM set: https://archive.org/details/NESrompack

As far as I know, they limit public access to basically any material with a complaint or request, but keep copies. They may do more, but that seems to be the default response.

Which seems smart enough: minimize litigation costs while not losing content permanently via court orders or dmcas or similar threats.

But it's frustrating for the wayback machine- where sites like Snopes and some newspapers have opted out of having their history published (after being accused of ghost edits to articles).

I don't know what the right answer is, but even if they didn't display the page, it would be nice to see the diffs (ala wikipedia), or at least show if and when changes to a page were made. Maybe that's beyond their mission scope, though.

> sites like Snopes and some newspapers have opted out of having their history published (after being accused of ghost edits to articles)

Is there any way to interpret this behavior as anything other than straight up admitting to being dishonest? Especially in context of accusations that could be easily disproved using the Wayback Machine... unless there is something to those accusations, that is.

There are exemptions to DMCA for certain content.

> Computer programs and video games distributed in formats that have become obsolete and which require the original media or hardware as a condition of access. A format shall be considered obsolete if the machine or system necessary to render perceptible a work stored in that format is no longer manufactured or is no longer reasonably available in the commercial marketplace.

That doesn't seem to stop Nintendo from shutting down ROM sites left and right. For example, emuparadise has complied and taken down all ROMs: https://www.emuparadise.me/emuparadise-changing.php

But here is a complete NES ROM set, just sitting there, on archive.org: https://archive.org/details/NESrompack

I just find it strange. Nintendo is vehement about their IP. For archive.org to get an exception doesn't make sense to me.

You raise good points. I wonder what a copyright lawyer would have to say.

(IANAL) Circumventing DRM is usually illegal under DMCA. That quote allows circumventing obsolete DRM. NES cartridge ROM data is still covered by copyright though.

I don't know how archive.org gets away with that.

It’s the only organization I actually donate to. Partially because it was so easy to set up a monthly donation ;)

They could set up a torrent that people could download parts of, I think it's an ideal system for a distributed backup.

But it probably needs tweaking for that purpose; for one you need to ensure that the data is evenly distributed, and second you're dealing with data that is appended to regularly.

But I think it can be done.

It exists! The ArchiveTeam had a project, you can see its status here: http://iabak.archiveteam.org/

Here's an overview: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

Sadly, it seems the project has fallen out of repair.

There are only two sites I donate to: wikipedia and archive.org. I honestly don't know what I'd do without them. I have been able to find web sites from 20 years ago on archive.org, it's an absolute treasure.

They should team up with Backblaze...

Reply on news.ycombinator.com