Hacker News Reader

Comment by nness | Hacker News Reader

Internet Archive is having its own legal troubles at the moment, might no longer be the archival safe-haven we'd hope...

View on HN

Storage is cheap, if there’s something you’re interested in it’s time to become a data hoarder.

Datahoarder here, the drives are relatively cheap, but the overall cost to do it right is expensive. I have a somewhat durable setup (about 370TiB raw, replica count 3, 5 node Ceph cluster). And just that before backups I work total cost to be about $1000CAD per usable TiB. I don't shuck drives though, that could probably bring it down to 800.

So you've invested six figures into a private data hoarding setup? Not bad.

I guess it is around six figures by now. Better not let my wife realize that, lol.

3 replicas as in 3 complete copies of the data? Wouldn’t something would erasure coding be cheaper?

Erasure coding is only used for local duplication. If you have cross-site or cross-machine copies, you really don't want the speed of the rebuild to be dominated by available network bandwidth.

For many datahoarding things “when I drive over to my friend’s house and retrieve the drives” is often enough for replicating restore speed.

I guess it depends. I run minio with erasure coding but I also have 2x100gb interconnects between nodes and even with NVMEs I’m not noticing any performance issues with rebuilds. That said with most disks I could see how this could be a problem

I wish I had 100Gbps interconnects! My network is only 10Gbps.

There are three copies of the data as part of ceph's replication. Note this is not replication for backup purposes, erase a file and all objects composing the file are gone. Ceph replication is for comparing checksums and that for guaranteeing integrity.

There is archival quality data hoarding like you’re doing, and then there’s “this YouTube channel is interesting and I don’t want to lose it, yt-dlp to a USB drive”.

Not being able to do the former doesn’t mean you shouldn’t do the latter, especially if the risk is “real”.

(My system is relatively cheap as it only consists of a ZFS mirror array and some offline drives; a major fire might take me out but the stuff that should survive that is offsite.)

For sure, usb sticks have their purpose! I keep secondary copies on other more temporary media of home videos and pictures, the actually important stuff.

Our neighbourhood just recently got fiber to the home, so I will be going from 30Mbps upload to 940Mbps upload in a couple weeks. Then I can get some proper offsite backups going.

Can you share depth and breadth of your archive?

My archive is composed of cloud images and volumes for virtual machines for an OpenStack homelab. Then around 90TiB of movies, shows, YouTube videos, etc. Remuxes when available. About one TiB of books.

Do I use it? My wife watches media on jellyfin, but I just have a compulsion to gather data and hoard it. I restrict myself to a 36U server rack at least, so it doesn't take up much physical space.

I have no judgement, I was just curious.

I didn't interpret you as judging. It is somewhat pathological I admit. The cluster is a bit more than 87% full, at 90% it slow to a crawl. I just asked my wife this morning if I could buy another hard drive (I buy 20TB seagate exos), and she suggested we delete some stuff. I find deletion of files when I do it psychologically stressful. So it is something I should get addressed, but I suppose we should be thankful I don't have dead cats lying around, lol.

I say you have saved it this long keep going. That drive seems to cost around $350. I use SSD drives for working storage and I use platters for archival. I believe the platters have a better recovery methods.

Do you have any opinions?

Thanks for more information.

I use enterprise SSDs for cloud images and VM volumes, I have about 7TiB useable of that. I don't know how data recovery from an SSD is, I think for a lot of issues you would have to replace the logic board and hope some sorcery works to figure out the ordering of data with wear leveling algorithms. Rust one can often recover data with something like ddrescue. I don't worry too much, with replica count of 3 for both flash and rust I can lose two drives with the same 4 MiB block and be fine, if the failing drives are within a chassis I can lose two whole nodes before I have data loss.

I wish I had access to a tape library though, just it is hard to justify the upfront cost. My cluster has organically grown over the last ten years or so, making a huge near $8000 single purchase for a new tape drive, plus however much automatic tape rotation costs (I presume I need some sort of tape library for that? I have never used tape before) would be hard to justify.

You can get used tape drives but no matter how hard I’ve tried to make it work it just really doesn’t, not for this stuff.

You can spend that money on way more drives and increase the number of copies you have on and offline.

As for automation the libraries aren’t much more than the drive, but you can also just manually load tapes if you had to.

$1000/TiB ?!

Why 20x overhead over raw storage cost?

May as well use

That is all in, including motherboards, RAM, chassis, CPUs, etc. then replica count three as opposed to erasure coding. That replica count alone is 3x the cost. And TiB, not TB, and Canadian prices, with shipping from the US for Supermicro parts. Each node is about 6-7000 CAD before drives and I use 6 enterprise rust and two enterprise SSDs per node. Cost would be lower of I built nodes with more drive capacity, but I want to keep nodes to 2RU to have more nodes in 36RU.

shuck was repurposing external drives popping them out and using the internal drive?

That is correct.

Storage is cheap, but moving it is expensive.

Reply on news.ycombinator.com