Ice cold archive storage

I need more information than this post provides before switching archival solutions.

Actually, since it’s google I likely wouldn’t consider them regardless.

I've never tried it, but I know https://www.arqbackup.com/ supports Google Cloud.

AWS's new Glacier Deep is actually cheaper than Google's Ice Cold, $1/TB/month.

https://aws.amazon.com/about-aws/whats-new/2019/03/S3-glacie...

One thing I do consider a real value add for AWS Glacier though has been their native support for offline media import/export. Ie., you can just send them a hard drive of your own for data load, and pay to get a hard drive back out as well. As gigabit (or faster) class WAN slowly spreads this will someday become unnecessary, but right now in many, many places a company could easily have terabytes to backup but 10/1 ADSL as their best available connection. Even with faster connections, aggressive data caps are sadly not infrequent. Whether it's for initial load, ongoing use or faster recovery, sometimes there is still nothing like a multi-TB drive or two in the mail.

There are 3rd parties that will do it for you (Iron Mountain is at least one) but that's an extra cost and Google takes no responsibility for it. I assume this is an example of a place where Amazon is able to leverage its wholistic business, with a Cloud service that can also take advantage of their physical logistics system. Google's service here is quite significantly cheaper and has some nice features though, but even if it's not worth a $4/$1.23 premium for Amazon I could definitely see continuing to pay Amazon some premium ($2 vs $1.23 say) for that alone anywhere with limited high speed WAN availability.

Borg Backup is mostly the same as Restic (regarding dedup / incremental backup) [1] and aggregates data into large chunks.

If you only backup from a single machine it has a local cache of already backed up data, this has the large advantage that it basically only needs to push the delta data to the remote, not do any kind of synchronization to check what is already there or not.

[1]: https://stickleback.dk/borg-or-restic/

What tool do you use to do your backups? rclone?

I've been using duplicati for some time. It works ok, not perfect. Wish I could send backups to multiple locations especially (eg local/B2)

The place where this won't be as cheap as Backblaze is retrieval. Unless Google makes a big change, you'll still have to pay for network egress, which is obscenely priced: https://cloud.google.com/storage/pricing#network-egress

The title seems like a reference to the 2003 Outkast song Hey Ya! https://www.youtube.com/watch?v=PWgvGjAhvIw

Alright alright alright!

FWIW, this is still an ideal model for backup storage: If your more regular backup model is robust and your network is well-secured, you'll never need retrieval. And if you need it, you need it, and it's justifiable to spend big to save your business.

Backblaze wrote a good piece on that https://www.backblaze.com/blog/cloud-storage-durability/

Nothing in engineering is 100%. As far as engineering goes 11 9s is pretty much as good as you can get. For comparison, AWS S3 and Glacier are 11 9s durability too.

It's worth bearing in mind the difference between durability and availability. Durability is roughly the chance of losing your data over a given time span (in this case a year), whereas availability is about how reliably you can access the data (and is almost certainly a lot lower than 11 9s). A service can be very durable but have very poor availability.

"What is the meaning of the claim about "99.999999999% annual durability"?"

It has no meaning whatsoever. Someone on the marketing side of the team decided that was a "competitive" number to present, outwards, and someone in engineering was tasked with, working backward from that number, coming up with some plausible calculation that resulted in it.

In the real world, they, like Azure and Amazon, will have single point in time outages that will wipe that out for a year or more.

Here is what an honest assessment looks like:[1]

"Historically (updated April, 2019) we have maintained 99.95% or better availability. It is typical for our storage arrays to have 100+ day uptimes and entire years have passed without interruption to particular offsite storage arrays."

...

"In the event of a conflict between data integrity and uptime, rsync.net will ALWAYS choose data integrity."

[1] https://www.rsync.net/resources/notices/sla.html

"Fast" is relative here. It's fast compared to Glacier and others, but it's going to be slower than the more expensive tiers.

You might not need to speculate much about how it works, it's probably implemented as described by Google themselves in slides 22ff. here:

http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...

(In a nutshell, pack some hot data with a lot of cold data on many large drives, then put a Flash-based cache in front of them to get long tail performance predictability back.)

You have to keep the data in that class for a certain period of time. That's the drawback. You can access the data at that price as long as you keep it there for a long time.

So I suspect this is not a fully cold storage. That's why they can retrieve the data faster. Seems more like an economics hack (Longer commitment to keep the data, allows them to buy and operate the storage hardware/software at a cost that can be amortized against those commitments)

Will that storage have "11 9’s annual durability" and stored in multiple location?

Let say that you only need to write to it once, have 2 secure location available for free, that would still means that you need 2 of them which would pay for itself in 28 months then.

Sure it's "cheaper" but it's far from being as good and the price difference isn't that big.

If you can use cp to move files around, you can use gsutil to do the same for GCS.

https://cloud.google.com/storage/docs/gsutil

In case you're still thinking about options, I would be fine to host a variant of this for you: https://old.reddit.com/r/DataHoarder/comments/7rjcdn/home_ma...

Encrypt the backups and store the encryption key in a normal non-archival bucket.

Encrypt backup data with a per-user key, keep the keys only in hot storage, delete the key when a user is deleted.

I was also looking for that, the only piece of info about that was:

Unlike tape and other glacially slow equivalents, we have taken an approach that eliminates the need for a separate retrieval process and provides immediate, low-latency access to your content. Access and management are performed via the same consistent set of APIs used by our other storage classes, with full integration into object lifecycle management so that you can tier cold objects down to optimize your total cost of ownership.

It seems to have the same pricing like the other storage classes: No fees for accessing the files within the same region and the typical bandwidth fees if the backups will be downloaded to somewhere else.

Eh... for consumer stuff, sure, and perhaps even for new/experimental GCP features. But this is storage, a core function, on GCP, an enterprise service with actual contracts and SLAs attached.