Arq (Mac backup) now works with Glacier
haystacksoftware.com
haystacksoftware.com
I'm mildly concerned about what could happen if I need to restore software in three years on a new version of OS X (or a different OS entirely), possibly after sreitshamer has been hit by a bus and not updated things.
http://sreitshamer.github.com/arq_restore/
http://www.haystacksoftware.com/blog/2010/04/open-source-arq...
Edit: the files on Github have not been updated in 3 months. So this most likely can't restore Glacier backups.
I use TimeMachine, but would like to have a "my house burned down" option that lives online somewhere...
I agree with some others that the cheapest for now is buying more HD and keeping them in sync with crashplan or something else, until Google Drive drops a bit further in price for 1tb.
You can currently get a 3TB, USB 3.0 TB drive for as little as $120. At $10 a month to store 1TB on Glacier, you'll surpass that cost in a year.
I know Glacier data is likely more safe than moving your files to an external HD and then putting it in your safe deposit box...but the HD ends up being cheaper in the medium-term and far more flexible and faster. It's unlikely your house will burn down and all of your available backups...it's even less likely that that situation will happen and that this last-resort external HD will somehow also be lost, right?
For example, as a photographer, I have roughly 1.5 TB in photo files. The vast majority of these I will never need to access again because the ones that I've liked, I've either stored in other drives (for example, a small drive of wedding photos for my portfolio or for recent customers) or on online services (yes, I know this is a cost in addition to the external drive, but I was using these services already, and for reasons other than just storage).
So for me, Glacier would be a place where I dump a chronological store of photos, a first-in-last-out kind of system. There's no need to regularly sync any of these photos. There may be a need to go back through certain folders years from now if I decided to compile something for a family album or something bespoke like that. And for that reason, it's necessary to have a last-resort store for these files.
So if I have 1TB of files that need to be rarely accessed and never updated, it's hard to justify spending that on Glacier (assuming I have a pretty safe external place to store this drive). At the rate that I accumulate files, it should only take me about an hour a year to backup photos. (plus the time it takes to find the hiding place for the hard drive).
Online storage services are useful for continual syncing, which using an external drive would be inconvenient for (if that drive was stored off site). Glacier is for files that you aren't likely to access again for months-or-years-on-end, unless there's an emergency or you're doing something bespoke, like making a family photo album and need to go back through all your files again.
So filling up an external drive and keeping it off-site and using Glacier is not as much apples and oranges.
I'm planning to loop back and investigate how restoration would work, but it seems like a potentially great way to have another layer of backup.
Does those $10 include the servant which does the backup and drives the hard-disk off-site for you automatically, as would be the equivalent of an automated job with s3cmd/rsync (and whatever you have for arq and glacier)?
If not so, you are missing something crucial in your equation.
The backup time and storage for my use-case is mostly one-off. I don't have a need to rsync a folder containing photos from 2008 because I will not be making critical updates to that folder.
When you're thinking like that, you need to ask yourself what 1TB actually means. Your USB disk the the equivalent of a VHS tape in 1980... it holds a few hours of video, but to keep the quality of the original, you need to maintain it. Will a USB drive in a safe deposit box work in 20 years? Will you have a USB3 interface handy in 20 years?
Digital media is really tough to archive over long periods of time vs. paper. I worked with a US State archivist on a project, and they stuggle with the issue. They have a good handle on the papers of the colonial government from the 1600's, but have a difficult time accessing the early digital records from the 1980's.
So putting stuff in glacier, and letting Amazon handle storage migration, break/fix, replication, etc for $120 per TB/year sounds like a value to me.
Amazon's service gives you an expectation that 99.999999999% of your data will survive intact on average every year. So while you need to maintain your data logically, you can set clear expectations with respect to the integrity of that data.
The fact that Amazon's infrastructure may not be around in 20 years isn't especially relevant either -- neither will any maintained infrastructure. Tapes need to be refreshed, archival storage infrastructure usually ages out in a 8-12 years, and you need to do a TCO analysis each time to refresh.
The other thing is that you get physical isolation. Flooding hasn't been a problem in lower Manhattan or Staten Island in recent memory. How many safe deposit boxes do you think were destroyed in those two places recently?
Compare this with S3, where you will lose 1 in one billion files per year. Shocking!
The presentation of documents are critical as well. Will complex documents created in Word 95 render properly on the iPad XXXVI in 2040?
Archival file formats like PDF/A, open specification formats like OOXML and open source formats like ODF will help with this issue, but I'm sure there will be many audio and video files that will be unintelligible in 30 years.
If you use it as a backup strategy and you're going to need access to all of your data then you're going to have to extract it from glacier in a relatively short time period. This might cost you an arm and a leg (relative to the storage costs)!
See: http://brandt.github.com/amazon-glacier-calc/ for an online calculator.
If your use case matches the price structure in that most of your data is write and forget and occasionally you'll need a small piece of it back then it's a fantastic service & as last-ditch backup options go it's a whole lot better than nothing at all!
(edit: not sure that calculator is entirely useful since it appears to squeeze the entire download into a single 4 hour period.)
In particular the assumption that "Retrieval time is how long it takes Amazon to fulfull your request. Retrieval time has nothing to do with your download speed." seems to be the root cause.
Plugging in numbers from Amazon's worked example from https://aws.amazon.com/glacier/faqs/#How_much_data_can_I_ret... gives different numbers than that web page.
My estimate for downloading 1 TB continuously for 1 MB/sec is a retrieval fee of approx $25 in addition to data transfer fees.
Plug: I am working on a startup (submitted to this YC round!) to solve this problem using user^Wcustomer-owned hardware for precisely the sort of reasons you describe. I am looking for co-founders. If anybody wants to talk, email is in my profile.
Plug #2: I wrote and submitted this article about Glacier yesterday but it sank fast: http://psranga.github.com/articles/possible-architecture-of-... Email me if you want to talk about going up against an 800 lb gorilla. :)
Cloud storage is awesome in many ways. Yet it doesn't replace your backup strategy, it merely complements it.
http://www.reddit.com/comments/hg9oa/your_platform_is_on_aws...
... that a single provider is really just a single "copy".
It is also the reason that we build 's3cmd' into our environment and so many customers use it:
ssh user@rsync.net s3cmd put abc.txt s3://account/abc.txt
Is this something that people should be using on their old files/backups?
I'm too lazy and don't really do it with my multiple DVD, CDs and HDD backup directories. But ideally, I should be doing it. My startup will make this sort of thing easy and automatic.
That's unsettling. Source?
1 TB = 2^40 bytes
Amazon claims 99.999999999% durability. https://aws.amazon.com/glacier/faqs/
2^40 * (1 - 99.999999999/100) = 10.99 bytes
In other words: shouldn't you calculate the loss over the total number of bits?
First, you would need more than one hard drive. You would need to buy at least two hard drives(even two may expose you to more risk than you desire). Even if you do opt for just two, and do manual mirroring, it now takes two years to catch up with glacier.
Second, $240 upfront is more costly in this environment than putting $10 up per month for 24 months. Glacier costs will very likely fall over two years, and that fact will be reflected in your statement at the end of the month. HDD costs will fall over the two year period, you can't utilize any of these lower rates, since you've already sunk your capital into two outdated hard drives.
Third, glacier is designed to have data constantly written to it, and only infrequently read from it. This is difficult to pull off with external hard drives that you have locked in a safe at the bank.
Of course, this makes sense for me because my backup is rather tiny (~300 GB). The "unlimited" plans at backup services are worth it in certain scenarios I guess.
I paid for a year, so I'm using it now as additional backup, with my primary backup on a local HD and Arq.
It works more or less the same as Glacier, only with a lot of cushion around it.
--- edit: Disclaimer: I am the owner of Bit Chest.
Concerning the price, it is indeed a multiple of pure Glacier. However, they include, apart from lots of convenience, also all auxiliary costs that you'd have to manage yourself for a "pure" Glacier account. As noted in another comment, you'd have to keep an eye on retrieval cost, early deletion and additional S3 storage for metadata.
If you're not interested in tinkering with IAM keys and retrieval rates, I still think Bit Chest is a good value proposition.
Disclaimer: I am the owner of Bit Chest.
I prefer to use the space I have on Amazon S3, under my own account with them, which I pay directly to them, than using some other obscure, third party, close beta service.
What is "ct" on your pricing? Just curious.
I understand that most people on HN will probably use Glacier directly, and that is totally OK. The service that I am offering is not "making it possible", rather it's "making it simple, so much so that your grandmother can use it". You do use Dropbox in the same way, don't you? Also, the reason it's obscure is that is less than a week old and has less than 40 users. I wouldn't expect any offering of that age/size to be well-known, so I'm quite satisfied with my obscurity.
ct is cents. For Europe, that will be Euro cents, all other areas will be billed in USD cents.
A shame, because TM seemed to work perfectly in Snow Leopard.
I don't think that alone merited a major version bump[1], but understand that the developer needs to feed his family, so will gladly pay for the upgrade.
[1] The ability to create the backup bucket, and not have it auto-create, for example, would be something that I would consider a major version bump.
"In the coming months, Amazon Simple Storage Service (Amazon S3) plans to introduce an option that will allow you to seamlessly move data between Amazon S3 and Amazon Glacier using data lifecycle policies."
If you go over that it's more expensive but they prorate it based on how much free data you get and how fast you were going while you downloaded it.
Assume you stored 1 TB (1024 GB) and need to retrieve all of it.
Free allowance per day = (1024 * 0.05)/30 = 1.70 GB/day
Free hourly transfer allowance = 1.70/24 = 0.07 GB/hour
Billable hourly transfer rate = 3.51 - 0.07 = 3.44 GB/hour
Retrieval fee = 3.44 * 720 * 0.01 = $24.76
Based on info here: https://aws.amazon.com/glacier/faqs/#How_much_data_can_I_ret...
Bumping up the speed by a factor of 10 will bump up the retrieval fee by the same factor, so $250 + $120 = $370 total restore cost in a little over a day's time.
Hm. I wonder if you can bill the insurance company for this in case of fire etc.
Alternatively, it would be nice if Amazon allowed several accounts to pool together their retrieval allowance - it's not likely that all of my friends will have their house burn down at the same time.