Introducing Google Cloud Storage Nearline

For disaster recovery 3 days might mean the end of your business.

That would be more expensive than just storing 1TB with the normal, full speed Cloud Storage service.

Or use Nearline for archival data as well as disaster recovery. You won't need to transfer the archival data for normal disaster recovery scenarios, so it'll be -- for that scenario -- extra data that boosts your speed, but it's still, on its own, a useful use of the service.

I understand, but please go stand in the corner there and think about what you've done.

The use case is people who have a requirement to store data and probably never need to access it, either for reg requirements, or legal requirements, etc. In which case paying a couple of hundred bucks for retrieval is fine, especially if they can access at a high level of granularity

edit: Also as others have pointed out, Nearline is limited in retrieval time as well, so the cost difference isn't nearly as large.

You assume Glacier uses tape storage... But the same principle still applies, they're trying to prevent people clogging up the network so when somebody does need to do an emergency restore there is lots of spare network capacity.

Note: The following is speculation best I know. I'm recalling from memory what I've read on the internet written by someone who did not have a direct source.

Glacier uses low-speed (5400 RPM) consumer drives, which they then clock down lower to save energy. Any given 'rack' only has enough power to power a few drives on that rack, the rest is powered down.

To prevent multiple customers from all trying to pull their data out they needed to introduce a rate limiting system, which they did with this exorbitant pricing.

In a word: yes.

You're thinking about data that can't be lost, or your customers are screwed. Not all data is like that.

Log files come to mind. They're _nice_ to archive for a long time, but in many businesses, they're certainly not _critical_ to archive for a long time.

Intermediate files, too. You retain the original files in secured storage. But because the intermediate files are large and expensive to re-create, you keep them here in AWS.

I'm thinking (unrecoverable data loss / total data stored) in their system per year.

It's 11, not 12, I misspoke.

Those claims are from glacier: http://aws.amazon.com/glacier/faqs/ which is priced the same per GB of data.

In this case, the durability refers to the loss of data/objects stored per year - if you're sending multiple PBs of data off to Glacier, you want to be able to retrieve them many years later. Even 5 9s would mean that 1 object out of 100,000 is lost every year, which is quite poor.

This is a durability metric, not an availability one. This typically tells you the likelihood of losing a given object in a year. (5 9's would be quite poor for this, implying a loss of 1 object out of 100,000 every year)

It was 11, not 12. I misspoke: http://aws.amazon.com/glacier/faqs/

In this case, it's not as much service uptime as it is data retention. If you're storing 5+ PB of data, even 5 9s of data loss per year can have a measurable impact.

If you'd like to put 10PB in storage, let's talk offline :)

source: SE at Google Cloud

It's safe to assume this is going to be just like Glacier and S3/Google Storage, ie: unlimited.

Also retrieval speeds increase with your data set size: Note: You should expect 4 MB/s of throughput per TB of data stored as Nearline Storage. This throughput scales linearly with increased storage consumption. For example, storing 3 TB of data would guarantee 12 MB/s of throughput, while storing 100 TB of data would provide users with 400 MB/s of throughput.

"Well, what do I have to hide?" is a bad argument with respect to whether the NSA should be doing the things it does, but with respect to whether I am going to spend money and effort to hide my photos and documents from them, I think it's a fair argument.

I care very much, however, whether hackers will have access to my files, as they can cause havoc with things like tax returns that the NSA won't (I mean, the government already has my tax returns....).

So I was mainly curious if there was some significant flaw in Backblaze's encryption that should worry me from the perspective of a non-nation state adversary.

It's much more flexible in terms of external drives, and deleted files.

You can also set multiple backup locations, by sharing space on your other computers.

They have a proprietary Java client, but it runs cross platform. It's completely painless to add your own encryption key to all backups.

Haven't used it, but Duplicati is another one: http://www.duplicati.com/

While I personally don't mind Google having all my backup data, we must appreciate the fact that in this post-Snowden world some people will justifiably refuse to go with the big USA-corp players like Amazon or Google.

rsync.net just saying: "Hey, we're not Google or Amazon" is already a big selling point for some.

I encourage you to email us and ask about the HN readers discount.

Further, Amazon S3 is a better comparison, as far as pricing goes - we're fully live, online, random access storage - not nearline or weirdline like google/glacier.

Google Cloud Storage is cheaper than Amazon S3:

http://aws.amazon.com/s3/pricing/

https://cloud.google.com/storage/pricing

Setting up your own with any of Ceph, Gluster, Riak, Swift or similar depending on your specific needs. Heck, I've got setups where we've been served well for years with a combination of inotifywait, grep and rsync (combined to trigger rsync instantly on modification events). It really depends on your access patterns, and there are lots of potential savings from making use of domain specific knowledge for your specific system.

In general, you can beat AWS with 3x replication across multiple data centres even with renting managed servers, especially as your bandwidth use grows as AWS bandwidth prices are absolutely ridiculous (as in, anything from a factor of 5 to 20 above what you'll pay if you shop around and depending on your other requirements). Lease to own in a colo drops the price even further.

Anything from 1/3 to 1/2 of AWS costs is reasonable with relatively moderate bandwidth usage, with the cost differential generally increasing substantially the more you access the data.

Most people also don't have uniform storage needs. If you use your own setup, people tend to be able to cut substantially more in cost by reducing redundancy for data where it's not necessary etc.

See also the Backblaze article that mentioned Reed-Solomon - if your app is suitable for doing similar you can totally blow the S3 costs further out of the water that way.

Consider that there are labor costs around operating services on AWS as well, and it's generally more expensive by the hour, and labor cost of managing your own servers is fairly low.

I have plenty of servers sitting in racks that haven't been touched in 5+ years; if you were to operate on a "buy and throw out in a year" practice, my typical labour costs would average a couple of hours per new server for setup. Or if you rent managed servers, you don't ever need to touch (or see) the hardware.

Conversely, you can get bandwidth at a tiny fraction of the cost of AWS bandwidth, so the moment you actually transfer data to/from your setup, AWS gets progressively more expensive.