Backblaze Vaults: Zettabyte-Scale Cloud Storage Architecture

Yev from Backblaze here -> It's true, we just have the one. Backblaze was bootstrapped, so we cannot over-expand and maintain profitability, which allows us to stay in business. We're pretty up-front about having the one datacenter. We'd LOVE to add more in the future, but truthfully, it would at least double our costs, and we'd need to raise prices. We're thinking of ways of avoiding that while maintaining our current business-model. If you are looking for something more geo-redundant, take a look at services like Amazon S3, they are great, but the downside there is they charge per GB to make up for the extra costs, so depending on the amount of data, it can get pricey. Either way, we recommend a 3-2-1 backup policy (3 copies of your data, 2 onsite but on different mediums and 1 offsite) as a good start to a backup strat. We're just one solution of many, though, we like to think we're the easiest one!

Brian from Backblaze here, I wrote that comment so if anybody has any questions, fire away!

I think the MOST IMPORTANT THING is that an online backup provider like Backblaze is totally transparent to their customers about their architecture and what we do and do NOT do. If our design is not reliable enough for your particular needs, then you are then able to make the choice not to use us. I never want to mis-lead customers and hide exactly how durable their backup really is.

Finally - what I recommend to my very closest family members and trusted friends is that for data you feel would be catastrophic to lose, I recommend you have at least three copies including the primary copy. That is two separate backups with TWO SEPARATE VENDORS who did not share any code, hopefully managed by two separate UIs. For bonus points, one backup should be "offsite". For example, many Backblaze customers use Time Machine on the Macintosh for a local backup, and Backblaze for their remote backup, and that is what EVERYBODY should be doing. I can show you many support cases where Time Machine failed to restore a file and Backblaze saved the day, and vice versa. The fact is that users make mistakes, UIs are hard to use, your 14 year old son decided to unplug the Time Machine USB hard drive to free up a USB port, or your 14 year old daughter decided to uninstall the Backblaze agent - in other words, stuff happens!!

No I'm not bothered by it as I wouldn't expect replication across multiple data centers at a "$5/mo for unlimited" price point.

I routinely recommend BackBlaze as a quick and easy cloud backup solution to family and friends. I view the risk of partial or complete data loss with BackBlaze extremely low compared to the local backup solutions that some people rely on (i.e. periodic backups to an external USB drive)

For myself and my own work files, I don't put complete trust in any single backup provider, regardless of what their replication policy is.

You get what you pay for. This product is very squarely aimed at people that want backup cheap (and there's nothing wrong with that). Should you be backing up your enterprise with this (and only this)? Probably not. Is it good enough as a backup of your family photos and videos? Probably, but it depends on your risk aversion level. Additionally, you can always throw another backup service into the mix and get the redundancy you want, at the appropriate cost.

If you are the type of customer for whom this is actually an issue, you are not backing up your data on BackBlaze. You are going (and paying) for someone like http://www.zetta.net/architecture.php.

This type of service starts at around $175 for 500 GB of SSAE16-audited datacenters, but you get all the things that consumer based backup systems don't - such as Plugins for SQL, Exchange, Hyper-V, NetApp, as well as Backup & DR software licenses for an unlimited number of servers - and, most importantly, 24 x 7 US-based engineer-level support.

Brian Wilson from Backblaze here (not the "Brian Beach" who is author of the blog post) - this was not a case of NIH. We use lots and lots and lots of existing software like Debian, Java, Ext4. We use tools like ansible and Zabbix. But this one thing just didn't exist for us in the form we needed it. We looked, we really did.

We did write the Reed-Solomon ourselves in a "clean room" so we did not have to pay any licensing fees and we clearly didn't steal anybody else's source code, but that is a very small amount of code. Like 80 lines of Java. Seriously. We referenced the technical papers we read to implement it in that blog post, but here it is again: http://www.cs.cmu.edu/~guyb/realworld/reedsolomon/reed_solom... And we unit tested the living heck out of that code, plus we mathematically verified various parts.

But I'm open to an alternative solution if you can suggest one? Remember our three highest priorities are: 1) reliable, 2) low cost, 3) simple. The "low cost" includes things like we do not want to pay ongoing licensing fees to other companies.

I doubt that Backblaze decided to go this route on a whim, particularly because it's really their only option for cost efficient large scale storage. MD is terrible for this use case. There's no mature, open source object store with R-S coding.

So everyone uses MD, but why does it suck for Backblaze-like storage? It only provides RAID levels, so you have partition-level redundancy but not necessarily object-level redundancy. You still need another layer to put objects in the right places to achieve the level of redundancy you want, and to re-balance as necessary. And a RAID array is local to one system, so achieving multi-host redundancy means duplicating the data as many times as necessary.

MD can also be finicky. Since it's part of the kernel, kernel upgrades can (rarely) produce weirdness. You can get stuck between a rock and a hard place, where you need to upgrade to fix a vulnerability but upgrading too quickly puts your data at risk.

According to Backblaze, they used to do 3 RAID6 arrays of 15 drives each per pod. This gives an overhead of 1.15:1, and two devices can fail before you lose data. Write performance is not going to be great, nor is rebuild performance. That is probably part of the reason why they could only push 950MB/s per host.

This only provides disk-level redundancy, and not host-level. So now you have to at minimum duplicate your array onto a second host, and your overhead is 2.30:1. A third host brings it to 3.46:1. I'm surprised that they were even using RAID for non-boot devices at all, given the overhead.

Erasure coding allows you to safely store data with far less than a 2:1 overhead. Their current design claims that they can lose 3 storage devices before they risk losing data permanently, with an overhead of 1.17:1. That is pretty compelling from a cost perspective.

Agreed. R-S encoding is hard. You should use something like zfec (extensively battle-tested by Tahoe-LAFS) or JErasure rather than rolling your own.

(I say this as someone who worked at a company where we rolled our own, but that was ten years ago and we didn't have those options.)

They don't say they did it because it was fun, only that it was fun. (To be fair, they also don't seem to say why they did do it.)

Brian from Backblaze here. We do angst over the idea of a "hot spare" where the very second we fail a drive it can begin rebuilding elsewhere. But that takes up redundancy even when it is not used (an extra drive waiting) which raises cost.

At our current scale it is becoming less and less of an open debate, because we now have 7 day a week staffing at our datacenter and the datacenter techs jump right in and replaced failed drives often within an hour or so. A "hot spare" would only save a couple hours of rebuild time. But remember, your mileage will vary - until you reach half our scale you cannot afford even a Monday-Friday datacenter tech, so you might only be able to replace failed drives on Mondays and Wednesday, which widens your exposure.

Yev from Backblaze -> we are SO jazzed to write that one up soon. We're trying to find a good way to present it, otherwise it would have been included here, but that would have made for a very long read. Stay tuned!

Here's a nice explanation of how ZFS does triple parity raid with reed-solomon:

http://people.freebsd.org/~gibbs/zfs_doxygenation/html/da/dc...

http://people.freebsd.org/~gibbs/zfs_doxygenation/html/d1/d7...

Yev from Backblaze -> That meteor question comes up a lot (https://www.backblaze.com/blog/vault-cloud-storage-architect...). We currently do have one data center, but this design allows us to bring others online. If the datacenter was hit by a meteor all our customers would get an email blast urging them to create a local backup immediately. The chances that both the DC and the user would get hit by the same natural disaster are relatively small. Still, it's not a storage service like S3 so geo-redundancy plays a smaller role. We do plan on building out other datacenters in the future, but since we're bootstrapped, we have to do that when the time is right, otherwise it would be very easy to over-extend ourselves and start losing money.

edit -> I ignored your Backblaze dev question, sorry. We have multiple processes running at all times on the pods, and they go shard-by-shard. We're always optimizing, but the short answer is, we're always looking for errors.

Yev from Backblaze here -> We're VERY proud of our datacenter techs, and you'd hear more about them if they weren't so shy. It IS a monumental achievement though, considering that a few years ago we only had two guys running our entire farm.

Did you do that math right? Shouldn't it be about one pod per day?

> It seems like nobody at BB cares about the restore process

Brian from Backblaze here -> I care! It just keeps getting bumped by something higher priority. I have a spec for how to speed up the restore tree browsing, it's just waiting for us to have a spare moment. For a while the Vaults took precedence.

Part of running Backblaze without VC funding is we can only hire programmers when we can afford it out of profits, and we're up to about 6 programmers (the result of a recent burst of hiring) which handle all of: Windows, Macintosh, iOS, Android, and in the datacenter they built the pods, the Vaults, and the web front end. But we'll get there, I swear.

> or that it doesn't sell new subscriptions.

This is unfortunately the heart of the problem. The most important thing to get smooth as glass is the BACKUP part, that sells new subscriptions. If we have your data safe, we can always hobble through a restore even if it is a little slow and clunky we can get all your files back after your laptop is stolen. If it held up sales, we'd jump over and do the one week of work to speed it up.

Yev from Backblaze here -> we're working on the restore process. It can DEFINITELY be smoother, and we're hoping to address that in the near (hopefully) future! As for the tracking pixels, a lot of those aren't actually in use anymore but are remnants of advertising that we've run in the past, but that could use some cleaning up as well. I'll chat with the team to see if we can expunge the one's we're no longer using!

Yev from Backblaze here -> Yea, we hope they start doing more stuff like this too! More information = everyone wins!

Yev from Backblaze here -> Nice! Glad it's working for you :)

That number is probably the estimated probability that over the course of a year, a single vault loses every file contained on it (which is what would happen if a single vault had 4 drives simultaneously and irrecoverably fail).

As a consumer, the type of failure you would experience, and the probability of experiencing that failure, given that Backblaze has suffered a vault failure, depends on how they distribute your data amongst their vaults. They don't explicitly say how they do this, so it's impossible to know for sure, but we can consider the two extreme scenarios.

Scenario 1: Each customer is assigned a single vault, and all your files are on it. In this case, if Backblaze lost a vault, you would either luck out and have your files on another vault and be completely unaffected, or get really screwed and have all your files on the bad vault, and lose them all. They've got 150 PB of storage, and each vault stores 3.6 PB of data, so we can estimate that currently you may have something like a 1 in 40 chance of having your data on any given vault. So under this scenario, you would have a 1 in 400 million chance of losing all your files.

Scenario 2: Each customer's files are uniformly distributed across all vaults. In this case, if Backblaze lost a vault, all customers would lose a fraction of their files. Again, using our estimate that they might have 40 vaults, you would have a 1 in 10 million chance of losing 2.5% of your files.

So up to now, we're basically just doing the math without questioning the assumptions of the model. In reality, I think your practical risk is mostly concentrated in things outside of the model: ie, an event that affects all of their vaults simultaneously, like a fire, earthquake, meteor strike, etc. If I had to make a bet about what that number is, I'd put it in the 1/10,000 to 1/100,000 range. In other words, orders of magnitude higher than losing data because some hard drives failed, or a backblaze employee spilled his coffee, or something like that.

It's really impossible to use that number in a meaningful way, since you also have to take into account that most instances of file loss on their end will be unnoticed anyway (you'll still have a local copy that will simply be uploaded again).

Stay tuned.

I'm happy their webserver has better cipher suites than it did last week.

The drive operations (including waiting for the head to seek, etc) could take place in parallel. There's also the chance the data needed was in cache. So it might not be horribly bad.