Petabytes on a budget: How to build cheap cloud storage (2010)
backblaze.com
backblaze.com
Its grand that its worked out for them, but there are a few big drawbacks that backblasze have software'd their way around.
Nowadays its cheaper to use an engenio based array like the MD3260 (also sold/made by netapps/LSI)
First you can hot swap the disks. Second you don't need to engineer your own storage manager. Thirdly you get much much better performance(2gigabytes a second sustained, which is enough to saturate to 10 gig nics). Fourthly you can get 4 hour 24/7 response. Finally the air flow is a bit suspect.
we use a 1u two socket server with SAS to server the data.
If you're brave, you can skip the raid controller and the JBOD enclosure instead and ZFS over the top. However ZFS fragments like a bitch, so watch out if you're running at 75% plus
Plus the software has an inherent value, it isn't purely a cost centre like you imply. The value of Backblaze is partly made by that very same software you scoff at.
they are using really wide raid6 stripes, with horrific rebuild times. no hot swap ability.
put in anotherway: with 100 pods, you're going to be loosing 10 drives a week (on average, sometimes more) it takes 48 hours to rebuild an array (although I heard it was upwards of 72 hours, without load.)
so every week, you have to powerdown upto 10% of your storage to make sure you don't loose future data. or you can wait till you've exhausted all your hot spares, but then you're in the poo if you loose a disk on power recycle.(yes raid 6 allows two disk failures, but whats to say you wont loose two disks, as I said they are using large stripes (24?) so there is a high chance of a raid rebuild failing)
hotswap is awesome. running low on hot spares? sure replace online, walk away happy.
to make a backblaze work you need to keep more data in tier 1 storage, which means higher cost. (even if you use a software layer over the top to do FEC you still need at least 1/3rd more disks. )
Then there is the woeful performance.
if you want JBOD, with large scalability, just look at GPFS. The new declustered raid is awesome, plus you get a single namespace over zetabytes of storage, with HSM so you can spin out cold blocks to slower cheaper storage (or tape)
It blows everything else out of the water, ceph, gluster lustre, GFS2 & hadoop. Fast, scalable, cross platform and real POSIX, so a real filesystem which you can backup easily.
> they are using really wide raid6 stripes,... no hot swap ability.
SATA drives are inherently hot-swappable, you can hot swap drives in a Backblaze Storage Pod if you want to. In our datacenter for our particular application we just choose not to. We shut down the storage pod, calmly swap any drives that failed, then boot it back up. The pod is "offline" for about 12 minutes or there about. The RAID sync then happens as the pod is live and the pod can be actively servicing requests (like restoring customer files) while the RAID sync occurs.
How often do you see multi-drive failures(in the same lun)?
We've just past a peak of drive failures (had a power outage) where we've had multi-drive fails in the same lun. The awesome part about that is that each rebuild caused more drive to fail. I think one array swallowed 5 disks in two weeks. It didn't help that the array was being hammered as well as rebuilding (70 hour rebuilds at one point)
> How often do you see multi-drive failures(in the same lun)?
It happens - usually for a reason. Most recently a bunch of drives of one model all seemed to fail very closely together after running for 2 years, it's like some part wore out like clockwork. We are pretty relaxed about 1 drive popping, nobody gets out of bed and runs to the datacenter for that, it can wait until the next morning (the pods instantly and automatically put themselves in a state where customers can prepare restores but we stop writing NEW data to them to lighten the load on the RAID arrays). So if 2 drives fail within 24 hours on a Sunday then Monday morning we go fix it.
> each rebuild caused more drives to fail.
Same exact experience here, we feel your pain. We even have a procedure where we line up all brand new drives, clone each individual original drive to a new drive, then take all the new drives and insert them back into the original pod AND THEN run the RAID rebuild.
But there's a lot of use cases these days where the majority of the data in question is "cool to warm" as opposed to hot/active. While vendors like NetApp and EMC would be happy for you to pay tier-1 vendor prices to store cool data on high-performance storage, if your app can handle it, it's not really necessary.
Think about photo storage, backups, database archive/transaction logs, log aggregation, etc. etc.
The Pods are not general purpose storage devices. They are literally as close to tape as you can get without getting dirty.
the average throughput of our teir1 storage is around 10 gigabytes a second (not bits) with a peak of 25-30. however they are just 60disk with 4 raid6 (with 4 hot spares) and dual 10 gig servers, 4tb disks.
They are just plain NFS servers with XFS. They are fairly good general purpose fileservers, great for high streaming IO. replace those 4tb disks with SSDs or 15k disks, you've got amazing random IOPs
> the average throughput of our tier1 storage is around 10 gigabytes a second (not bits)
Any one Backblaze storage pod is absolutely limited by the 1 Gbit (not bytes) ethernet interface. So if you need your data sequentially from one pod it is not a very fast solution.
But remember we (Backblaze) have 863 pods (with another 35 being installed each month), each pod has 1 Gbit connection directly on the internet. So actually our "fleet" supports peak transfer rates of close to 863 Gbits/sec (107 GBytes/sec).
Backblaze storage pods aren't the right solution to all problems, but if your application happens to be parallelizable don't count them out as "slow".
Do you see the need yet to dump in a 10 gig card? or does the economic seem a way off?
We have a few of the pods in the datacenter running with 10 Gbit copper on the motherboard as an experiment. I'd have to double check, but it raises the price of the motherboard about $300? That's not bad AT ALL (only a 3% cost increase), the real cost hit is if you want to utilize the 10 Gbit you have to buy a 10 Gbit switch, which raises the cost of a 48 port network switch by thousands of dollars.
Netgear ProSAFE XS712T is a (recent) fairly cheap 10 Gbit switch. I haven't tried it yet, but soon.... And hopefully the fact that it exists will drive down the ridiculous price gouging of the more "datacenter" brands. Not to say Backblaze is above putting consumer hardware in our datacenter when the quality and price are right. :-)
> Nowadays its cheaper to use an engenio based array like the MD3260
Cheaper than what? Backblaze still builds it's own storage pods because we can't find anything cheaper. We would LOVE getting out of that part of our business, just show up with something 1 penny cheaper than our current pods and we'll buy from you ENDLESSLY. NOTE: we have evaluated two solutions from 3rd parties in the last 3 months that were promising, but when the FINAL pricing came in (not just some vague promise, wait for the final contract/invoice) it was just too much money, we can build pods cheaper.
If your goal is to keep your job as an IT person - spend 10 or 100 times as much money on super highly reliable NetApp or EMC. You won't regret it. Your company might regret the cost, but YOUR job will be a bit easier. But if you want to save money - build your own pods. NOTE: remember Backblaze makes ZERO dollars when you build your own pods.
> We would LOVE getting out of that part of our business [...]
That's a common theme when I meet with some unique companies that are still building out their own stacks of kit and just enough software to keep their consumption patterns consistent. It is about necessity some years but often comes down to the assembly line methods and sunk investments.
I'm curious how much variance you guys have / tolerate or if you ever see reaching VMI (maybe you are already there?) procurement levels?
Market wise, it isn't too outlandish to expect many of the "super highly reliable" options to have a future that is a spectrum of DIY + software (100% roll your own) all the way up to a full experience (status quo). Today, that roll your own option is a progression of hardware compatibility lists to reference architectures but there are only a few companies that go beyond that.
I'm curious to hear how much you see as "too much money" to keep your team from getting out of the rack/stack/burn/test/QA/QC/deploy. Do you plow the "too much money" less "build pods cheaper" into a value that can be compared an opportunity cost for the labor involved?
Early on the "too much money" would have been maybe a 20 percent premium. But now that we have scaled up and have all the teams in place, "too much money" is probably even a 1 percent premium. In other words, if we moved to a commercial solution now it would be because we actually SAVE money over using our own pods. That shouldn't be impossible - a commercial operation could make their profit margin out of a variety of savings, like maybe their drives fail less often than ours do because they do better vibration dampening. Or since this is "end-to-end cost of ownership" if the commercial system could save half of our electricity bill by being lower power then they can keep 100 percent of that.
We have a simple spreadsheet that calculates the cost of space rental for datacenter cabinets, the electricity pods use, the networking they use, the cost of all the switches, etc. It isn't overwhelmingly complex or anything. It includes the cost of salaries of datacenter techs with a formula that says stuff like "for every 400 pods we hire one more datacenter tech", stuff like that. Hopefully if we went to a commercial built storage system we could increase the ratio like only hire one more datacenter tech every 800 pods. The storage vendor could pocket the salary of that employee we did not hire.
Or heck, maybe a large vendor can get a better price on hard drives than we get. We buy retail.
> reaching VMI (maybe you are already there?) procurement levels?
I'm not completely sure I understand the question, but here is one answer: in a modern web and email and connected world with tons of competition, I'm not sure there are any economies of scale past the first 10 computers you build. Places will laser cut and fold sheet metal in units of "1" for extremely reasonable prices, it's AMAZING what the small manufacturers can do for you nowadays.
At 4U, you guys can fit 45 drives in your case. The equivalent OpenCompute Knox would be 6U, at 45 drives. They have a different backplane arrangement, and more redundancy than y'all, so I expect it'll be a little bit more expensive, but at the benefit of not having to manage your own supply chain.
I think you missed the "on a budget" part.....
Netapp and dirt cheap do not belong in the same thread
1) The object replication between the pods will be hard on the array, as the object replication is simply an rsync wrapped inside nested for loops. If you have a bunch of small files with a lot of tenants, it'll hurt.
2) While swift allows you to simply unmount a bad disk, change around the ring files and let replication do its thing, there are actual issues. First of all, out of band smart monitoring of bad sectors actually cases the disk to pre-empt some sata commands and do the smart checks first. On a heavily loaded cluster, under a smart check for a bad block count could kill the drive, and take out the sata controller along with it. We've downed storage pods before by that way. The only way we got around it was to take a pod offline once a week, run all our smart checks, then put it back.
3) To replace a drive, you have to open the machine up and power it off. As any old operator will tell you, drives that have been running for awhile do not like to be turned off. If you are to power off a machine to replace a bad drive, do realize that you might actually break more drives just from the power cycle.
4) Once you change a drive out and use the ring replication of files to rebuild your storage pods, your entire storage cluster will take a non-trivial hit.
5) Last but not least, it is almost of tantamount importance to move the proxy, account and container services away from the hardware that also host the object servers. It's probably good to note that the account and container meta data is stored in sqlite files, with fsync writes. If you add and remove a bunch of files to multiple accounts, the container service is going to get hit first, then the object service. Further more, every single transaction to every meta data/data service, including replication, is federated through the proxy servers. If you look at a swift cluster, the proxy services take up a large chunk of the processing space.
Source: Was Openstack admin for a research group in Middle Tennessee. Ran an openstack cluster with 5 gen1.5 pods for the entire swift service, then moved account/container/proxy to three 12 core 2630s with 64 gigs of ram. Cluster was for a DARPA vehicular design project, first part fielded 3000 clients, second part fielded about 1/10 of that, but with more files and bigger files (these were cad and test results respectively).
If I were to do it again, I would stay away from using enclosures that were inspired by the BackBlaze models. Supermicro enclosures are fine.
Curiously enough, I've also ran smartctl against Samsung 840Pros using Highpoint RocketRaid controller cards during the same project. Sometimes this crashes the controller card.
(This was for a gluster cluster, when gluster had broken quorum support. A copy of a piece of data, when out of sync with the other copies in a replica set, would be left unchecked. This was produced after one smartctl command took out the controller card, and then the machine along with it.)
> open they are about something that is definitely a competitive advantage.
From our perspective, our biggest competitor is apathy. Something like 90 percent of people simply don't backup their computers. So pretty much anything we can do to have people hear about Backblaze and possibly overcome their apathy and start backing up their laptops is good for us. These blog posts have worked very well for us - people enjoy the information, and by disclosing the info we have potential customers hear our name and we get a bump in subscriptions. It's a win-win, the BEST kind of win. :-)
The NAS problem isn't a technical one, it is a pricing problem. We charge a fixed amount of money "per computer" ($5) and we don't throttle or limit customers in any way. We survive on "averages" - most laptop users only have less than 1 TByte and we make money, some have 50 TBytes and we lose money, but on average we are profitable.
If we allowed a single user for $5 to mount network drives with Petabytes of data on them and back those up, this would tip the averages and put us out of business. So we need to solve the pricing problem.
With all that said, stay tuned, we are actively trying to figure out a pricing system that will make everybody happy (NAS owners with Petabytes of data included).
I'd be happy to pay a premium (way over the $5 per computer) for something like that -- I'm sure I'm not alone. Extra if the software was released under the AGPL or something -- so that I could migrate to self-host if I wanted/needed to. I don't think too many would be able to start up and compete directly with you wrt your experience with pods etc... so I think it should be viable. Less sure if anyone would "want" to move off Amazon.
I may not work in IT but I crave well-written accounts of technical problem solving. Yours are among the most interesting as you're so open about sharing details that no one else would consider releasing.
Awww, you are so kind you make us blush. :-)
I do wonder why big companies like Google and Yahoo and Apple don't release more interesting statistics on drive failure. I've heard many people assume they are getting some sweet pricing on hard drives with an agreement never to release those numbers. But if that were true, why won't Seagate or Western Digital or HGST ever give Backblaze a deal to shut us up? Those three companies have refuse to sell to Backblaze "directly" (their business plan is to sell to distributors like Ingram who mark up the product and sell it to NewEgg who marks up the product and sells to Backblaze). I assure you, there are no backroom undisclosed pricing deals for Backblaze.
So I suspect Google and Apple pay retail drive prices also, and is not disclosing drive failure rates because they don't see the profit in it?
This is EXACTLY one of the reasons we released the information. People would write about Backblaze and say we were losing money and it was an unsustainable business funded by deep pockets. That was frustrating to hear because we self-funded the company for years (no VCs) and I assure you there are no deep pockets here.
So we released the info on the pods partly so we could point at it and say "see, the math really DOES work!"
> But the intelligence of where to store data and how to encrypt it, deduplicate it, and index it is all at a higher level (outside the scope of this blog post).
I'm curious about their software that works outside the nodes too. I've been working on storage clusters over this past 9 months using the Ceph ( http://ceph.com/ ) open source storage software. Its pretty amazing -- and I suspect it could be deployed to a set of backblaze pods too.
It seems to be that for production environment where you wanted to maintain availability you would to build at least 3 of those pods for any deployment - enabling replication across pods/storage nodes.
So far seems fantastic from a VM perspective -- I can take down storage nodes, add new nodes etc, all while the VMs running up top stay alive and well. I once lost 1/3 of my cluster (on a 3 node cluster with 30 disks) ..and everything stayed running until I fixed the broken node.
Ceph also supports Object storage and an S3 interface -- I haven't experimented much with that yet, but I hear its good.
What's performance like so far?
finally, when the vm is booted you can use that block device like any other machine. for first install it will be empty ( need to attach a cdrom image to get going ) then youncan start making snapshots for "thin provisioning" using copy on write clones ...very sweet stuff :)
re performance ...i think its good, but i have heard its not as good as expensive proprietary systems. the product we're building is around storage analytics on open source. ..so eventually well know more :)
Say you store 3-4 copies of the data on the BackBlaze pods in order to attempt comparable durability, and then the cost comparison comes down to power, space, network, and manpower costs. These costs are extremely variable, so depending on access to these, I could see someone still going with S3. S3 also scales larger and smaller quite quickly, unlike investing in your own equipment.
S3 = ~USD $1,085,552.64
Glacier = ~USD $377,487.36Playing with Swift now - but it has really opened my eyes to how much more difficult 2-3 petabytes of storage is (disk failures, number of disks in your infrastructure, the time to redeploy a datacenter on a 1gpbs connection). All the little problems become much bigger!
Faced with that sort of thinking, the lower price likely helped them convince consumers. At $5, it's a low enough fee for most people to kind of just forget about it rather than consciously weigh the opportunity cost involved. The last think Backblaze wants is for its customers to think that they'd be better off spending the money on a really good hot dog or two rather than a backup service.
> their market research indicated consumers are highly responsive to changes in price
That makes it sound like we carefully thought it through. :-) Our "market research" was the 5 Backblaze founders asking their friends and family questions like "Mom, how do you feel about paying $5/month for a backup?" After asking that question to 15 different people, it seemed like there were reasonable tipping points around $5/month and then another one at $1/month. The math didn't work at $1/month - so $5/month it was! And we've never changed prices, because we got lucky and it worked out for us. So we've never run the experiment of whether we would lose half our customer base if we raised the price to $6/month. And I hope we never have to raise prices, because subscription customers seem to HATE that - like when Netflix raised prices it was a hate fest towards them.
> I'm quite curious about the economics of storage here.
When we started almost 8 years ago, we were CLUELESS about the economics of it all, and scared witless over it. :-) But I'm happy to report the grand experiment has been a modest success, we are profitable (by a small amount). We even have a little money left over every month to spend on marketing efforts now, and we're growing by every metric - more customers each year, more revenue each year, more employees to swap failed drives (38,790 spinning drives as of yesterday - some fail like clockwork each and every single day).
Are they checksumming on a higher level and is that cheaper than using ZFS with necessarily more expensive hardware?
https://help.backblaze.com/entries/20926247-Security-Questio...
Yep, we checksum the file on the client before sending it. In practice for any one user the number of "silent bit flips" on disk is very low and you probably wouldn't notice it (maybe 1 note in your mp3 file is slightly off). But when we check the checksums every few weeks we absolutely see it occur in our datacenter at our scale probably every day and take steps to heal the files (either from other copies we have or ask the client running on the customer's laptop to retransmit the file).
Seems to me like the design is backwards and doesn't make any sense.
The focus of our backup system is being "easy to use". We get a lot of advanced users who are computer savvy, but we also get some mom and pop novice who aren't very good with computers.
Ok, so we don't allow you to unselect your C:\ (main) drive. Here is the reason: when given that choice an alarmingly high number of novices accidentally unselected the drive. (sigh)
We don't charge customers any more or less on what they backup, so there are almost no legitimate reasons to UNSELECT a drive. What data do you have that you hate so much you MUST lose it when your computer is stolen? So we error on the side of helping these novice users and don't allow them to unselect their main laptop drive. The down side is it sometimes irritates the advanced customers who know what they are doing. :-)
Correct, we are entirely on ext4 now (there might be one or two old pods running JFS left, I would need to double check).
We figured out a way to work around the 16 TByte volume limit originally, but now ext4 even supports larger volumes so that isn't a problem at all anymore.
More random history: At the time we chose JFS it looked like XFS might lose support (XFS was created by a now defunct company I used to work for called SGI). Well, the world is a strange place and XFS now has more support than JFS, despite SGI going out of business. But in the end, ext4 is the "most commonly used" linux file system and therefore is arguably the most supported with extra tools and such, and the performance of ext4 in many areas seems higher than JFS (at least for our use case) so the decision becomes easier and easier to go with ext4.
In the meantime, does anyone have a link?
To expand on this, we require our client because of our pricing model, which is $5 per computer for unlimited storage FOR THAT ONE laptop or desktop. An API would mess with that pricing model, so we haven't opened it up yet.
edit: just spotted it, their boot drive is PATA?! Why is this, given that PATA drives are slower and more expensive than SATA ones?
A 4U beer cooler, not sure if awesome or stupid... but still, a nice idea.
My guess is that they wanted to keep the SATA ports for their storage, and the motherboard had a PATA port available.
It may also have been non-trivial to boot from JFS, which is probably why they wanted a separate disk.
Love to see more write up on software selection process, tradeoff, failure recovery process/methods and benchmark data.
https://www.backblaze.com/blog/backblaze-storage-pod-4/
for more details on the new one.
As long as they have stored my data across multiple replicas, then there is no caring if one of the drives goes bad...
"Funny thing about providing an online backup service that is currently storing over 100 Petabytes of customer data, sometimes people need their data back. In fact, nearly half (46%) of our customers restored one or more of their files in 2013 and that adds up."
https://www.backblaze.com/blog/six-billion-files-restored-an...
So as was said earlier: As long as there are replicas (and presumably a repair process to maintain a reasonable replica count), a dead disk can sit in the enclosure until a high enough percentage is dead that it's cost effective to pull a a couple of them and consolidate into a new one.
Not just because it's awkward to pull them, but data centre visits gets expensive when you factor in the cost of ops people so there are definite tradeoffs in terms of how often it is cost effective to deal with.
> data centre visits gets expensive
Early on going to the datacenter was expensive (and time consuming, the datacenter was a 1 hour drive away from the office). We had some success with training "remote hands" (the datacenter provides some basic services like rebooting servers for hourly fees).
Now we employ several full time people who go to work in the datacenter (they don't even have desks at the corporate office). It makes our world SO much better and smoother now that there is enough work that they can be there full time.
The worst case of course is that actually lose your data, probably as the result of a data center fire/explosion, though people should, in most cases, still have the primary copy of the data on their machines. However re-backing this up would take a long time.
Data centres do fail, have fires etc.
This outage post makes no mention of a backup datacenter and they say backups were halted as a result of this outage: https://www.backblaze.com/blog/data-center-outage/
> how do you determine which drive to swap
Every two minutes we query every drive (all 45 drives inside the pod) with the built in linux smartctl command. We send this information to a kind of monitoring server, so even if the pod entirely dies we know everything about the health of the disks inside it up until 2 minutes earlier. We remember the history for a few weeks (the amount of data is tiny by our standards).
Then, when one of the drives stops responding, several things occur: 1) we put the entire pod into a "read only" mode where no more customer data is written (this lowers the risk of more failures), and 2) we have a friendly web interface that informs the datacenter techs which drive to replace, and 3) an email alert is sent out to whomever is oncall.
Each drive maps has a name like /dev/sda1 and these drive names are reproducibly in the same location in the pod every time. In addition (before it disappeared) the drive also reported a serial number like "WD‑WCAU45178029" as part of the smartctl commands which is ALSO PRINTED ON THE OUTSIDE OF THE DRIVE.
TL;DR - it's easy to swap the correct drive based on external serial numbers. :-)
They only had 9,000 drives back then.
I'm a long time crashplan user and the reason why is because they have a family plan ($15 for 10machines). I have multiple machines that I need backed up, so that's why I go with this. I wished i can use backblaze because it looks very good and have native apps. If you have a family plan, I'll switch in a heartbeat. :)
CrashPlan is awesome, I have ALWAYS liked those guys.
Backblaze focuses on being really simple, we already have 3 different pricing choices which is probably two more than we should have: 1) $5/month, 2) $50/year, or 3) $95/two years. All are unlimited, none have throttles or caps, we don't even price businesses differently. Simple.
Our theory is (may not be correct) that if you offer customers too many complicated choices, they get stuck in the decision making process and don't buy any of the choices. If we offer a family plan that is MORE expensive for two computers but LESS expensive for three computers, blah blah, too much thought. It's $5/computer/month - done. You can get started IMMEDIATELY with one computer, and add them as you go, you don't have to decide to commit to 10 licenses to get the best price, etc, etc.
We don't have any magic knowledge more than anybody, so internally we sometimes angst over stuff like a family plan, but for now we just want it to be simple and straight-forward.
However, I am seriously considering Crashplan myself when my subscription expires in February because I do not like the 30 day expiration policy. It is definitely a pain with large external drives. But I suppose I might not be profitable so maybe that's not worth fixing :P
> I suppose I might not be profitable so maybe not worth fixing
No no, you are (one of) our target customers! Here is why: you post to ycombinator and you are technical enough to understand the implications of the 30 day roll back policy, and you are actually evaluating software on merits - this alone puts you in something like the upper 5 percent of technical computer society. Yeah, maybe we lose a dollar a month on your backup, but if you are using Backblaze you'll probably recommend it to five other people we make money off of. Your recommendation is GOLD to us.