Farming hard drives: how Backblaze weathered the Thailand drive crisis
blog.backblaze.com
blog.backblaze.com
the hard drive shortage hit us all. this one company coordinated a mass buying operation from commercial stores when the rest of use were looking for personal use.
while i don't think this is illegal, it doesn't make me feel warm and fuzzy inside either.
when there's a shortage, the "noise" in traditional distribution means that small buyers (like me, and many others) can still find what we need if we put in some extra effort. it functions like a "soft buffer", making the transition to a complete lack of resources more gradual. action like this removes that buffer.
this feels like one group of people - driven by their desire to do business - beat out a load of other people who didn't have the same logistical resources. there's something uncool about it. i am not sure i have explained very well, but it's to do with crossing the line from business to personal.
[edit: ah, thanks - separate thread: http://news.ycombinator.com/item?id=4630910]
[edit2: wtf? i thought someone replied here? was about to upvote you. thanks anyway.]
Important to remember that really it wasn't this one company, they were just in the rare situation of needing drives enough to try and buy lots, and not needing enough that they could go further up the food chain in the search of them.
Think of companies such as PC manufacturers, you'd be amazed what was going on behind the scenes.
a : to obtain by energetic activity <hustle up new customers>
1. freeze registration of new accounts and purchase only the drives necessary to keep existing contractual obligations
2. charge new accounts at a different rate, promising to drop the rate once drive prices returned to normal
3. change plan storage volume limits
The only reason they didn't pick any of these 3 options, as far as I can tell, is because all of them were bad for their profit margins and it would be nice if they were upfront about that as the real driver for their actions. It's not clear that their business would have failed had they chosen one of these options.
So they chose a path that would screw over the little guy needing a new hard drive in San Francisco. Perhaps I'm wrong, though, and the timing made it very difficult to act on these other options, or their cashflow couldn't handle it. That too would be nice to know, if it's the real driver of their actions.
Entirely rational behaviour (in the same position I'd do the same), but while an entertaining story, it's a hard to story to feel enthusiastic about even though in every other way it shows great leadership.
It can't be cheaper to produce the external drives, so the internal drive prices must have been raised above the actual cost to the manufacturers, presumably because the internal drive demand was guaranteed. Consumers might hold off purchasing an external disk if the prices doubled, but Apple aren't going to stop selling Macs because of it.
Sometimes capitalism isn't perfect.
I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.
Plus 2 people got banned from Costco.
I've only ever bought five external drives in my lifetime. Incidentally, all of them have been "shucked" over the years, and none of them have failed. They vary from 3-5 years old.
In fact, the only two drives I've had fail on me were OEM packaged, so based on my anecdotal, statistically insignificant personal experience you could even say the opposite, that external drives are more reliable than their OEM counterparts.
You never know. Perhaps the additional packaging prevents damage to the drives in transit, and the process of installing software on the drives (as I believe some manufacturers do with external drives) provides another opportunity to quality check the drives.
I hope they're keeping some basic metadata about the shucked drives, so they can compile some statistics for us. It would be fascinating to see what the lifespan of these drives will be over the next few years!
The well known Google study was interesting.
We use consumer-grade drives across our entire 40+ petabyte storage farm because we do not need high-performance drives. Instead, we need low-cost, low-power, high-density drives to put into our custom-designed Storage Pods: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
We tested "enterprise drives" and found that from a pure reliability perspective, they actually performed worse in our testing than the consumer drives. With about 5 years and 40+ petabytes of data, we're fairly confident this holds true.
Finally, we assume that all drives will fail and replace about 10 drives every week. The software is written such that the system takes drive failures into account and data is stored redundantly and continuously cross-checked.
I did the same thing when the "crisis" happened, I bought a load of external drive and removed the hard disk from them.
Economics (not quite capitalism) says that if you attempt to replace a market with a rationing system, you will get a market anyhow, because markets are emergent features of reality and as soon as you have multiple agents with differing values for the same goods trade happens. (Capitalism would add "... and that's a good thing!") Rationing will generally result in there being an artificial consumer surplus created in the system versus a functioning market, and that consumer surplus will generally be captured by a market-maker rather than either any of the intended beneficiaries of the rationing scheme or the entity who has the product requiring rationing in the first place.
In the absence of a rationing scheme, a) Backblaze would have had as many drives as it wished to purchase at the market price (slightly lower than the price they think is unconscionable in their normal channels) and b) most home users of hard drives would have been unable to source new hard drives at rates they were willing to pay, and would have turned to substitutes like "use refurbished hard drives", "decrease consumption of hard drive space", and "defer hard drive purchases."
The entire point of the 2 item limit on deals like those is to prevent companies from doing exactly what Backblaze has done. By snapping up all of these deals, they're reducing the number available for consumer purchase, which impacts the revenues of these big-box retailers. Basically, Backblaze absorbed the (economic) consumer and retailer surpluses for HDDs in the Bay area and passed it on to their consumers.
I am sure there is some mathematical formula that will express the risk based on how infrequently one expects to spin the drive up (and then down again), but I don't know it.
We really never considered "shucking" external drives, since the drives inside these enclosures are the worst spec'd, highest failure rate drives ... never mind the (usually) nonsensical enterprise vs. non-enterprise, they're just crummy drives.
Good for BB for making it work ...
BTW, hard drive price weirdness continues to this day - 3TB Hitachi enterprise drives two months ago were $198 (or so) and last week they were $248 (or so).
I don't know if the 50TB/day figure includes the replacement drives or only expanding the capacity as I guess their failure rate is quite high.
edit: fixed a typo.
In fact, this whole article is making me doubt my faith in Backblaze as an off-site backup solution.
And those pods are cool. But I agree, a somewhat flawed approach.
What are their reliability promises? I think that they're doing some mirroring of the data, but I'm still not sure about the rest of their service details.
I'm considering using them for backups, but have not decided yet.
I came upon this [1] discussion here, from 3 years ago. I'm diving in :)
Yeah, the pods are really sweet and they provide all the design documents [2]!
[1]: http://news.ycombinator.com/item?id=803136
[2]: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
edit: added additional details about the design
I phoned up all the official Microsoft distributors in my country, who phoned up Microsoft and whatnot. Not one of them could offer me a price below about £35. And getting in contact with Microsoft hardware directly? Ha, I should be so lucky.
I'll bite. Why? :)
Anybody can win our business by under bidding the other resellers, and we're ALWAYS open to suggestions. If you know any way to achieve lower drive prices, let us know! PLEASE!!
I can't ask for more.
And realistically, if push comes to shove, I'm sure BackBlaze would send out a heartfelt email to their loyal customers explaining the situation and kindly requesting a price increase. Especially after reading a post like the one they just submitted...I'm sure no one would have a problem ponying up a few more dollars.
I suspect, though, without any actual data to back this up, that the drives that go into (low-cost) external HDD enclosures have lower MTBF ratings than the desktop-oriented internal drives. I think that the usage pattern for an external HDD is very different in terms of longevity in operation than desktop hard drives (and these have lower guarantees than eg. server-grade SATA drives).
If I was a manufacturer of such enclosures I'd be willing to buy cheaper drives with lower promised MTBF for embedding them in enclosures that stay off most of the time.
So, even if they're the same, I guess the manufacturer performs some tests and sells drives that fail some non-critical ones at a lower price (pretty much what Intel and AMD do with their chips - those that can't sustain the highest frequencies are downgraded to clock rates that are stable).
edit: The comment from @dsr_ tends to support my hypothesis, although I haven't looked at the differences in guarantees of external enclosures, it's true that internal desktop HDDs can have up to 5 years of guarantee (i.e. WD's Black series has a 5 years guarantee).
It's why their restore isn't instantaneous. You have to ask for the files and they'll let you know when they're ready, this takes time (up to several hours).
AIUI they essentially fill up each Pod with incoming data and, when it's full it goes into a low power mode where they spin down all of the disks in that Pod.
Restore requests (from many users) will usually span quite a few Pods so they batch up requests for each Pod so that can minimise the number of times that drives in a Pod need to be spun up to copy off files. Rather than spinning up and down once every 10 minutes to service 36 requests over 6 hours they wait 6 hours, spin up once and service all 36 requests for that pod in one go. (Figures are for example only.)
(Small restores may be instantaneous if the Pod is still being filled or just happens to be spun up for restore soon. Large restores will more than likely span multiple pods and so there's more of a delay until all parts of the restore have been obtained from all of the pods involved.)
If anything this maps pretty well to the expected usage pattern of an HDD in an external enclosure as many of them are used for infrequent backups rather than extending constant use storage. It's how I expect Backblaze will be hoping to get much more than the usual MTBF out of their drives they have purchased.
HDD guarantees may be time based, but HDD failure rates are linked much more closely with usage patterns; number of spin-ups/downs, number of seeks, number of sector writes, etc. A drive in constant use on a desktop machine is going to be hammered a lot more than one that's essentially written to just once (as the pod fills up) and then spun up every now and again for partial reads.
(I know nothing about using tapes to seriously store data, so this is conjecture.)
Experience: I helped route and process data for the Large Hadron's CMS detector.
They achieve the best bang for buck by using ordinary consumer hard drives in RAID arrays. During the hard drive crisis, the cheapest way of procuring these was by purchasing external drives from ordinary stores.
There's nothing wrong with that. If anything, it shows agility and ingenuity.
What else can Backblaze do? Stockpile drives? The hard drive industry moves fast, with major product releases every 6-12 months and constant price reductions; this would almost certainly increase the cost to the end user.
No business can adequately prepare for a supply shock like this. Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business.
In this case, would you be berating the ingenious airline that goes out of its way to track down fuel at a cheaper price, finding a low-cost fuel that they can easily refine themselves to aviation fuel, so as to honour their commitments to their customers and avoiding raising their prices?
No. They use hedges to control their cost (this is the original point of hedging...). If a company is reliant on a constant supply of hard-drives and doesn't want to be exposed to the risk of extreme price jumps it should probably do the same.
e.g. http://www.delta.com/business_programs_services/delta_cargo/... the first column is the fuel surcharge that floats with fuel prices.
The Backblaze article clearly said that paying more for drives and passing it on the customers was an option. They just found a better one. They didn't mention it, but just losing money for a while is also a reasonable option if you don't want to antagonize your customers.
Given that hard drive prices almost always drop constantly, is that tactic really a good idea? Is it possible to reasonably hedge against price spikes in a commodity whose price you expect to consistently and rapidly drop? (Serious questions, really.)
Typically if you're hedging a commodity for actual use, what you're trying to do is to lock in your cost and not to make money out of it. Let's look at an airline for example. The price of oil currently is $90/barrel and I think it might go up even more to $120/barrel. I buy a contract for 100 barrels of oil at $100/barrel deliverable in 3 months. Say I use 100 barrels of oil every 3 months. I now know that my fuel cost is now $10,000 for those 3 months. This then allows me to price my product based on that $10,000 assumption.
In the case of hard drives, I have no idea if it would be a good idea to hedge since the prices keep on dropping and the Thailand flood is not an everyday occurance and in general prices of hard drives are pretty stable and don't fluctuate like oil prices. But fundamentally, you're not trying to avoid losing money; you're trying to lock in your cost so you can price your service accordingly and not be in a bind if the price of your commodity fluctuates.
In this case internal drives are ordinary aviation fuel, external drives are the alternative fuel, the price of which has been largely unaffected by the supply shock, and the process of taking the external drives apart to reveal internal drives is the equivalent of a simple refining process that turns the alternative fuel into ordinary aviation fuel.
In the end, they are cheap, if you are worried about their methods I'm sure they have competitors who approach things in a more traditional fashion and charge accordingly.
They have built a backup architecture designed to use low reliability commodity hard drives.
This post shows the lengths they go to just to keep growing. Imagine what they'll do to keep the system running!
The machine plans are on their site and they explain why they chose off the shelf components. My only regret is that no one on our team has welding skills, so we couldn't make awesome face plates. lol
Backblaze is one of the more transparent companies operating in the space, which increased the trust level.
(Gigaom blogspammed it.)
I suspect the biggest issues are that all their suppliers and customers are in that region. Think fabs to make the chips, the various motors and related electronics, coating processes, and then sending them off to the places that make HP/Dell/etc systems. If just the drive factory were moved then you'd still have the drive parts crossing the ocean to get to the US and the completed drives crossing once again to go back into the systems.
Usually the relocations happen when a disruptive solution comes along. You get the advantage of a somewhat clean sheet. For example SSDs don't need all the mechanical components and fine tuning, so they can be made with a completely different supply chain.
It won't happen in hard drives but for other products customization is what can spur relocation. For example Lenovo is opening a manufacturing plant in the US to deal with customization which allows quicker response times. (Customized orders often can charge a little more which offsets increased costs.)
They should have moved quickly before Best Buy/Costco implemented constraints that should have been anticipated. During one weekend, via train or one-way flights, employees could have been mobilized to cities across the US. Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have been shipped back to HQ en masse. Three days, disaster is over.
I give them a C- on execution.