Backblaze Hard Drive Stats Q3 2020 | Hacker News Reader

Backblaze Hard Drive Stats Q3 2020

backblaze.com

238 pointsLaSombra112 comments

I wish that more companies would provide information on their disk fleets like this—especially on SSDs, which would be much more useful information for everyday consumers.

I also wish that Backblaze would focus on their product security[0] as much as they focus on promoting this quarterly hard drive report.

[0] https://twitter.com/zetafleet/status/1304664097989054464

Publishing SSD reliability of datacenter setups would yield no useful information for everyday consumers. See my comment at https://news.ycombinator.com/item?id=24840914. Neither load nor the operating environment of such reports will be anything remotely similar to everyday consumers' use and environments.

In 2015 Facebook published* its study of flash memory failures at scale. By all means it's a terrific paper (of course I'm biased) but I think it was irrelevant not only for consumers but pretty much 99% of the flash users in the industry.

[*] http://users.ece.cmu.edu/~omutlu/pub/flash-memory-failures-i...

Sorry, I was not clear. When I said company disk fleets, I was thinking of places like corporate head offices which have thousands of workstations, not datacentres.

Ah. That'd actually be useful I think.

Lol, that's assuming your workstations are deployed with SSDs. My company thinks they're a waste of money for the plebes.

Well I am under the impression most DC would be operating in ideal condition compare to consumer usage. So it does provide some useful numbers in the grand overview.

That tweet is a big deal if true.

Were there any follow-ups from Backblaze?

> That tweet is a big deal if true.

One thing you can independently verify is that the backdoor string is still in their client. The installer starts transmitting data during the installation process, so I would not recommend installing it outside of a VM—just look at the files directly. The macOS installer has it in `Backblaze Installer.app/Contents/Resources/instfiles.zip/bztransmit`. The Windows installer is a self-extracting ZIP file, so just use unzip and look in `bztransmit.exe` and `bztransmit64.exe`.

  $ strings bztransmit |grep BACKDOOR
  DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file exists: 
  ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file could not be read: 
  ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file existed but less than 10 chars or could not be read: 
  ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file did not contain bz_cvt: 
  ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file contained bz_cvt but wrong num digits: 
  ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file did not contain bz_upload_url: 
  ERROR DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file contained bz_upload_url but did not start with http: 
  DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml file exists and is valid and bz_cvt=
  DoHttpPostSyncHostInfo - BACKDOOR_prefer.xml SUCCESSFULLY_swapped_in new_bz_cvt=

What does the corresponding code do? I genuinely don’t know. My goal was to find backup software, not to do a security analysis. An easy-to-exploit root code execution vulnerability was enough for me to uninstall the software, submit a report as a professional courtesy, and go do something else.

Clearly it’s dumb to put the word BACKDOOR in your code if your goal is to plant a secret backdoor, but it’s also pretty dumb to use world-writable directories, disable host certificate verification, use magic hard-coded strings to “sign” updates, and implement data encryption in a way which requires the ‘private’ password to be sent to the server, so who knows. As I said in the tweet, even if it turns out to be innocuous, the optics are terrible and show a serious lack of good judgement on their part, especially given how much they claim to be security experts who care about their reputation[0].

> Were there any follow-ups from Backblaze?

No. The only “follow-up”, as it were, was to cancel their HackerOne public bug bounty programme. (Though this was a month ago, their web site still tells people to “visit our public bug bounty program managed through Hacker One”[2].) They have not communicated with me at all except for one tweet about that change[1]. I have seen no public statement from them acknowledging that this happened, or that they made mistakes, or that they have steps they plan to take to improve their internal software development practices.

[0] “We stand by our reputation as trustworthy, careful programmers who have worked in the security field for over a decade. […] we have LOTS of interest in keeping our reputations rock solid and utterly clean.” https://help.backblaze.com/hc/en-us/articles/217664798-Secur...

[1] https://twitter.com/backblaze/status/1308157606368882688

[2] https://www.backblaze.com/security.html

This is pretty disappointing to see, I've been looking forward to use b2 vs s3 in prod just from a cost perspective and only using their api, but if they are shipping stuff like this to run on other peoples machines, It makes one wonder about whats running on their servers… though I guess if things are uploaded via the api and are encrypted beforehand or meant to be publicly available are still pragmatic production use cases as along as someone assumes they are compromised when building stuff around it.

> My goal was to find backup software, not to do a security analysis

Did you achieve your goal (i.e., did you find backup software you trust)?

I’ve been happy using Carbon Copy Cloner for local backups for years. It may mostly be a wrapper around rsync, but it’s a very good one. :-)

For cloud backups, I’m still looking. My personal criteria are security (of course), good data integrity management, continuous backup, upload speed, and low resource utilisation. I was hoping to find something which was also simple enough that I could also tell non-technical friends and family to just use <whatever>. What I found was:

Backblaze… well, you know.

Carbonite uses Java and is slow.

SpiderOak uses Python and is slow.

Duplicacy uses Go along with a local web interface for the consumer version. It currently consumes way too much memory during backups[0].

IDrive I never tried installing; I was put off by how it appears to uses a fleet management model where configuration relies on visiting their web site and having stuff get pushed down to the client.

Acronis True Backup somehow manages to be 700 megabytes on disk. I thought they must be using CEF, but no, it seems to just contain a truly colossal amount of code along with a 100 megabyte help file.

At some point I stumbled across Arq, which looked like it could be a winner: they’ve published a couple CVEs on their site like a good company should; they have a monthly sub or you can just buy the software outright and use your own cheap cloud storage like S3 Glacier; they even publish the format of their backup files[1]. But then they went and released Arq 6 using CEF for the UI. It looks like they listened to feedback from users and will be going back to a native UI for Arq 7, and so I plan to take a serious look at this once it is released.

[0] https://forum.duplicacy.com/t/memory-usage/623/

[1] https://www.arqbackup.com/docs/arqcloudbackup/English.lproj/...

Tarsnap?

Am I misreading this pricing information? They charge $250 per terabyte per month? I know I didn’t specify cost effectiveness in my list but I did at least expect everything to be within the same order of magnitude in cost.

Thank you very much for your analysis and explanation. I was a happy customer and would recommend them to almost anyone, up until a few minutes ago.

There was always a nagging voice in the back of my head - "their backup encryption can't possibly be good" - but I wanted to believe it since their deal is so good, in terms of storage per dollar, and it's hard for me to afford otherwise right now. I did a bit of digging not too far back and their help pages go into a small amount of detail but ultimately just say "trust us". I trusted them to throw away my encryption password after a session, which is terrible, of course. After reading all of your work, I cannot even countenance trusting them anymore.

Ironically the first thing that comes to my mind for offsite backup is Backblaze B2... if you do all the encryption before it leaves your machine, then it doesn't matter what they do, and their storage is cheap. Rclone or similar means that their client software need not be trusted. It's up to whether you tolerate their business practices.

You mentioned in another comment that you are still looking for cloud backup software which ticks all the boxes - why did you not go with Rclone? On the surface it looks great but I haven't tried it yet.

> Ironically the first thing that comes to my mind for offsite backup is Backblaze B2... if you do all the encryption before it leaves your machine, then it doesn't matter what they do, and their storage is cheap. Rclone or similar means that their client software need not be trusted. It's up to whether you tolerate their business practices.

Business practices aside, the problem I personally have is: seeing how they are dishonest about their client security, how much can I trust that they’re honest about their durability or server-side security? Encrypting blocks myself ensures confidentiality, but how do I trust that data aren’t being silently lost or corrupted at rest? They hand-wave about how 11-9s durability doesn’t matter[0] in the same way they hand-wave about how zero-knowledge data privacy doesn’t matter, while simultaneously mentioning that they have, in fact, experienced unrecoverable data corruption in the past[1].

> why did you not go with Rclone?

There’s a long tail of roll-your-own cloud backup tools like this that I didn’t really evaluate: Rclone, BorgBackup, restic, Duplicity, Tarsnap, etc. I just ran out of energy and really wanted to avoid something that required a bespoke setup. These tools might work for me, but they wouldn’t be something I could recommend to a non-technical user, and I’m not sure if any of them have daemons for performing continuous backup, so I’d have to figure out a separate solution for that which might be buggy.

In any case, it looks like rclone may have some of the same problems as Duplicacy and restic with using too much memory[2][3].

[0] https://www.backblaze.com/blog/cloud-storage-durability/

[1] “If a cosmic ray has thrown a bit, we ask your computer to retransmit that particular file. This rarely, rarely happens, but in our datacenter of quadrillions of trillions of files in 350 petabytes of customer data. It DOES occur once in a while.” https://help.backblaze.com/hc/en-us/articles/217664798-Secur...

[2] https://rclone.org/faq/#rclone-is-using-too-much-memory-or-a...

[3] https://github.com/restic/restic/issues/2446

Yev here -> Hey there! We've just signed with a new program provider and hope to be rolling out the private program in early November with a public rollout to follow. For now though we ask folks to reach out to bounty@backblaze.com directly with any bugs they find so we can track them internally!

Backblaze doesn't provide data on DOA drives that need to be sent back, do they?

I went through three rounds of 4x 12TB Seagate Exos X16 drives last month through different safe vendors. 100% of the drives arrived DOA (tested in a bunch of different machines). Switched to 16TB Toshiba MG08s and smooth sailing since.

I know the dead ones tend to cluster because of their manufacturing, but I ended up having serious questions about their QC process. Talking to some of my peers, Seagate seems to be a bit of an embarrassment for a long time.

Yev from Backblaze here -> no, any drives that come DOA wouldn't have made it past burn-in and into our fleet, so they aren't represented.

I really wish you tracked this data, because my colleagues and I in various parts of this industry have been noticing some trends...

With failure rates of in-service drives mostly being similar, I start to become much more interested in data about a brand/product's quality control process.

Yev -> that's really interesting. Do you want to publish your data and maybe we can compare notes ;-)

I just have anecdotes from about 4-5 people buying maybe in the 150TB range of disks per year. Mostly only Seagate or HGST.

Not nearly as clean (and representative) a datsset as y'all would have.

Is the takeaway there that you need to be doing burn-in runs too?

You can't burn in drives that can't even be used in the first place. They literally did not function straight out of the packaging.

Sounds like it could have been some kind of incompatibility or firmware/software bug, either on the machines you were using or the drive itself....

Possibly 3.3V issue? (I don't know Exos affects this)

https://www.instructables.com/How-to-Fix-the-33V-Pin-Issue-i...

That only affects WD drives, these are Seagates.

Negative.

I treasure these reports, and yet every time I go to buy more hard disks, it is very difficult resist the low, low prices of those Seagates. I know they'll fail faster than the WD/HGSTs, but they're so much cheaper!

Clearly, given these numbers, Backblaze thinks the same way.

After getting burned on a consecutive $3800 worth of dead-on-arrival Seagate drives, well, never again.

Yev from Backblaze here -> price plays a large factor in our purchasing decisions, but you'll notice if you look at the failure rates that they're not THAT much different across the manufacturers. For home usage, you're likely going to be fine regardless of what drives you buy. Though those sweet sweet 4TB HGST drives are getting harder and harder to find!

To me, cheaper disks will always win because I expect hard disks to fail and take precautions against data loss anyway. If they fail within the warranty period it doesn't really matter where they came from. If they fail outside of it I probably still got my money's worth.

The net effect, paradoxically, is that this encourages and supports low quality. Or perhaps it redirects the resources towards those who need it most, and the average quality improves?

A disk failure is still a cost though in terms of restoring a backup, replacing the failed drive, waiting for an RMA/new drive to come, etc.. If the price delta isn't anything crazy I'd prefer to get something that I know is more reliable.

Agreed. Within $10/20, of course I'll go for the better drive. When it's $50/60 or more, I get the Seagate and hope I'm lucky.

Earlier this year I had a Seagate fail within months of purchase, putting my NAS into degraded performance mode. So I quickly ordered a replacement drive (a WD Red) and started the RMA process for the Seagate. At some point during this process I removed and reinserted the Seagate, and... its been fine since. So now I have a WD cold spare in my closet for whichever drive fails next.

I have a few RAIDZ/RAIDZ2 arrays and I've been wondering ... are let's say 8TB from WD/Seagate all the same size making them interchangeable? What if you have an array with WD drives and the Seagate drives turn out to be a few hundred/thousand bytes smaller?

The useable vdev space depends on the smallest drive in the vdev. ie, let's say you have a mirror with 1x8tb and 1x7.9tb, the vdev will be 7.9tb large. In the case of it being just a few bytes smaller, I would assume you just loose a block.

Only if all the disks when the vdev were created were the 7.9tb drives.

If you have a pool of 8tb drives and want to replace with a 7.9tb drive, you're out of luck.

I've intentionally under-allocated with array space on my drives and use the rest mostly for swap (which I don't want on ZFS anyway, mdadm raid10 is fine) and backups of /boot; all of which I can sacrifice without worry to make the data fit inside of smaller drives.

> I've intentionally under-allocated with array space on my drives

Same here - created on each drive a partition slightly smaller than the HDD's capacity (using ZFS raidz1).

If reliability is a concern, you could still put them in a RAID array.

(but of course this may not be economical in every situation)

Indeed, all of my disk purchases go into either my WD PR4100 NAS or the USB-connected Drobo next to it.

These types of things aren't free, but they're inexpensive enough to be worth it, IMO.

Just had a third well reviewed Seagate Nytro Enterprise SSD fail after a very short period of use, very low write volume. Starting to wonder whats up with Seagate.

Wish Backblaze had SSD stats.

Those HGSTs are still the best in terms of reliability.

My question is, what is happening to HGST the brand?

Western Digital were supposed to phase out the brand, and only keeping the Gold or UltraStar branding. That was may be 2 years ago.

Except HSGT is still everywhere. And I know people demand HGST. I think partly due to Backblaze's numbers. Even the Western Digital UltraStar still has HGST in labels, stickers or description somewhere. Retailers simply call them HGST WD most likely because that is what customers want or are looking for.

It seems Western Digital did at one point try to move their marketing material to focus on UltraStar, but my suspicious is that people still want HGST. Again, may be due to Backblaze quarterly report cause they dont show any UltraStar branding, they use HGST.

The market responded to the Ultrastar rebrand poorly and they brought back HGST officially.

Oh really? Thank You! That is great news to hear.

( Stupid Westen Digital )

Who cares about the label? The internals of the drive makes the quality or lack thereof.

"UltraStar" wasn't a renaming of "HGST"; it was an existing WD brand with many other drives in it, that the HGST drives were going to become aggregated under. This would have erased the distinction between the drives made at the HGST factories, vs. the drives made at other factories, since both would have just been "UltraStar." It would have been a big detective job with every new WD UltraStar model to figure out whether it was "one of the good ones."

Whereas, if some WD drives explicitly say HGST and some don't, you can just buy the HGST ones.

(Very closely analogous to smartphones that have the same exact model name, but which vary between Snapdragon and Exynos SoCs depending on the precise model number that only appears in fine print in a few places. When buying one of these phones used, you often have to google the precise model number to figure out whether you're getting a fast phone or a slow one!)

I'd much prefer an unlockable Exynos to a bootlocked Snapdragon (thank heavens for OnePlus).

I started with MicroG 3 years ago, and there is no way that I'm going back.

Correlating my memory to actual events, anyone else remember the "Deathstar" issue back when the IBM Deskstar drives had that head-drop failure? I'm fairly sure that the Ultrastar line is a successor of that lineage (in marketing terms at least).

https://en.wikipedia.org/wiki/Deskstar

https://en.wikipedia.org/wiki/History_of_hard_disk_drives#Ma...

I do. I remember it fairly well. And due to the whole Death Star fiasco, IBM absolutely over engineer it for their successive Hard Drives. Which is what ultimately lead to today's UltraStar.

I really appreciate how Backblaze is always so consistent in these reports. Quality and disciplined reporting like this is appreciated, and I always find them very interesting.

Side note, does Backblaze not run any enterprise or datacenter SSDs?

Yev from Backblaze here -> we do have SSDs in the fleet but they are primarily boot and data drives, so they're not really an apples:apples use-case. We're constantly looking for interesting news from our fleet though, so if there's something that pops out, our default approach would be to talk about it :D

Oh cool, thanks for commenting here. This is off-topic but I had a question from a post a few days back about google drive dropping unlimited storage. Someone mentioned that the economics just don't work long-term due to people who abuse it. How does this apply to Backblaze's unlimited backup service?

Yev -> Great question. The problem does lay with the abuse of the unlimited services. We've been offering our unlimited backup for 10+ years now and our only price increase was from $5 -> $6/month after 10 years of keeping them steady. It's possible to provide the service, but we have some natural "gates" that were built into the product to help mitigate abuse. Primarily our unlimited backup service is for Mac and PC computers – we don't back up NAS/Servers/Linus machines.

That helps quite a bit because it reduces the outer-bound of how much data can store with us from typical Mac/PC devices. Granted we do have people with 80+TB backed up with that service, but that VAST majority store A LOT less. The average is low enough that we can continue to provide the unlimited service - kind of like a buffet model. Some will eat more, most will eat less and buy a drink (Extended Version History).

Keeping it to Mac/PC only does make Linux, NAS, Server folks unhappy, but thus far no one has been able to crack the code on an unlimited, fixed-rate service that can back up those devices economically, but if we figure it out we'll let folks know :D

> Keeping it to Mac/PC only does make Linux, NAS, Server folks unhappy, but thus far no one has been able to crack the code on an unlimited, fixed-rate service that can back up those devices economically, but if we figure it out we'll let folks know

Please do! I really would prefer to use Backblaze.

In the meantime, CrashPlan for Small Business is an unlimited, fixed-rate service that supports NAS backup on macOS and Linux. I depend on it to backup a lifetime of photos (100K+), for example.

https://support.code42.com/CrashPlan/6/Backup/Back_up_networ...

I have a great relationship with the Code42 folk - they're good people and do good work! Their Small Business service is a bit more expensive, but yes at that increased rate they've been able to provide service to a few more platforms. I have no idea, but I do hope it works for them! I send people there all the time as an alternative if they don't want to use B2 Cloud Storage w/ an integrator (which is what I normally tell folks to use if they have Linux/Server/NAS but not a ton of data).

Thanks for the reply, Yev. I've been experimenting with B2 (via QNAP's HBS 3) as well — works great, but $20/month vs. CrashPlan's $10 — and I'd just ask that Backblaze consider whether there may be room for a "Personal+" or "Family" fixed-rate plan that supports home NAS devices. Thanks!

It's Orwellian to advertise a service as 'unlimited' and then characterize people who expect you to live up to your advertising by using the service as unlimited as abusers.

Expecting vendors to live up to their marketing claims is not abusive behavior; it's the cornerstone of markets based on informed decision-making by both buyers and sellers.

I'm somewhat inclined to agree at first glance. If I see unlimited, I expect to get the value. I only order the "unlimited free refills" coke if I'm going to sit and work somewhere for four or five hours, since it's otherwise pretty expensive. On the other hand, Backblaze doesn't price it's personal backup service like restaurants price "unlimited" coke. I've helped family and friends switch over from other services; some were so scammy they were charging thousands of dollars per year. As far as the average consumer is concerned, it's functionally unlimited in that the marginal gigabyte of data he backs up has no additional cost associated with it. I have a hard time calling that "Orwellian".

Edit: I just saw that another employee mentioned the largest customer is storing 430Tb. That's pretty darn unlimited for the price point.

It's not really "Orwellian" to do that, it's just standard, crappy, marketing doublespeak.

You know the term doublespeak comes from Orwell, right?

Yes, and you ALL missed the joke.

Typically the term "abuse" is reserved for cases where you offer free backups of a phone and someone figures out how to mount their 16TB media collection into that, or unlimited email and someone writes emailfs. You can make a profit on the proposition as you intended it, but cannot sustain the retrofitted use cases.

Disclaimer: I work at Backblaze.

> the economics just don't work [for Unlimited] long-term due to people who abuse it. How does this apply to Backblaze's unlimited backup service?

So far, it has worked out for us (going on 14 years now). If you are curious about the size distribution of our customers, here is a histogram of our Personal Backup customer data sizes from the end of 2018: https://i.imgur.com/iVEuwUT.jpg (You will need to zoom in to see all the data points - the largest customer stores 430 TBytes for $6/month.) Backblaze never raised any funding, we're self sufficient with no deep pockets, so it works for us financially.

Just to be clear, Backblaze doesn't price the Personal Backup product as $6/month for an unlimited amount of storage to attract gigantic customers. Our driving force for almost every decision on the Personal Backup product was to make it "easy" for people who either are not experts at using computers, or just don't want to spend a lot of their time on it. My 88 year old father doesn't know the difference between a MByte or a GByte or a TByte, and he would be stressed out worrying about "paying per GByte". The fixed price of $6/month is intended to reduce "sales friction", not attract massive storage customers. Another issue for non-computer-expert customers is they don't know where all their files are. So to make it "easy", Backblaze simply backs up EVERYTHING by default, and only excludes things like the Operating System if we absolutely know what it is and know it can be recovered in other ways. So the Backblaze Personal Backup product can be installed with a username and password, ZERO CONFIGURATION, and the customer's data is backed up. But that wouldn't work very well if we charged per GByte, customers would think we were doing it just to increase how much they are charged. :-) So the "fixed price for any amount of storage" is part of the overall "easy to use" strategy.

What we ask of our above average storage customers is that they recommend the product to their less technical friends with less data. We get a good reputation by supporting the large customers, and those customers are often the "technical mavens" that recommend products to their friends and family.

I work on the Personal Backup client that runs on your laptop. Personally I like the super ultra big customers simply as quality assurance. If I can keep the client running for a customer with 400 TBytes, it will absolutely purr like a kitten for an average sized customer with 1 TByte of data.

Backblaze mostly just stores data. The major advantage of SSDs is in their radically higher IOPS, but that doesn't matter for data at rest. SSDs also have higher maximum read/write speeds, but I wonder if actually the network connection of the storage pods is the bottleneck here.

https://www.backblaze.com/blog/hdd-vs-ssd-in-data-centers/

This article says Backblaze uses SSD a little bit. Somehow, I remember they used Samsung branded SSDs and they had very few failures that weren't DOA, but I can't find the relevant article, so that memory might be entirely fabricated.

SSDs have such an orderly and predictable failure regime that I doubt a quarterly reliability report would be of interest to anyone.

Not my experience lately with Seagate Nytro Enterprise SSD's, had a few of them fail completely within the first few months of use. No warning just disappeared and no longer recognized on the bus.

Would love to see failure rates on SSD brands.

Firmware bugs (an overflowing counter, etc) are not predictable and this is where long-term enterprise usage data could be helpful as drives that are always-on for years are more likely to trigger those bugs than casual desktop usage.

BackBlaze has been doing this for years, and they're well received in many online communities. They don't need to do this, and the data is arguably valuable to keep proprietary.

Are there any other examples of companies doing similar releases of valuable internal data?

The only reason I know about BackBlaze is because of their reports and the storage pod build outs they do.

If they never did these I wouldn't even know about their existence. It is all apart of long term marketing and branding.

> It is all apart of long term marketing and branding.

For sure - I can't think of any other companies with a similar approach to things though. I appreciate what they do, and the good quality write-up they provide as well.

If I remember right, years ago YouTube was releasing their prior year server build. I recall things like a cool air stream being directed right at the processor, rather than cooling the whole room enough for traditional cooling methods to work well. My Google fu isn't finding the blog posts I remember reading, probably from ~2010 era.

> Are there any other examples of companies doing similar releases of valuable internal data?

the early OkCupid blog was really good at this

This one has always stuck with me (archive version: https://www.gwern.net/docs/psychology/okcupid/whyyoushouldne...)

> It turns out you are 12.4 times more likely to get married this year if you don't subscribe to Match.com.

more archives: https://www.reddit.com/r/gwern/comments/aapn1l/okcupid_blog_...

Yev here -> I LOVED the OkCupid blog posts. So much good stuff in there. <3

I truly appreciate Backblaze publishing these but it's paramount to remember their disk usage pattern is fundamentally different than yours. This data is almost irrelevant to majority of their audience trying to deduce something out of it to help their buying decisions.

In addition to usage patterns, there's a lot more that contributes to mean time between failure of hard drives; like operating temperature (ambient, fluctuations, max), relative humidity, vibration and alike.

There's also well researched and published literature around this.

From Microsoft 2016: Finds relative humidity to be the dominant factor for disk failure https://www.usenix.org/system/files/conference/fast16/fast16...

From UToronto 2012: Finds no correlation with DRAM failures, nor high temp and disk failure but concludes temperature variability affects disk reliability more than maximum operating temperature http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf

From Google 2007: cannot find strong correlation between failure rate and temperature or utilization. https://static.googleusercontent.com/media/research.google.c...

From Microsoft 2013: focuses on operating temperature being the driving factor of failure, more than load. http://www.cs.virginia.edu/~gurumurthi/papers/acmtos13.pdf

Is there a data issue here? If you look at ST16000NM001G they reference a failed drive in Q2 2020 (https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2...)

I quote - >Note: The Seagate 16TB drive (model: ST16000NM001G) does show 59 drives and is listed in the report because the one failed drive had not been replaced at the time the data for this report was collected.

How can we be back to zero drive failures?

It seems a top cause of drive failure is affixing the word "Seagate" to the drive.

It's comforting that some things are consistent between the enterprise and consumer worlds.

I can anecdotally vouch for those 4TB HGST drives.

Bought about thirty H....4040ALE640 units several years back and they've had fantastic reliability. No failures, and IIRC only one's been replaced (as a precaution after its SMART Reallocated Sector Count ticked up). And no RAID hiccups whatsoever.

Where can a consumer find those drives in individual units in new non-refurbished condition? Amazon seems to be selling refurbished junk.

Ah yes it's that time of year for me to check up on how my lame photos are stored. Looking good chaps

Feelin' good!

Yev here -> I should add that we have a Q&A webinar going live tomorrow with Andy the author, you can join the Q&A here: https://www.brighttalk.com/webcast/14807/446738?utm_source=T....

Unfortunately this webinar is at my 4AM. I have a question about the drive data here - I've downloaded the Q3 data and it looks like there are 59 drives listed as at 2020/9/30 for the ST16000NM001G (and the same at 2020/7/1).

I compared it with the April 1 data set and it which appears to have 60. Serial number ZL202NFZ appears to be dropped in the later version and serial number ZL202400 seems to not be reading as at April 1 but is available at a subsequent date.

Just curious what's going on here! I recently received one of these and the backblaze drive data was very helpful with my evaluation.

While we're on the topic of enterprise hard drives, does anyone know who makes Hewlett Packard Enterprise drives?

https://www.cdw.com/product/hpe-hard-drive-600-gb-sas/450348...

We have a few of their machines and I've always been curious but haven't been able to find much.

Yev, since you seem to be in the comments, if it isn't company secret sauce, what is the primary driver to retire drives at Backblaze? Power consumption, storage density/colo space, drive days, mfg specifications?

Yev here -> Good question! The ops team may have a different theory, but I think it's when they die! We still have 15k 4TB HGST drives in our fleet that are crushing the game, and have been for years. That said, we'll eventually migrate off of them to get more density, but so long as we're not cramped for space, we tend to keep high-performing drives in our fleet for as long as we can. There is a math equation that flips after a certain point where we can get a lot more density by replacing those smaller-capacity drives - so that tends to be the other main driver for migrating the fleet.

*Edit -> For giggles here's an OLD post where we dissect drive failure and talk about migration a bit -> https://www.backblaze.com/blog/3tb-hard-drive-failure/

HGST drives are doing significantly worse than before. WD ruined the solid brand that fast?

Q2: https://www.backblaze.com/blog/wp-content/uploads/2020/08/Q2...

Q1: https://www.backblaze.com/blog/wp-content/uploads/2020/05/im...

Are you surprised?

Do you have a stash of cool magnets and shiny disks? I use dead HDD's disks as drink coasters. It's pretty cool.

Is there a reason why they don’t use WDC drives? Makes me nervous because I just picked up two during amazon prime sale

Table 1 (Lifetime Annualized Failure Rates) and Table 4 (Annualized Failure Rates for Q3) should be swapped, I believe.

Thanks for the heads up, fixed!

I don’t get why they keep buying seagates when every report they do shows they have higher fail rates?

Are they just cheaper?

I think they buy whatever their suppliers can get a hold of. They've said that they would buy HGST only, but they are too expensive.

Yes, by around half.

HGSTs are in short supply states-side right now and in high demand. Toshibas are also in limited supply here (at least the desirable helium-filled ones) and are about 2/3 between the Seagates and HGSTs.

Also in their product, drive failure is okay!

I am definitely finding Seagates the cheapest option at Microcenter, generally $5-$10 cheaper. Can't say I'm proud of cheaping out on storage, but that's my current price-point and haven't yet made the leap to trying to set up a properly-RAIDed NAS.

The price gap of high-capacity Enterprise drives is in the hundreds (as in multiple) of dollars between the two brands.

Just some recent numbers because I just replaced some drives in my home server...

I grabbed 2x14TB Seagate EXOS drives for $250USD each.

A 14TB WD Ultrastar looks to be a bit over $400USD each.

For 25% more you can get 200% the capacity with the Seagate drives. So for 25% more I can run RAID1 and have redundancy versus a single WD Ultrastar.

The HGST branded models (HC530, HC550) were around 530 two weeks ago.

i seem to remember in the past they said price and ease of procurement

Thank you for verbalizing a question I have always had haha.

Backblaze's usage pattern is NOT the same as yours with 99+% certainty. Their drive conditions are also unlikely to match yours.

They have a solid system set up for the installation and replacement of large numbers of drives on a regular schedule and the way they segment data would require multiple drive failures to cause data loss (I think it used to be 3 drives out of 20, but don't quote me on that). A slightly higher failure rate on a noticeably cheaper drive still works out in their favor financially without adding significant extra risk.

Yev from Backblaze here -> Most of the comments have it. They tend to be more affordable and the failure rates aren't that much different from their counterparts. Since we account for failure with our software layer a small failure rate delta isn't a big deal and is eclipsed our cost savings which we pass on to our customers!

Hi Yev! Great to see you here.

Curious if you have any impression of Toshiba's MN series? Especially the helium-sealed ones look compelling with their lower power consumption.

Yev -> Hey there! At the moment I don't believe we have any of those in our fleet (some MD and MG but no MN) so I don't have any data on them or how they'd function in our environment at the moment!

Thanks for explaining!

in comparison with WD, seagate price saving is significant?

If a drive has a 1% higher failure rate, but is 2% cheaper, it's still cheaper to buy it, and just run more of them in redundancy to cover the failure rate.

Interesting that they appear to not be deploying WD drives?

HGST is WD and their Enterprise tier now.

IIRC, Backblaze used to install consumer grade drives because they found that whatever they saved in increased reliability and warranty more was more than offset by the price savings with consumer drives. I didn't go through the SKU's, but I'm assuming some part of the equation changed?

HGST is effectively a brand name of WD at this point

https://en.wikipedia.org/wiki/HGST

Reply on news.ycombinator.com