Hacker News Reader

Comment by controversial97 | Hacker News Reader

Backblaze has stated in a blog post that they purchase drives in large quantities direct from manufacturers and negotiate for good prices. Backblaze says that a model of drive which is significantly cheaper and somewhat less reliable can be ok for them.

This means that these stats may not be a very meaningful indication when purchasing a few drives from a retailer.

Never the less, the world is full of situations where you have to make choices with insufficient data. I'm still going to prefer to avoid Seagate hard drives where practical.

View on HN

> they purchase drives in large quantities direct from manufacturers

Given that the manufacturer knows that Backblaze publishes its influential drive stats... do we know that the drive units that the manufacturer ships to Backblaze are representative of the quality of the same models available at retail?

Disclaimer: I work at Backblaze.

> Do we know if these drives are the same as you would purchase in a retail outlet

Seagate (for instance) won't sell ANYBODY drives directly, so they force us to get bids from various resellers and distributors. So when we pick the lowest price, I don't think Seagate knows who the drives are going to, but there might be a trick in there somewhere I am not aware of.

Have you considered becoming a distributor yourself? I would assume at a certain purchasing scale, the cost benefit is apparent, but perhaps not.

This is why every major tech company eventually becomes a domain registrar.

Do all major tech companies really register that many domains? I assumed that becoming a registrar was more about pushing your control up the pipeline to reduce risk if things like losing your domain name was the main reason. Rather than saving money on registrations.

If we assume that Backblaze gets special treatment from every manufactured who supplies it, that would suggest we're seeing stats that are the best the manufacturer can manage. Which seems a reasonable proxy for what they sell (lots of statistical quibbling about this view aside).

Any more special treatment than they give AWS, GCP or Azure? I feel they just manufacture them all the same because there's no point in doing anyone any favors.

I don't think anyone imagines that the hard drive manufacturers are making special drives for Backblaze. The question is 'Are they getting the cream of the crop?' out of all the drives that are manufactured.

Every drive must go through a series of tests before it is judged suitable for sale to the consumer. Drives that fall below a certain threshold are judged 'failed' but there can be a variety of drives that fall within the 'passed' range. Some are just barely above the threshold, others are way above.

It is kind of like binning for CPUs. The very best ones can have their base clocks increased and sold at a premium. The same kind of differentiation could be done for hard drives.

During the time that I interned at Imprimis in 1989 (just before they got bought by Seagate, and before I started my first "real" job at the Defense Information Systems Agency in the basement of the Pentagon, I know that I heard the manager of Manufacturing Engineering tell the engineers on multiple occasions that a certain manufacturer was coming through for a clean room visit in a certain number of days, and so the engineers had to clear the space of all drives that had certain types of failures.

They also definitely did a binning process where all drives being sold to certain manufacturers would be heavily tested for certain types of failure modes and only those that passed the most rigorous tests in a certain area would be approved to be sold to certain manufacturers.

Of course, each manufacturer had their own things that they were concerned about.

I don't know if they still do binning today, but they certainly did it back in 1989.

Oh, and even back then, Apple always got the cream of the crop.

You converge to the mean fairly quickly. Build a NAS of 12 drives and those stats become meaningful.

That being said with 50TB drives just around the corner [1] and SSD caching becoming the norm, 12 bays consumer NAS will become rare.

[1] https://www.anandtech.com/show/18733/seagate-confirms-30tb-h...

Why do you think that 50TB drives will make 12 bay NAS become rare?

We'll just hoard bigger files, like thus far.

My iPhone shoots 90 megabyte photos.

Thats is quite poor compression. My 26 mpix Fujifilm cameras makes ~30 MB compressed RAFs that for sure has better image quality in final output.

The iPhone is 48 megapixels.

Yea, clear demonstration why more pixels ≠ better quality. But ridicolous amount of data.

https://en.wikipedia.org/wiki/Computational_photography

Specifically, the night mode on recent (2y) devices.

I don't know, my own experience is that drive capacity increase exponentially but my storage need are fairly linear, bigger files but not that many more bigger files.

I've noticed that media files seem to look "good enough" at 720p for cartoons and 1080p for live action. I know some people obsess over the highest quality of everything (4k 10bit HDR) but even those came out in 2016[1] and aren't that popular. Higher definition like 8k is basically non existent, it can't run on old HDMI cables and needs both a beefy decoder (Blasting heat) and a powerful display driver as well.

[1] https://en.wikipedia.org/wiki/Ultra_HD_Blu-ray

At what point to increased rebuild times mean that the increased redundancy requirements overwhelm the increased capacity? If you're running large parity groups, the redundancy required is rather small, but when you move to smaller raid groups allowing for a second (or third) drive failure significantly impacts total storage.

Agree, but that's what backups are for. RAID5 isn't a backup, it's a way to stay online during the rebuild. If you are using RAID5 but don't backup your files, you probably don't care about those files that much.

If you want <X% chance of having to restore from backups, and rebuild times go up, you'll need to switch from RAID5 to RAID6 in order to maintain that.

RAID or not, if you lose an active drive you will need to 1) restore from backup, 2) fill a new backup drive

If it takes days to fill your new 50 TB backup drive, you have the same problem RAID5 has with rebuild times. The drive might fail in the middle.

RAID isn't a backup mostly because if you overwrite RAID data, you wrote over every copy at once. Non RAID offsite backups don't solve any problem related to drive size, they just make it a lot less likely for a single event to take everything out in the same minute.

Absolutely, you want RAID + backup, not one or the other.

50TB at 200MB/s is about 72h. Doesn't seem to be a particularly problematic rebuild time (and that's assuming 100% filled). Of course you need to do regular data scrubbing. If your rebuild is the first time the data is being read in 5y, that might not go so well.

> 50TB at 200MB/s is about 72h.

Assuming the device is offline for users during that time, otherwise there may be reduced throughput.

Yes, and this can be quite painful.

If you have standby sets (like a read replica of a db that can take over for master, or a DR site), you can switch temporarily.

It’s good to know what throughput you have under load for this reason among others.

Also, you can slow or pause some RAID during peak hours.

But if we are talking about a NAS, hopefully you also have SSD caching which should reduce a lot the workload on your disks.

> RAID or not, if you lose an active drive you will need to 1) restore from backup

RAID 1 joins the chat

> RAID or not, if you lose an active drive you will need to 1) restore from backup

Er what? I've had production machines lose a drive, have it replaced, and never leave production. Why would you need to restore from backups?

Right, that was said very poorly. What I really wanted to express was that you need to fill a new drive either way, either by resilver or by making a new backup. Point one is bad, not sure why I wrote that!

100% with you.

Seagate is the manufacturer with drives that fail at 10x the rate of the competition. I'll avoid, because drive failures are annoying.

The implication being that they might knowingly buy a subpar batch or that a manufacturer would offer rebates for a subpar batch, or what?

Not sure if that's what OP meant, but it could be that they get better batches from manufacturers, while retail gets the rest?

The other way you could read the comment would be that the stats don't mean much since Backblaze can tolerate higher defect rates if they negotiate a low enough price. But to me, that doesn't make sense since it wouldn't affect the failure rate statistics at all anyway.

If drive A has a 3% failure/year rate and drive B a 1% failure/year rate, but you can get 12 of drive A for the price of 10 drive B and you're going to have stock on hand to replace drives anyway and you're confidant in your backup strategy then for a business it may still make sense to buy drive A.

As a consumer, you're unlikely to keep an in-box spare, so even if you have a good backup system, a drive failure is still going to be an inconvenience, and leave you without redundancy until your replacement is shipped to you, so you might pay 20% more for drive A.

I don't follow. The statistics backblaze post doesn't implicate that you should buy the same drives they do.

It is because of theses statistics that you have (at least an inkling of) a fighting chance to pick the drive that more suits your usecase. Or perhaps a notion of brand performance.

Of course, you won't get statistics about drives that they don't buy. And if you then assume that more expensive drives have better failure rates then that might skew the data. But in there lies a lot of assumptions you really can't make. For one, that an expensive drive for a consumer is also an expensive for backblaze. And also, that there is correlation between end user price and failure rates.

Which generally I don't really think there is, within the alternatives that the consumer have access to (and reasonable price-ranges) at least. And it probably will vary between different markets as well.

Backblaze's article points out that sometimes they will buy drives that are less reliable but cheaper, as that makes sense for their business model.

The top level commenter points out that this strategy may not make as much sense for the individual consumer where you don't have backblaze levels of drives to amortise the failure probability and deal with it.

That's it, that's the point. It's not a rebuttal of the backblaze post at all, nor is it intended to be. The top level commenter was simply pointing out "don't buy mostly seagate drives because backblaze does" as backblaze explained their strategy in drive selection and some people may not consider the differences between their use case and backblaze's.

The whole point of the statistics is that you can pick what you want. Why would you dismiss the statistics and buy what backblaze does? Defeats the entire point of the statistics and article in the first palce.

Top level comment: >This means that these stats may not be a very meaningful indication when purchasing a few drives from a retailer.

That doesn't follow.

>Why would you dismiss the statistics and buy what backblaze does?

You assume people read the fine print and consider fine details.

The vast majority of Joes are going to ask who/what Backblaze is and what they're buying most and buy that, screw the details.

Even when they were shucking retail drives in the 1-4TB era, Seagate didn't have the greatest stats, with about a 5-30% failure rate (IIRC, over ~3 years or so), especially their 3T drives.

I ran an array with 1.5T and 3T Seagates and within a 1-3 years, I replaced at least 1/3 of my Seagates, but I made sure to replace the Seagates with Hitachis and Western Digitals, even though they were slightly more expensive.

ST3000DM001 was terrible one. Everyone prefer WD after that (and before the WD Red SMR gate)

I don't see so much significant higher failure rate on my 12TB Seagate drive array.

I really struggle to follow this logic.

You could argue to remove the driver that are represented less than 100 times (and it will be a stretch).

But once you have so many samples, there is no reason to not believe that your driver will not follow the same distribution.

Reply on news.ycombinator.com