SSD drives’ 40k-hour “death bug” continues to catch enterprises unawares

That, and the curse of working the same 18 months 4 times on the way to 'senior'.

All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008.

But for drives in particular, people are pretty superstitious about giving them up.

"Point and click" is unnecessarily derogatory. During my career as a developer I spent my share of time racking my own systems, diagnosing (both network and disk) cable faults, etc. I also spent time working with the kinds of software tools needed to manage dozens of clusters up to 10K machines each at 36 disks per machine. At that scale you really need to minimize the number of people who get to muck with the physical bits, and the amount of time they spend doing so. The complexity has merely shifted, not disappeared, and most developers or sysadmins from the previous generation would themselves be hard pressed to master the management of modern systems. It's no more "point and click" now than it was "lift heavy things" before.

> if all you know is cloud, you don't need to know things like this.

That's part of the cloud value proposition, isn't it? Leave the physical (and other low-level) stuff to people who do know that stuff, and can also reap economies of scale, while you focus on your own parts of the problem. Do you not approve of such specialization?

ClickOps is a term I heard second hand for this.

Once you cross a certain size, how else is it supposed to work, realistically speaking? My team administers servers in Brazil, Australia, Japan, Netherlands, the US, and several other countries. We can't do racking, cabling and hardware maintenance by ourselves. If we did, we'd spend 50% of our work hours just commuting on international flights. The only realistic option is to rely on local DC Ops teams to perform the manual steps involved in hardware buildup, maintenance and decom.

This isn't "SSD failure due to random overuse"

This is a specific bug, like an overflow that is consistently triggered after about 4 years of use.

Firmware updates on hardware have a very very low update rate outside of professional datacenters.

Uh, what? In most industries, it would be a recall and fix. The recall is used to notify people, and to pull unfixed products off shelves.

They also know about mixing brands/batches/etc. to avoid exactly this kind of correlated failure. During one of my oncalls, we had an uncomfortably large number of failures fitting this kind of pattern, but they were well enough distributed and we had enough data redundancy that there was no data loss. Sometimes there's safety in numbers, if you know how to make those numbers work for you rather than against you.

Mostly HDDs, though.

Or you can choose to install the firmware update fixing the issue and get back that sweet sweet uncertainty that your drive might live longer.

There's still a huge long tail, so iops matter. Netflix definitely wants to minimize any time you ever see a waiting on network spinner.

I dunno what talk the above comment is referring to but most likely the young hero was saying something somewhat clueless, not being kept down by the tyranny of older idiots or whatever is being insinuated. Netflix's content is pre baked. They have to do like a dozen versions of every episode to get the best experience across the variety of hardware, but it's not livestreaming or such. Where they use SSDs it's not an erase/update heavy workload, at least for the CDN part (Netflix does a ton of other stuff including basically running entire movie studios out of the cloud, not referring to that). Density and power also matter a ton as they want their POPs to be as easy as possible to convince people to run at peering points.

Netflix has a lot more to it than just serving video data

No idea now. I usually look to BackBlaze's periodic drive performance reports when I'm shopping, but even that's been a while for me.

samsung or intel would be strongly preferable in my opinion

For an SSD, you need to buy "Enterprise" or "Data Center" class drives for this kind of application. That class of drive have higher write endurance which is important since enterprise applications can have orders of magnitude more writes to their drives than a typical desktop.

That said, there's no real standard blanket vendor recommendation so long as the "Enterprise" criteria is met.

I'd imagine. Do they make stuff now that's intended to be spec'd for enterprise servers?

Testing that a circular buffer can wrap is a very very low bar though.

I'm not talking automated testes or anything. Just once, on your own machine.

I agree that it is a bit unfair to pick an example after the fact. But it does not inspire confidence.

A lot of preventable accidents come from the human brain discounting the threat after a large number of positive interactions. For all we know there once was a test three refactorings ago and someone fenced it away.