The SSD Endurance Experiment: Casualties on the way to a petabyte

I've recovered many hard disks that have gone bad in various ways, including multiple Seagate Barracuda 7200.11 drives that suffered from the BSY bug, and I've never had a disk that I couldn't recover at least some of the data from, even if that meant hooking up serial cables and re-reading the data multiple times (eg: Spinrite).

But a drive that disappears completely when things go bad is never a good outcome.

You will notice that their Enterprise SSDs do not self-brick.

[EDIT] Read down for clarification regarding "almost-dead"

I agree with you - I was surprised by the behaviour the article describes. That said, the obvious alternative of entering a permanent read only state is still problematic: how do you dispose of such a drive safely when you can't overwrite the data that remains? You'd need people to be aware of the secure erase command and continue to support that in read only mode.

> For a drive that voluntarily went into read only mode, I didn't expect it to get bricked, especially for a consumer drive. I am disappointed with Intel's engineers.

> I'd like to be able to read my data, even most of my data, if my SSD has failed. Disabling writes seems quite fair, disabling reads seems unnecessary and potentially cruel, depending on what data has been laid down since the last backup.

> ...I think it's a big deal, I'd want my SSD drive to tell me when it's time to move my data safely to another drive, right?

> Not only Intel but all three. Read only, even with some corrupt data, is far far better than becoming inaccessible. The three failed miserably, SMART warnings and writing-much-over-spec notwithstanding.

> ...I find it concerning since the SSDs appear to brick themselves instead of going into read only mode, once they have reached their write limits.

Seriously, it's the 21st centuary and everyone should always be making backups.

Think about it. If the SSD fails to erase, do you want that to mean it's readable forever? Would you be happy if the NSA was snooping around and your SSD refused to do anything other than disgorge secrets? This feature is good privacyware. If the drive can't be erased it also can't be read.

The ones I listed are the main wear indicators, for the others I'd worry if they had any value at all in the RAW column, even if the normalized value was not close to 0. If that happens then carefully research the implications of whichever one it happened to.

If this is a linux server then if you configure smartmontools correctly it will email you when it gets close.

I have quite some experience with Flash in the form of eMMC and SD/CF. SSDs aren't that much different from those on the low level.

The controller that comes with the flash storage contains a core that manages the bad blocks. Comparable to bad sector management on HD. The software these controllers run contain a lot of rules of thumb to manage bad blocks, which is where these full failures come from IMO.

Each controller has access to a pool of reserve blocks that are used when bad blocks are detected. Once those run out the embedded software starts showing weird behavior when using the device and shortly after there's a complete fail.

I think the pool of reserve blocks is "Used_Rsvd_Blk_Cnt_Tot" in your list. Apparently there are 100, of which you consumed 0. There's a threshold at 10 so I assume that's where the diagnostics software will warn you.

You should run the short and long smartctl tests occasionally to get more reliable results though (smartctl -t short and smartctl -t long)

Your drive looks in good shape in my opinion but thinks can always change suddenly so always backup.

For most rational definitions of reliable, absolutely. In this case inexpensive MLC or even TLC drives withstood hundreds of TB or more of writes before failing, clearly indicating their remaining lifespan the entire time. In the common applications of these drives, they're unlikely to ever see tens of TB of writes.

They can't last forever. It is odd that the Intel "bricks" itself, and that sounds more like a fault than anything (I'm at a loss to explain how that makes sense as a behavior, beyond maybe "hiding your data after you've lost the ability to wipe it"), but again it was clearly communicating the entire time that its death was imminent.

As an aside, that power test you linked was an extreme test of thousands upon thousands of abrupt power cuts while a write-back cache was enabled and populated, against a very small number of drives. It is not relevant to consumer products (where few or no drives have power retention), but it's even odder in the enterprise space, where power assurance is at the rack or even center scale, not component by component.

Compared to HDD, SSD failure modes are highly desirable and easy to handle.

Mirroring is really useful with spinning rust, where failures are (more or less) unpredictable and you need a spare to use while the first is replaced. I wonder if there's a better solution when the failure of the drive is more predictable that avoids having a second unit sitting there all the time - especially for applications where the data isn't absolutely critical.

and

> would not endanger data

in the same train of thought does not compute. Raid arrays do not protect your data. Only backups will do that.

In fact, striped raid arrays will tend to lose data faster than a drive by itself - when it comes time to calculate parity, they'll find an overlooked mistake and take out 2-3 more drives, writing off all the data on that cluster at once.

In any case, all the drives so far indicated they were destined to keel over via the SMART attributes long before it actually happened... Certainly long enough to buy a new drive and copy all the data, even for the less reliable TLC drive.

In fact even under a heavy workstation load you'd have enough time to save up money working a minimum wage part-time job, order the SSD when it goes on sale, wait for it to arrive via ground shipping, forget that you need an external SATA adapter, save up for that, order it, wait for shipping, then copy the data.

My original Intel X25-M G2, which I used for several years in my daily workstation, had around 4 TB total writes before I replaced it. That's two orders of magnitude lower than the apparent limit for these drives.

As opposed to hard drives, which are always so good about failing predictably /s

My take away was that SSDs are somewhat easy to predict failure on. I'd take predictable failure over unpredictable any day.

Even with only six subjects, the fact that we didn't experience any failures until after 700TB is a testament to the endurance of modern SSDs. So is the fact that three of our subjects have now written over a petabyte. That's an astounding total for consumer-grade drives, and the Corsair Neutron GTX, Samsung 840 Pro, and compressible Kingston HyperX 3K are still going!