Hacker News Reader

Comment by nneonneo | Hacker News Reader

A major problem with SSDs seems to be “firmware death” - where the flash chips are physically fine (or mostly fine), but the firmware (or firmware memory) has gotten corrupted due to some programming error, electrical glitch, or cosmic ray. I’ve had scores of older SSDs die after things like power outages and sudden shutdown events. This is super frustrating because the data is physically OK but the controller just isn’t responding to any requests anymore.

I wonder if there’s an easy way to distinguish a controller failure from a flash failure from the behavior of the device over the last few seconds/minutes of operation. In theory a controller failure should cause a fairly abrupt loss of service, but I’m sure there are soft lockup failure modes too.

View on HN

I have seen some weird issues with SSDs. I had an OCZ Vertex 2 die on me multiple times, but one thing that stood out most is that after a power cycle or complete system shutdown (note: reboots were just fine) everything I had done last time - install software, update Windows, create files - was gone. The state was reverted to before the time it booted. It was like my computer contained some kind of Reborn chip, except it was the Sandforce controller malfunctioning.

Modern SSDs can have very large DRAM write buffers (8-256MB), this is pretty plausible if it was failing to flush it.

It was probably some kind of wear leveling metadata that didnt get flushed.

Modern SSDs have incredibly large caches. For example, HP EX920, which is a TLC (triple layer cell) SSD, in its 1TB model contains a whooping 200GB of SLC (single layer cell) cache. It's entirely possible that your changes are simply in cache only, and hasn't been committed to the actual storage.

I don't know if I'm getting exactly what you're saying, but a sudden power off isn't going to wipe your SSD's SLC cache like it would do the DRAM cache on more expensive drives.

But if the controller gets into a weird state and "forgets" to flush the cache and then doesn't even look at the SLC cache on the next boot...

The SLC cache is managed by the controller, it's not going to forget to write it to the flash cells being managed as TLC/QLC, the controller is transparently storing your data in both. The controller knows if put some data in the SLC cache and some data in the QLC, and that that data needs to move from SLC to QLC, and that that other data in the SLC was marked deleted so now it can return that entire block to being addressed as QLC.

I don't know. I'm just trying to say that since flash isn't volatile the SLC cache isn't treated as volatile cache either.

It's probably more like the superblock not being updated to point to the newest data. Instead it points to old data that hasn't been garbage collected.

Like many early adopters, I too had a bunch of failed Vertex 2 drives and sometimes observed similar things. I think this might be because the drive lost some updates to its FTL about where it wrote new data, which could plausibly lead to both new files vanishing and changes to existing ones being apparently undone.

Yes, the Sandforce controller in Vertex 2 was a real unstable beast. I lost maybe three or four Vertex 2 drives. All under warranty, except the last one which suddenly vanished and was not detected anymore.

OCZ does not exist anymore. Not sure of if this was one of the causes, but I would otherwise never buy a drive from them again.

but the firmware (or firmware memory) has gotten corrupted due to some programming error, electrical glitch, or cosmic ray.

A lot of SSDs also store their main firmware in the same flash that is used to hold user data... this is something which was done with hard drives too (hence why dead/dying HDDs sometimes show up as a small drive with a weird name --- that's the "recovery mode").

What's involved in recovering the data if the controller fails?

just based on having seen data recovered from other flash devices, and the advertising copy of these devices:

hot-air rework to lift all the flash chips off and get them hooked up to something (probably custom) that can read them.

if the controller was encrypting everything that went to flash, you also get to try and find the key in the controller's memory.

why something custom? shouldn't you just solder the chips into an identical donor ssd?

a new controller is probably worthless to you, even if the old one wasn't storing the data encrypted. you don't have the map between real sectors and locations in flash.

your data is probably jumbled up one way or the other, but at least a custom board will let you read all of the underlying flash, instead of just the portions of it that a new controller would believe are in use. (keep in mind that the SSDs will have more physical blocks than they advertise logically.)

If that's the case, I wonder if controller failure could be prevented by ECC controller memory? Or would software failure recovery be sufficient to make a highly reliable controller?

With drives you used to be able to rip off a controller from an identical model and wire it back on to read the data.

With SSD or NVMe the controller isn't really a separate component you can just replace. Maybe it's possible to saw off the broken part and bodge-wire it to a working surrogate, but that would be extreme.

A tear-down of a broken SSD might reveal more about what could be done.

Does this explain why I often can access my 'dead' SSDs with my Ubuntu install rather than Windows?

I havent run into this issue for ~4 years, so I'm not sure if this is a solved problem or I got lucky a few years back.

Several years ago I was able to revive a "dead" SSD simply by updating the firmware to the latest version.

I haven't had a dead SSD problem in a while though, so I don't know how common firmware issues are now.

FWIW I switched entirely to using Samsung’s SSDs and haven’t had any issues. They seem to be above average for firmware quality, but I don’t have statistics to back that feeling up.

Sometimes the dead SSDs will respond to a handful of commands anyway, meaning that you can attempt a firmware upgrade or reset. That’s the lucky case. But in the majority of cases I’ve seen, the firmware/controller goes into some hard lockup where it no longer processes SATA commands at all. I wish these things had JTAG ports...

I have a laptop where the SSD wouldn't accept a Windows install, but worked fine with Linux. Months later, it took Windows with no issue...

Reply on news.ycombinator.com