Hacker News Reader

Comment by Waterluvian | Hacker News Reader

"When a HD died early, you could also imagine undetected manufacturing flaws that finally gave way. With SSDs, at least in theory that shouldn't happen"

Why shouldn't it? Isn't it just hardware too?

"With spinning HDs, drives might die abruptly but you could at least construct narratives about what could have happened to do that"

Why can't you do the same with SSDs?

It feels like the author's main complaint is the frustration of not understanding SSD hardware as well.

Is this a valid complaint? Are SSDs magical in some way? I'm not an expert but... It's just hardware with pieces that do stuff. Why can't we come up with an understanding of why it fails?

View on HN

"It feels like the author's main complaint is the frustration of not understanding SSD hardware as well."

What is so frustrating about SSDs is how very poorly they compare to previous incarnations of solid state storage.

Using Disk-On-Chip and/or IDE-pin-compatible CF cards, I had many, many devices in the field that lasted, mounted read-only, for decades An entire sect of the computing industry came to rely on these parts as alternatives to spinning media that could not mechanically fail.

This is not the case with SSDs at all. They fail left and right, even mounted read-only, for all manner of complicated and interesting reasons. It's very frustrating that SSDs are not a step forward in reliability from spinning media and are a step downward compared to (for instance) a 16MB consumer CF card from Sandisk, circa 2000.

rsync.net filers, which need a boot mirror, are always constructed with two unrelated SSDs - usually one Intel part and one Samsung part - so that when the inevitable usage-related failure occurs, it does not occur simultaneously to both members of the mirror which have, being a mirror, been subjected to identical usage-lives.[1]

We shouldn't have to do that.

[1] I can't overstate this - if you need a RAID mirror, do not use identical SSDs for the two members of the mirror. There are many, many cases of SSDs failing not due to "wear" or end-of-life, but due to weird usage edge cases that cause them to puke ... and in a mirror, you give the two parts identical usage ... we either get two different generations of Intel part (current gen and just-previous gen) or we get current Intel and current Samsung ...

I've heard very similar advice for non-SSD mirrors too. Use different manufacturers or, at the very least, use different batches of disks from the same manufacturer.

Most importantly, never assemble an array from drives that took the same shipping path i.e. came out of the same parcel.

Most people use different batches but the same manufacturer, mostly because they remember advice from back when you could get a hardware raid card that would synchronize your scsi drives.

Also, though, because most serious RAIDs contain more drives than you can find manufacturers of hard drives.

On modern MLC/TLC SSDs, read-only mode doesn't really exist. The NAND blocks must be re-programmed after a number of read accesses to mitigate read disturb. If anything, mounting read-only is probably a corner case, stressing the firmware's read disturb mitigation.

The tradeoff from worse read disturb characteristics is NAND that is 100x cheaper per GB than in 2000.

Do you have any more details on the re-programming that would be occurring on non SLC flash cells, even if mounted in read only mode? This is something I was always concerned about too.

Every failure of an SSD feels like an exceptional event. Some harbinger of doom that needs to be shouted from the rooftops. The prions of storage.

But magnetic hard drives failed all the time. I have a giant stack in my office closet just from my dev machines over the years. But it wasn't new and scary -- it was just a hard drive failing -- so it was just normal. Some had controllers fail, suddenly blinking out of existence. Another had cache memory corrupt so it just gave ridiculous readings occasionally. Others had physical failures.

I don't know where to begin relative to prior flash memory (e.g. CF cards) which were absolutely notorious trash.

It is worth noting that every smartphone the world over has an "SSD" in it. We spend remarkably little of our mental power concerned about the flash storage. It is the cause of a negligible amount of device failures.

Yeah agreed 100%, for my RAID mirror setup I use drives from distinct manufacturers for that exact reason -- I can presumably expect a different failure rate (hopefully). :)

compared to (for instance) a 16MB consumer CF card from Sandisk, circa 2000.

That would almost certainly be small-block SLC flash rated for 100K program/erase cycles and 10 years of retention. The huge-block TLC now is ~1K program/erase cycles and 2-3 years of of retention depending on where you look (the manufacturers are not surprisingly quite reluctant to release these specs...)

"We shouldn't have to do that."

I don't know about that. I know most managers in embedded software go cheap on engineers and software assurance on purpose to get bonuses and such. The hardware side makes me think you haven't studied deep, sub-micron hardware much. I started looking into it a few years ago or so, just reading the slides on lots of stuff even though not understanding much of it. They helpfully put a lot in lay terms, though, with lots of comparisons. To say the stuff gets harder every time you shrink to a smaller node is an understatement, esp for solid state.

If anything, the modern flash should probably be considered broken right as they ship out of the factory. If not, the process nodes after 90nm or so just keep having more and more ways for individual components to screw up or change behavior across same wafer. Some happen instantly by design. Some happen later with aging. The memory technologies are closer to that analog level of things than most with harder verification. The high-density flash on newer nodes uses less-reliable tech than most just to operate at that low cost. So, they add all kinds of firmware and hardware tricks to try to make it work for a period of time like it's a whole, functional unit of storage despite pieces of it misbehaving all throughout.

It's a nice, man-made miracle these techs even work at all. Those that last longer like you mentioned still exist. I'll add a comment with links to one type so you can compare price/storage/performance to these broken-by-design SSD's you use. I'll throw in another two that mention shrinking challenges so you can see what they face every time they have to upgrade or just deploy new designs in mixed-signal.

https://news.ycombinator.com/item?id=18528023

https://www.electronicdesign.com/digital-ics/understanding-2...

https://anysilicon.com/ic-design-impact-in-moving-from-28nm-...

Well to be fair, there is an entire layer of abstraction at the SSD controller level that does tons of black box magic. It allows the OS to treat the SSD as another other storage device without letting the OS know what is going on.

So the combination of non-moving parts (making it hard/impossible to debug via physical inspection) combined with a tons of wear leveling/misc magic can defo make it seem like SSD's are magical.

The exact same thing happens hard drives. There's a ton of magic going on.

Lots of extra magic happens in an SSD though

https://www.youtube.com/watch?v=C5JoC4-qsO0 https://www.forensicswiki.org/wiki/Solid_State_Drive_(SSD)_F...

Thanks for this youtube link. In the case of recovering deleted files from SSD's that have the trim command enabled (Forensics Writeblocker was used,) the following drives have a low probability of recoverability: Crucial, Intel, and Samsung (3core controller). Whereas Seagate, Supertalent (parallel ATA to SATA bridge chip,) OCZ, and Patriot; files could be recovered. If the drive is quick-formatted, and if trim is enabled, the data is completely gone on the following drives: Crucial, Intel and Samsung. The trim state impacts the most whether or not the data can be recovered. You can check if trim is enabled with the command: fsutil behavior query DisableDeleteNotify - if your result is 0, it is enabled (default).

Yes, but most technical people seem to have a mental model that explains most failures pretty well. Not so for SSDs, apparently.

Is there a good reason why we use separate SSD controllers instead of letting the primary cpu handle it? The obvious reason is backwards compatibility, but as more of computing moves to SSDs, is this still relevant?

ZFS has shown that removing layers of abstraction with regard to storage can be beneficial.

https://azure.microsoft.com/en-us/blog/project-denali-to-def...

That adds a round of latency, and makes it pretty much impossible to boot off the drive. The blocks aren't in the same order in the Flash as they are presented by its interface, and one of the main jobs of the controller is to re-order them.

It would be an interesting product to have, a raw block API to a Flash device with all the temporary state stored on the host - but a hard one to sell, as it's not differentiated in any way.

There are host-managed SMR drives, which share a number of characteristics with flash storage.

Apple uses their T2 security enclave as the SSD controller and it's caused quite a stir.

I wonder if you can read any extra state out of the T2 as a result, e.g. more information on wear leveling, temporary read failures and so on, more than the standard SMART counters?

Where can I read more about this?

The controller makes it possible to get a standard bus and a pre-installed driver and use it to access any kind of memory from any manufacturer and any technology. It's the kind of convenience that makes people buy hardware - it's the kind of thing that made SATA and USB win. The alternative is that once in a while you plug a driver into your computer and it won't work.

Besides, I don't think manufacturers want to release the best practices for using their memory.

> Why can't you do the same with SSDs?

Because they don't die incrementally. Wtih a hard disk you'll get bad sectors, growing slowly over time. Or a head crash and then it's all dead.

What could cause an entire SSD to die at once? I would totally understand bad sectors, but the whole thing at once? Where it doesn't even try to read existing data?

One potential cause is poorly constructed drive firmware which fails to account for a minor failure of some kind and crashes. If that unaccounted minor failure is persistent, the firmware may crash constantly and then you'd be unable to access even theoretically good parts of the drive.

Firmware -- undebuggable, unobservable, unfixable software, jammed into your devices -- is the enemy.

Sounds like there is a business opportunist here for open firmware or even one running on the CPU especially for the enterprise? But maybe they won’t bother either despite valuable data and uptime, and just rather throw more redundant SSD’s at the problem.

You would need to convince the very small number of Flash memory fabricators to release their detailed specifications and known product weaknesses.

>What could cause an entire SSD to die at once?

A component failing? Electrical components fail. Sometimes it's a manufacturing defect, sometimes a design defect, sometimes something under or over-volted and it was enough to cause damage to any given component.

Could be an IC, could be a capacitor, could be a poorly laid trace. A poorly shielded RF source could even damage any number of components.

I mean, in theory a single high charge particle from that rare cosmic ray that reaches the surface of earth running full-steam-ahead through an IC could cause just the right amount of damage to make it fail although this would be an incredibly improbable scenario.

Same goes for HDDs, televisions, your clock radio, whatever.

> What could cause an entire SSD to die at once?

SSD has a problem in that almost every failure case looks more like a head-scratch than a bad sector.

Even the most local problem that is excessive writing on a sector can take the entire chip down.

There's a single wire in all computers, for some reason they always just have to include it, which if severed will completely disable the computer, as in, it won't even turn on, not even try to turn on.

That wire is the one delivering power. Point being, while things are so complicated that you often can have a lot going on and still try things, there are still single points of failure that can never be fully covered.

<i>Why shouldn't it? Isn't it just hardware too?</i>

In a mechanical hard drive, there are moving parts which can wear out due to friction, etc.

SSDs are solid-state, so it seems like at least theoretically, it should be possible to build one that keeps working for decades. e.g. I have solid-state hardware from the 70s and 80s which still functions.

I've always been a little mystified as to why SSDs' data areas wear out for that reason,[1] but that's a whole separate issue. The author of the article is writing about sudden failure of the device as a whole.

There are only two explanations I've ever heard for short lifetimes in electronics as a general industry.[2]

For devices manufactured after about 2000 there's tin whiskers,[3] which began to be a problem because of RoHS requirements. I'm not sure if that applies here, though.

If the device includes electrolytic capacitors, my understanding is that those generally have a finite lifetime as well, and it can be fairly short if they're poorly-made.

I'm not a hardware expert, though, so I'd be interested in hearing about other factors, and I'm sure the author of the article would too.

[1] I've seen lots of writeups of how wear-leveling works, etc., but never a good physical explanation of what is actually wearing out over time.

[2] Obviously there are other factors for specific devices, or specific designs. E.g. maybe parts of a specific device break over time due to thermal expansion and contraction if the device wasn't engineered to handle that properly.

[3] https://nepp.nasa.gov/whisker/background/

On the subject of capacitor death. Is there a robotic device that can take in a circuit board and swap out all the [electrolytic] caps?

i've never seen or heard of an automated board rework system. thinking about the steps and things i've had to do to manually rework boards, your machine would need not only be able to apply force to pull parts off of boards without damaging the board, but also be prepared to restore pads / through holes to usable states after desoldering parts, before new ones could go back in.

there's a reason companies don't repair circuit boards in consumer electronics.

I've done it myself, save restoring pads -- to me it's just the sort of fiddly, finnickety thing that it seems like robots sold be good at. And, they're not going to burn their fingers or run out of hands to hold things!

Robots are good at repetitive tasks and economy of scale. Doing a tricky action for ten thousand boards is something that robots are good at; doing a different tricky action on each of a hundred boards is not.

It doesn't seem plausible to have a business case where you'd get be able to get a large quantity of identical boards (that are otherwise good!), replace caps on them, and be able to sell them for much more than you got them - i.e. that the boards haven't become obsolete in that time. If there's no mass production, there's not much use for automation.

Yes, and it mostly speaks Ukrainian, or Chinese.

On 1): It's because each cell is essentially a consumable with a limited number of state transfers. It's similar in that way to how a solid state accellerometer still has a moving part inside it that can break or wear out over time.

EDIT: More reasons and methods for failure of "solid state" circuits and systems: https://news.ycombinator.com/item?id=14765868

I understand that there is something that makes the cells stop working over time. blattimwind's reply is the first actual explanation I've ever seen of what that something might be.

> I've seen lots of writeups of how wear-leveling works, etc., but never a good physical explanation of what is actually wearing out over time.

SSDs are flash are basically EEPROMs. In an EEPROM one bit is stored in a dual-gate MOSFET. One Gate is a normal gate, the other is floating, i.e. it just is a small conductive island. The information is stored by (quite literally) shooting electrons through the insulation of the floating gate into it. They're then trapped on the gate; if you turn the second gate on, the transistor conducts iff the floating gate is also turned on. This shooting action happens to damage the insulation, which at some point is degraded enough that it can't keep the electrons trapped on the floating gate. Hence the electrons leave, together with your information.

> For devices manufactured after about 2000 there's tin whiskers,[3] which began to be a problem because of RoHS requirements.

The solder joints themselves usually don't form whiskers, which mostly grow on pure tin-plated surfaces, e.g. the pins of components (which previously used lead). Using lead-free solder is mostly co-incident with whisker risk, not the cause of the majority of problems.

The electrons aren't shot through the insulation. They end up in the floating gate by quantum tunneling. The insulation by its name is non-conductive, and must be so that the electrons stay put. The degradation happens through general thermal wear.

The exact wear mechanism doesn't really matter (you must be referring to "hot carrier injection") --- the point is that to record data, electrons are being forced through a material which gradually wears it out.

That was also my point. I was just pointing out that the idea that electrons shoot through the insulation to end up in the floating gate was not true.

Thermal wear? I thought it was mostly based on write cycles. How can that be 'thermal'?

The write cycles are realised by moving electrons through a material. This creates heat, which leads to gradual degradation of the material.

Does the individual gate get that much hotter than the rest of the device?

If I keep my SSD 40 degrees cooler with a freezer, will it last an indefinite number of write cycles?

I just looked this up, and it turns out I was completely wrong about how the wear is caused. Apparently it’s due to a build-up of electrons remaining in the floating gate over time, and not general thermal wear. It turns out that heating can even reverse this wear. Explained in this link: https://arstechnica.com/science/2012/11/nand-flash-gets-bake...

> Why can't we come up with an understanding of why it fails?

Indeed, that is the question that the article's author seems to be asking.

Well, considering it's the cheapest consumer SSD they could afford, and then connecting it to a SAS port might be a whole story in itself.

The post resonated with me because of a stupid bug I hit in a ca 2011 SSD. (Samsung?) After 100 power-on cycles, the drive would brick itself. Required a firmware update in time to avoid.

That defect doesn't strike me as being inherently related to SSD media, but really left a bad taste in my mouth with what might be going on in the development process to lead to such instability.

I think the difference is familiarity. We've been using hard drives for decades so kinda know what to expect. SSDs have only been widely used 5-10y, so we're all just getting used to know how they work and how they die.

Reply on news.ycombinator.com