Power Failure Testing with SSDs

"...but nobody in the business of deploying production Posgresql should ever have been using the various models of desktop SSD that were tested." this is one of the reasons why this article was written. To show you how to test, to see what you should use and what you shouldn't in production. When we did these tests we already knew we shouldn't use those SSDs, we mostly did it just for the sake of testing, to see if the tests we wanted to use make any sense and to show others how to do it.

Also I know it may sound funny to use these SSDs in production, but when we started with our game Top Eleven we didn't know that, when you have 5 people working on a game, you don't have time (nor resources) to think about a type of an SSD; you order server with an SSD, you get an SSD (you usually don't even have a choice when you rent) and you use it. This is how it still works with almost any server renting company.

You're both right and wrong.

RAID controllers that you could afford for yourself, costing less than $1000, are just another point of failure. I've seen them fail more often than the disks, causing corruption as they went. However I've also seen the very expensive datacenter RAID controllers keep a whole bunch of servers up for years as their (15k rpm spinning rust) disks were failing and being swapped.

SSD problems with power failure are such that ZFS won't save you. ZFS's checksumming will at least know what's corrupted and what isn't. But the extent of the corruption will be practically unlimited if the OS can't trust the drive to honor write barriers / flushes - any byte since the last powerup could still be only in volatile cache on the SSD and never make it to persistent NAND.

Every filesystem depends on these occasional flushes/barriers to establish checkpoints where all previous writes are really written, including ZFS. Consider, it may have created an updated copy of the filesystem tree root node, made sure it was flushed, made another updated copy, made sure it was flushed, and then (only after the flushes) re-used the space that the original copy occupied for other data. If you can't trust that the flushes actually happened when the drive indicated they were completed, then it might be that only the last write, over the original root node copy, actually made it to the NAND.

I'm not sure that you could say that RAID controllers are a placebo - I've been through multiple hard disk failures that either hardware or software RAID has enabled a graceful recovery from (with no data loss).

As for whether or not it's superior to ZFS, though, that's a tricky question. High-end hardware RAID gets you a lot of neat features that you don't find elsewhere, but it's expensive and the RAID controller itself tends to become more and more of a point of failure (always keep spares, especially if they go out of manufacture!)

You and the author of the title are confusing cache layers and "battery protection." A RAID BBU will only protect the RAID controller's write cache; the thing that your write IO sits in until the controller flushes to the disk set. That's either 4-5 seconds depending on Dell/LSI model or until it gets pushed out for cache capacity.

The above has absolutely zero to do with the drive write cache and power protection. The drive write cache is used to cache the write on the drive after the RAID controller. Spinning metal drives have caches. SSD's have caches. Whether you use the cache is up to you and your OS. The safest setting is off. You can usually get more performance by leaving it on. This is why Windows would throw warnings all over the place when enabling write caching without a connected UPS. I'm not sure exactly what the logic is on Linux for sd/sg devices.

The default Dell RAID controller behavior with write caching on drives is based on drive interface. SAS = off. SATA = on. If you run non-power safe SATA drives with a Dell RAID controller, you must disable the write cache if you want data durability.

Then again, you could also run your databases on power-safe, non-consumer SSD's. This whole article was basically a giant warning sign saying "run away they need serious help."

I worked in Dell's High complexity support for a couple of years. I saws RAID disk failures all day long. I BELIEVE SOFTWARE RAID IS YOUR FRIEND. I've seen hardware RAID system become corrupted because a miner version of the controller software was used after a failure. Except for multiple platter crashes, there has not been a software raid (linux) and Spinright there has never been a RAID I couldn't recover. So... Hardware Raid is NOT worth the extra speed.

>Am I the only one who thinks RAID controllers are a placebo and wouldn't trust anything but ZFS?

No, there are others that cargo-cult believe in ZFS too

-- a hyped single vendor OS, without first-tier-support on Linux, and with its own issues, compared to a industry wide standard, used for 3 decades in the most demanding data-centers protocol and its implementations.

I currently deal with a few disk failures every week, all on RAID6 (on Dell H7xx controllers) and a few RAID-1 on HP and Adaptec controllers. Mostly SAS drives, but a few older model SATA SSD drives (that are dropping like flies after lots of writes over a year). This is across about 32000 disks so the chances for some disk failure every week is high. The RAID controllers work as advertised almost always, file system intact. Sometimes there's a performance degradation during a disk rebuild, but only on arrays where disk i/o is near max ability. There has been, I think two, catastrophic failures, for example when there's an issue with the cable or backplane, causing corruption, then having more than two disks go corrupt before getting to the bottom of it and invalidating the entire array. That said we have very few issues with power, and the batteries have died on many of the controllers, so I'm not sure how it will pan out on dodgy data centre power flapping.

Most non-enterprise SSDs do not have an internal supercap or other power protection mechanism and they are not intended for a server use-case and shouldn't be used in such capacity.

An HDD will not hold data that is written to its write cache either so the SSDs are well within the spec.

Yeah I know that these are old drives, we've done this testing about year and a half ago and when we started renting servers more than 5 years ago, we didn't think about which SSDs are in there. At that time we didn't have an option to choose, nor time to think about the implications of different SSD models, we used what we got. Later on when we grew and started having problems, we started investigating what models we should use. The main reason for writing this article is for people to be aware that they should test their drives and how to do it.

In theory, yes. But in practice the drive would still see power applied and would not trigger the 'dump the cache' firmware until it was too late. You need a little bit of extra circuitry to warn in time before the power gets too low.

Usually this works like this: an external voltage powers the drive, inside the drive this voltage is split into two circuit paths. One drives a voltage regulator with a large capacitor on the far side of it, the other drives, usually accompanied by a pull down resistor an input of the cpu in the drive. If that input goes below a specified value then the cpu knows the power is about to fail and will trigger the cache write to the persistent media.

Obviously that only holds for the data already present in the memory of the CPU on the drive, the rest of the computer will have to fend for itself.

So just adding a capacitor on the outside will keep the drive powered up for half a second longer but won't initiate a cache dump (assuming that you're not going to end up powering the rest of the circuitry as well, you need to add a diode or something to avoid backfeeding the circuit that charged the cap in the first place).

I have developed power loss protection for SSDs in the past. You also need to give an early warning "power failed" signal to the controller to dump its internal buffers and state.

Power loss can be very challenging to get robust especially if the controller uses clever algorithms at runtime to get better performance because recovering the drive state from sudden power loss is more difficult. That is why I think the Intel 520 controller has so many problems because they are using a SandForce controller which was known to use no external DRAM and compression algorithms which just complicates things.

There is nothing that makes SSDs with power loss protection inherently expensive. As always, it's just market segmentation...

A large supercap in the 100-200mF range for an SSD is around $1 or $2. In fact you can implement power loss protection with less capacitance with regular tantalum caps like the Intel 320 did (http://www.storagereview.com/intel_ssd_320_review_300gb). But drive manufacturers see the consumer market doesn't care about power loss protection, so they decide to scrap the feature, which saves a buck or two, and saves some PCB space.

It is not just power outages that can corrupt SSDs, it is any unexpected power loss without the preceding SATA commands, so I think this would include unexpected OS reboots, kernel panics, etc. [1]

I've recently bought a new SSD and was searching for information on power loss protection, and the only vendor documentation on the matters seems to be [1] and [2]. SSD reviews have plenty of performance numbers (that interest me less), and besides sometimes describing what the vendor says about power loss protection they don't perform any actual testing for power loss protection, and end up being fooled by the vendors sometimes [3]. The only actual test I found was [4].

Capacitors are one way of protecting, but for some reason even some of the newer enterprise SSDs sometimes have them, sometimes not, even if older versions had them. Some SSDs claim to have journaling (on SLC NAND) instead, but given how the firmware is closed source there is no way to inspect it for bugs.

[1] "Storage devices require a graceful removal of power to ensure data integrity is preserved. Graceful removal of power includes commands to signal to the storage device that power might be imminently removed" http://www.sandisk.com/Assets/docs/Unexpected_Power_Loss_Pro...

[2] http://www.intel.com/content/dam/www/public/us/en/documents/...

[3] http://www.anandtech.com/show/8528/micron-m600-128gb-256gb-1...

[4] http://lkcl.net/reports/ssd_analysis.html

The capacitor doesn't itself significantly increase the price but there are more differences than just a capacitor in the datacenter drives. Look at the power consumption for the start, DC ones use much more power on idle. Also look at the technology behind: DC variants have to be more conservative with the stress to the cells (less bits per cell, potentially less cells per area, potentially more spare cells to replace died ones etc).

And the market segmentation is the natural way to go. The different needs and different sensitivity to price determine the price.

You need a trigger as well so when the power rail voltage drops, the firmware dumps its cache. With a substantial write-back cache, the capacitor or battery needs to be sized sufficiently large to write out potentially hundreds of megabytes.

Supercaps are themselves expensive, bulky, and may not be reflow-compatible.

Indeed. See Luke Leighton's excellent "Analysis of SSD reliability during power outages" at http://lkcl.net/reports/ssd_analysis.html which came to the same conclusion some years ago.

It would be interesting to see them use Samsung SM843, which would be more comparable to Intel S3500.

> CentOS 6.5 was used, SSDs were formatted with XFS and they were mounted into /mnt/ssd1 and /mnt/ssd2. XFS was used because that is our main file system for databases.

write barriers is a mount option [1] [2].

[1] http://linux.die.net/man/8/mount

[2] http://lwn.net/Articles/283161/