But really isn’t the point of a journaling file system to make sure it is consistent at one guaranteed point in time, not necessarily without incidental data loss.
Journaling FSes are all about safety in the face of such things. That is, unless the drive lies.
But what if the front does fall off?
That would be a power failure. Kernel crash is not equivalent to that.
System reboot doesn't entail power failure either. The disks may be powered by an independent external enclosure (e.g. JBOD). All they see is that they stopped receiving commands for a short while.
That said, macOS does issue a flush on kernel panic, and I'm hoping to be able to do the same on Linux.
I measured the performance of different size direct I/Os on some Samsung SSDs, and the write performance when switching from one test to another was significantly affected by the previous test parameters, or not, depending on whether a "sleep 2" was inserted between each test.
The only explanation I can think of is that the flash reorganises or commits cached data during that 2 seconds.
Probably not on lockups though. A watchdog reset won't flush NVMe. Not sure if they have a special pre-fire path that tries a last ditch NVMe flush...
Actually it is (through a small one) to name some examples where it can still lose without full sync:
- OS crashes
- random hard reset, e.g. due to bit flips due to e.g. cosmic radiation (happens). Or someone putting their magnetic earphone cases or similar on your laptop or similar.
Also any application which care about data integrity will do full syncs and in turn will get hit by a huge perf. penalty.
I have no idea why people are so adamant to defend Apple in this case, it's pretty clear that they messed up as performance with full flush is just WAY to low and this affects anything which uses full flushes, which any application should at least do on (auto-)safe.
The point of a journalism file system is about making it less likely the file system _itself_ isn't corrupted. Not that the files are not corrupted if they don't use full sync!
This shit does happen.
> Specifically, if the drive loses power or the OS crashes, the application may find that only some or none of their data was written.
It's also worse then just write losses:
> The disk drive may also re-order the data so that later writes may be present, while earlier writes are not.
But reading it again there might have been some confusion about what was meant.
Battery-backed drives are free to ignore such commands. Those that aren't need to honor them. That's the point.
Battery- or capacitor-backed enterprise drives are intended to give you more performance by allowing the drive and indeed the OS to elide flushes. They aren't supposed to give you more reliability if the drive and software are working properly. You can achieve identical reliability with software that properly issues flush requests, assuming your drive is honoring them as required by the NVMe spec.
The problem is these drives don't provide integrity even when you explicitly ask them to.
There will always be trade-offs to any implementation. If you’re just using your M2 SSD to store games downloaded off Steam I doubt it really matters how well they flush data. However if your financial startup is using then without an understanding of the risks and how to mitigate them, then you may have a bad time.
I think much improved write performance is a good example of how it can be beneficial, with minimal risk.
Everything can be nice ideals of abstraction, until you want to push the envelope.
Not saying UPS-based integrity solutions don't make sense, you are right it's a tradeoff. The issue to me is more device vendors misstating their devices' capabilities.
UPS is not perfect though, it's better if your data integrity guarantees are valid independent of power supply. All that requires is that the drive doesn't lie.
> Specifically, if the drive loses power or the OS crashes, the application may find that only some or none of their data was written.
Except that _one time_ you need to work until the battery fails to power the device, at 8%, because the battery's capacity is only 80%. Granted, this is only after a few years of regular use...
Micron also talks about power loss resistant FTL design in their whitepaper, and although their older SSDs had caps, I think their recent ones mostly do away entirely.
https://www.micron.com/-/media/client/global/documents/produ...
This kind of stuff has been done in filesystems for ages (journaling etc.); there's no reason why FTLs can't be designed the same way.
> The hold-up circuitry also preserves enough time and energy to ensure that the FTL addressing table is properly saved to the NAND. This thorough amount of data protection not only ensures data integrity in unexpected power-loss events, but it also enables the system designer to leave the SSD’s write cache enabled, giving a significant advantage in data throughput speeds.
Last time I checked (and I could very well be out of date on this), there wasn't really a difference. It wasn't like an 18650 were the cells themselves have protection, but a cohesive power management subsystem that managed the cells more or less directly. It had all the information to correctly make such choices (but, you know, could always have bugs, or it's a metric that was never a priority, etc.).
Running lithium cells under 2.7v leads to permanent damage. But, I'm sure laptops have a built-in safety margin and can cut off power to the system selectively. That's why you can still usually see some electronics powered (red battery light, flashing low battery on a screen, etc) even after you "run out" of battery.
I've never designed a laptop battery pack, but from my experience in battery packs in general, you always try to keep safety/sensing/logging electronics powered even after a low voltage cutoff.
Even in very cheap commodity packs that are device agnostic, the basic internal electronics of the battery itself always keep themselves powered even after cutting off any power output. Laptops have the advantage of a purpose built battery and BMS so they can have a very fine grained power management even at low voltages.
I don't know if macOS explicitly does this, but it does try to hibernate whem the battery gets low, and that might be conservative enough to handle all normal cases.
Also, thanks for all you're doing with Linux on the new macs. I love your blog posts!