Apple's custom NVMes are amazingly fast – if you don't care about data integrity
twitter.com
twitter.com
POSIX spec says no: https://pubs.opengroup.org/onlinepubs/9699919799/functions/f...
Maybe unrealistic expectation for all OSes to behave like linux.
Maybe linux fsync is more like F_BARRIERFSYNC than F_FULLFSYNC. You can retry with those for your benchmarks.
Also note that 3rd party drives are known to ignore F_FULLFSYNC, which is why there is an approved list of drives for mac pros. This could explain why you are seeing different figures if you are supplying F_FULLFSYNC in your benchmarks using those 3rd party drives.
OpenBSD, though, apparently behaves like macOS. I'm not sure I like that.
That doesnt make it worse - in fact it permits the flexibility you are now struggling with.
edit: downvotes for truth? nice. go read the posix spec then come back and remove your downvotes...
There is no real excuse for a single sector write to take ~20ms to flush to NAND, all the while the NAND controller is generating some 10MB/s of DRAM traffic. This is a dumb firmware design issue.
Why not compare macOS and linux on approved x86 mac hardware. i.e. fusion drive or whatever.
Also, as suggested - try F_BARRIERFSYNC, which flushes anything before the barrier (used for WAL IIRC).
We've looked at NVMe command traces from running macOS under a transparent hypervisor. We've issued NVMe commands outside of Linux from a bare-metal environment. The 20ms flush penalty is there for Apple's NVMe implementation. It's not some OS thing. And other drives don't have it. And I checked and Apple's NVMe controller is doing 10MB/s of DRAM memory traffic when issued flushes, for some reason (yes, we can get those stats). And we know macOS does not properly flush with just fsync() because it actively loses data on hard shutdowns. We've been fighting this issue for a while now, it's just that it only just hit us yesterday/today that there is no magic in macOS - it just doesn't flush, and doesn't guarantee data persistence, on fsync().
I couldnt find much info about it - but the official docs are here: https://kernel.org/doc/html/v5.17-rc3/block/writeback_cache_...
Im not sure what commands are being sent to the NVMe drive. But what you are describing as a flush would be F_BARRIERFSYNC - NOT the F_FULLFSYNC which youve been benchmarking.
I don't know why you keep pressing on this issue. macOS has the same performance with F_FULLFSYNC as Linux does with fsync(). Why would they be different things? We're getting the same numbers. This entire thing started because fsync() on these Macs on Linux was dog slow and we couldn't figure out why macOS was fast. Then we found F_FULLFSYNC which has the same semantics as fsync() on Linux. And now both OSes perform equally slowly on this hardware. They're obviously doing the same thing. And the same thing on Linux on non-Apple SSDs is faster. I'm sure I could install macOS on this x86 iMac again and show you how F_FULLFSYNC on macOS also gives better performance on this WD drive than on the M1, but honestly, I don't have the time for that, the isssue has been thoroughly proved already.
Actually, I have a better one that won't waste as much of my time.
Plugs in a shitty USB3 flash drive into the M1.
224 IOPS with F_FULLFSYNC. On a shitty flash drive. 58 IOPS with F_FULLFSYNC. On internal NVMe.
Both FAT32.
Are you convinced there's a problem yet?
(I'm pretty sure the USB flash drive has no write cache, so of course it is equally fast/slow with just fsync(), but my point still stands - committing writes to persistent storage is slower on this NVMe controller than on a random USB drive)
Sure fsync allows that behavior, but also it's so widely misunderstood that a lot of programs which should do a "full" flush only do a fsync, including Benchmarks. In which case they are not comparable and doing so is cheating.
But that's not the point!
The point is that with the M1 Macs SSDs the performance with fully flushing to disk is abysmal bad.
And as such any application with cares for data integrity and does a full flush can expect noticable performance degradation.
The fact that Apple neither forces frequent full syncs or at least full syncs when a Application is closed doesn't make it better.
Though it is also not surprising as it's not the first time Apple set things up under the assumption their hardware is unfailable.
And maybe for a desktop focused high end designs where most devices sold are battery powered that is a reasonable design choice.
Does the battery last forever? Do they never shut down from overheating, shut down from being too cold, freeze up, they are water and coffee proof?
Talk to anyone that repairs mac about how high-end and reliable their designs trully are - they are better than bottomn of the barrel craptops, sure, but not particularly amazing and have some astounding design flaws.
Sequential logic (flipflops) has a setup time requirement. This means the combinatorial computation between any two connected pairs of flops (output of flop A to input of flop B) has to do its job fast enough such that the input of B stops toggling some amount of time before the next clock edge arrives at the flipflop. Violate that timing, and B will sometimes sample the wrong value, leading to an error.
Setup time is what most people are thinking about when they use LN2 or other exotic forms of cooling. By cooling things down, you usually improve the performance of combinatorial logic, which provides more setup time margin, allowing you to increase clock speed until setup time margin is small again.
But flops also have hold time requirements - their inputs have to remain stable for some amount of time after the clock edge, not just before. It's here where we can run into problems if the circuit is too cold. Imagine a path with relatively little combinatorial logic, and not much wire delay. If you make that path too fast, it might start violating hold time on the destination flop. Boom, shit doesn't work.
Edit: For actual temperatures, in my experience its when the device is in use for a sustained amount of time in under 10f weather
Luckily they often operate in lower temperatures too, but not seldomly by hoping they don't get cooled that much themself (because they are e.g. in your pocket).
Fahrenheit, Celsius, horizontal angle?
Spilled drinks are a viable cause for concern, but if they do enough damage to cause an unexpected shutdown, you've probably got bigger issues than unflushed cache.
On many laptops even with water damage you can recover your local data fully, not do for Macs (for more reasons then just data loss/corruption due to non flushing).
Especially if you are already in a bad situation you don't want your OS to make it worse.
I have never heard of this functionality being included in any major OS, could you please provide some reference to this being documented?
And yes the hardware is failable. But the kind if failure that would cause the device to completely lose power is extremely rare. The OS has many chances to take the hint and flush the cache before powering down.
Note: this is pure conjecture.
How sure are we the drives that flush caches more quickly are actually flushing the caches?
A simple test can be to see the degree of dataloss you can occur with a hard power off.
I think the author did that test for M1 Mac but idk. if they did the test with the other laptops.
But then the M1 Mac is slower when flushing then most SSDs out there and even some HDDs. I think if most SSDs wouldn't flush data at all we would know of that and I should have run into problems with the few docent hard resets I ran into in the last few years. (And sure there are probably some SSDs which cheap out on cache flushing in a dangerous way, but most shouldn't as far as I can tell).
I'm sorry but this is incorrect. NeXTSTEP was the primary foundation for Mac OS X, and the XNU kernel was derived from Mach and IIRC 4.4BSD. FreeBSD source was certainly an important sync jumping off point for a number of Unix components of the kernel and CLI userland, there was some code sharing going on for a while (still?), but large components of the kernel and core frameworks were unique (for better or worse).
4.3, only Rhapsody incorporated elements from 4.4, but that was the tail end of nextstep, essentially the initial preview of macos (it was released as osx server 1.0, then forked to darwin from which the actual OSX 10.0 would be built, two major pieces missing from rhapody were Classic and Carbon, so it really was nextstep with an OS9 skin).
Everything is a million times more refined and overall better now but I do have a bit of nostalgia for the community and really getting your hands dirty back then while still having a fairly decent fallback. I haven't actually needed to mess with kernel stuff since 10.5 or so but thinking back makes me wonder about paths not taken.
Sorry to be pedantic, but Rhapsody's user interface is modeled after the Mac OS 8 "Platinum" design language. Though 9 also was modeled on Platinum, Rhapsody's interface appears nearly identical to Mac OS 8's except for the Workspace Manager which doesn't exist in 8.
It has been over a decade, so I'm really not sure how much is left ATM.
Regardless, I certainly agree that the performance hit seems excessive. Hopefully it's just an algorithm, issue and Apple can fix this with a software update.
So, as of about 2014, any difference here not being backed by per manufacturer secret knocks or NDAed, one-off drive firmware was just a magic show, with perhaps Linux at least being able to say "hey, at least the kernel tried and it's not our fault". The cynic in me thinks that the BSDs continuing to define fsync() as only hitting the drive cache is to keep a semantically clean pathway for "actually flush" for storage appliance vendors to stick on the side of their kernels that they can't upstream because of the NDAs. A sort of dotted line around missing functionality that is obvious 'if you know to look for it'.
It wouldn't surprise me at all if Apple's NVME controller is the only drive you can easily put your hands on that actually does the correct things on flush, since they're pretty much the only ones without the perverse market pressure to intentionally not implement it correctly.
Since this is getting updoots: Sort of in defense of the drive manufacturers (or at least stating one of the defenses I heard), they try to spec out the capacitance on the drive so that when the controller gets a power loss NMI, they generally have enough time to flush then. That always seemed like a stretch for spinning rust (the drive motor itself was quite a chonker in the watt/ms range being talked about particularly considering seeks are in the 100ms range to start with, but also they have pretty big electrolytic caps on spinning rust so maybe they can go longer?), but this might be less of a white lie for SSDs. If they can stay up for 200ms after power loss, I can maybe see them being able to flush cache. Gods help those HMB drives though, I don't know how you'd guarantee access to the host memory used for cache on power loss without a full system approach to what power loss looks like.
Apple implementation is weird because actual amount of data written doesn't seem to affect flush time.
FWIW, macOS also has F_BARRIERFSYNC, which is still much slower than full syncs on the competition.
My understanding was, the capacitor thing on HDD's is to ensure it completely writes out a whole sector, so it passes the checksum. I only heard the flush cache thing with respect to enterprise SSD's. But I haven't been staying on top of things.
WRT to the capacitor thing being about a single sector, think about the time spans. You should be able to even cut out the drive motor power, and still stay up for 100s of ms. In that time you can seek to a config track and blit out the whole cache. If you're already writing a sector you'll be done in microseconds. The whole track spins around every ~8ms at 7200RPM.
Of course, this approach describes a completely inverted complexity scenario in terms of sector remapping management, with the size of the associated tables probably being orders of magnitude larger. :<
Now I wonder how much power is needed for flash writes. The chances are an optimal-and-viable strategy would probably involve a bit of multichannel flash on the controller (and some FEC because why not).
Oooh... I just realized things'll get interesting if the non-volatile RAM thing moves beyond the vaporware stage before HDDs become irrelevant. Last-millimeter write caching will basically cease to be a concern.
But thinking about the problem slightly more laterally, I don't understand why nobody's made inline SATA adapters with RAM, batteries and some flash in them. If they intercept all writes they can remember what blocks made it to the disk, then flush anything in the flash at next power on. Surely this could be made both solidly/efficiently and cheaply...?
Hardware raid controllers with Battery Backup Units was really popular starting in the mid 90’s until maybe mid 2010’s? Software caught up in a lot of features and batteries failed often and required a lot more maintenance. Super caps were to replace the batteries but I think SSDs and software negated a ton of the value add. You can still buy them but they’re pretty rare to see in the wild.
An inline widget (SATA on both sides) that just implements a write cache and state machine ("push this data to these blocks on next power on") seems so much simpler. You could even have one for each disk and connect to a straightforward RAID/SAS controller. (Hmm, and if you externalize the battery component, you could have one battery connect to several units...)
You are indeed right about the battery/capacitor situation ("you have to open the case?!"), I wouldn't be surprised if the battery level reporting in those RAID cards was far from ideal too lol
With all this being said, a UPS is by far the simplest solution, naturally, but also the most transiently expensive.
And also unbearable slow for loopback device backed docker container in the vm due to double layer of cache. I just add eat-my-data happily because you can't save a half finished docker image anyway.
This article is a non-issue, people just like to upvote Apple bashing.
Macs in datacentres are becoming increasingly common for CI, MDM, etc.
Perhaps.
> MDM
I’m sure my IT department’s spyware will recover just fine.
Doesn’t seem like an issue worthy of hundreds of HN comments and upvotes, just people raising a stink over a non-issue.
Note: the tweeter couldn't provoke actual problems under any sort of normal usage. To make data loss show up he had to use weird USB hacks. If you know you have a battery and can forcibly shut down the machine 'cleanly' it's not really clear what the need for a hard fsync is.
"Macs in datacentres are becoming increasingly common for CI, MDM, etc."
CI machines are the definition of disposable data. Nobody is running Oracle on macOS and Apple don't care about that market.
And how do you know they didn't, did you do a poll?
How many people had random files dissapear or get corrupted or settings get reset and probzbly thought they must have done something wrong?
At least the OSX man page admits to the detail.
The rationale in the POSIX document for a null implementation seems reasonable (or at least plausible), but it does not really seem to apply to general OSX systems at all. So even if they didn't define _POSIX_SYNCHRONIZED_IO it would be against the spirit of the specification.
I'm actually curious why they made fsync do anything at all though.
Nope: https://opensource.apple.com/source/Libc/Libc-1439.40.11/inc...
Okay, so OSX is right by the letter of the standard. Not by the spirit though, when you look at the rationale for allowing the exception.
TBH, im not so sure its that different. Scanning through the linux docs it seems that this behaviour can be configured as part of mount options (e.g. barrier on ext4). At least its explicit on macOS (with compliant hardware).
No just I did a ctrl+F ctrl+C ctrl+V without thinking enough. No need to apologize though, my reply was actually flippant I should have been more respectful of your (correct) point.
> TBH, im not so sure its that different. Scanning through the linux docs it seems that this behaviour can be configured as part of mount options (e.g. barrier on ext4). At least its explicit on macOS (with compliant hardware).
I disagree (unless Linux short-cuts this by default). The reason is in the POSIX rationale:
*RATIONALE*
> The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk. Since the concepts of "buffer cache", "system crash", "physical write", and "non-volatile storage" are not defined here, the wording has to be more abstract.
The first paragraph gives the intention of the interface. It's clearly to persist data.
> If _POSIX_SYNCHRONIZED_IO is not defined, the wording relies heavily on the conformance document to tell the user what can be expected from the system. It is explicitly intended that a null implementation is permitted. This could be valid in the case where the system cannot assure non-volatile storage under any circumstances or when the system is highly fault-tolerant and the functionality is not required. In the middle ground between these extremes, fsync() might or might not actually cause data to be written where it is safe from a power failure. The conformance document should identify at least that one configuration exists (and how to obtain that configuration) where this can be assured for at least some files that the user can select to use for critical data. It is not intended that an exhaustive list is required, but rather sufficient information is provided so that if critical data needs to be saved, the user can determine how the system is to be configured to allow the data to be written to non-volatile storage.
Now this gives a rationale for why you might not include it. And lists three examples of where it could be valid to water down the intended semantics. The system can not support it; the functionality is not required because data durability is guaranteed in other ways; the functionality is traded off in cases where major risks have been reduced.
OSX on a consumer Mac doesn't fit those cases.
Linux with the option is violating POSIX even by the letter because presumably mounting the drive with -onobarrier does not cause all your applications to be recompiled with the property undefined. But it's not that unreasonable an option, it's clearly not feasible to have two sets of all your software compiled and select one or the other depending on whether your UPS is operational or not.
However if in fact it's quite common for other OS's to exhibit the same or similar behaviour (BSD for example does this too, which makes sense as OSX has a lot of BSD lineage), that argument of least surprise falls a bit flat.
That's not to say this is good behaviour, I think Linux does this right, the real issue is the appalling performance for flushing writes.
SQLite, MySQL et al. [1] fall back to `fsync()` if F_FULLFSYNC fails, in order to cover this case of 3rd party or external drives.
[1] https://twitter.com/TigerBeetleDB/status/1422855270716293123
From that page:
The fsync() function shall request that all data for
the open file descriptor named by fildes is to be
transferred to the storage device associated with the
file described by fildes. The nature of the transfer
is implementation-defined. The fsync() function shall
not return until the system has completed that action
or until an error is detected.
The fsync() function is intended to force a physical
write of data from the buffer cache, and to assure
that after a system crash or other failure that all
data up to the time of the fsync() call is recorded
on the disk. Since the concepts of "buffer cache",
"system crash", "physical write", and "non-volatile
storage" are not defined here, the wording has to be
more abstract.
Crashes and power failures are just not part of the POSIX worldview, so in POSIX there can be no need for sync(2) or fsync(2), or fcntl(2) w/ F_FULLFSYNC! Why even bother having those system calls? Why even bother having the spec refer to the concept at all?
Well, the reality is that some allowance must be made for crashes and power failures, and that includes some mechanism for flushing caches all the way to persistent storage. POSIX is a standard that some real-life operating systems aim to meet, but those operating systems have to deal with crashes and power failures because those things happen in real life, and because their users want the operating systems to handle those events as gracefully as possible. Some data loss is always inescapable, but data corruption would be very bad, which is why filesystems and applications try to do things like write-ahead logging and so on.
That is why sync(2), fsync(2), fdatasync(2), and F_FULLFSYNC exist. It's why they [well, some of them] existed in Unix, it's why they still exist in Unix derivatives, it's why they exist in Unix-alike systems, it's why they exist in Windows and other not-remotely-POSIX operating systems, and it's why they exist in POSIX.
If they must exist in POSIX, then we should read the quoted and linked page, and it is pretty clear: "transferred to the storage device" and "intended to force a physical write" can only mean... what that says.
It would be fairly outrageous for an operating system to say that since crashes and power failures are outside the scope of POSIX, the operating system will not provide any way to save data persistently other than to shut down!
MacOS does that.
> the fsync() function is intended to force a physical write of data from the buffer cache
If they define _POSIX_SYNCHRONIZED_IO, which they dont.
fsync wasnt defined as requiring a flush until version 5 of the spec. It was implemented in BSDs loooong before then. Apple introduced F_FULLFSYNC prior to fsync having that new definition.
I dont disagree with you, but it is what it is. History is a thing. Legacy support is a thing. Apple likely didnt want to change peoples expectations of the behaviour on OSX - they have their own implementation after all (which is well documented, lots of portable software and libs actively uses it, and its built in to the higher level APIs that Mac devs consume).
> MacOS does that.
Depends on the definition of "storage device", I guess. If it's physical media, then OS X doesn't. If it's the controller, then OS X does. But since the intent is to have the data reach persistent storage, it has to be the physical media.
My guess is that since people know all of this, they'll just keep working around it as they already do. Newbies to OS X development will get bitten unless they know what to look for.
> The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk.
This seems consistent with user expectations - fsync() completion should mean data is fully recorded and therefore power-cycle- or crash-safe.
int fsync(int) {}
> The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes.
If I wrote that requirement in a classroom programming assignment and you presented me with that code, you'd get a failing grade. Similarly, if I were a product manager and put that in the spec and you submitted the above code, it wouldn't be merged.
> You are quoting the non-normative informative part
Indeed, I am! It is important. Context matters, both in law and in programming. As a legal analogy, if you study Supreme Court rulings, you will find that in addition to examining the text of legislation or regulatory rules, the court frequently looks to legislative history, including Congressional findings and statements by regulators and legislators in order to figure out how to best interpret the law - especially when the text is ambiguous.
It's a good thing operating systems aren't made up entirely of classroom programming assignments.
Picture an OS which always runs on fully-synchronized storage (perhaps a custom Linux or BSD or QNX kernel). If there's no write cache and all writes are synchronous, then fsync() doesn't need to do anything at all; therefore `int fsync(int) {return 0}` is valid because fsync()'s method is implementation-specific.
This allows you to have no software or hardware write cache and not implement fsync() and still be POSIX-compliant.
> Context matters, both in law and in programming. As a legal analogy, if you study Supreme Court rulings, you will find that in addition to examining the text of legislation or regulatory rules, the court frequently looks to legislative history, including Congressional findings and statements by regulators and legislators in order to figure out how to best interpret the law - especially when the text is ambiguous.
The POSIX specification is not a court of law, and the context is pretty clear: fsync() should do whatever it needs to do to request that pending writes are written to the storage device. In some valid cases, that could be nothing.
Sure, I'll give you that, in a corner case where all writes are synchronized to storage before completing. However, most modern computers cache writes for performance, and the speed/security tradeoff is the context of this discussion. We wouldn't be having this debate in the first place if computers and storage devices didn't cache writes.
> The POSIX specification is not a court of law
Indeed, it isn't; nor is legislative text (the closest analogy in law). Hence the need for interpretation.
> fsync() should do whatever it needs to do to request that pending writes are written to the storage device
We are in violent agreement about this :-)
And, if cheating will give better numbers on benchmarks, I’m willing to bet money most manufacturers will cheat.
> If _POSIX_SYNCHRONIZED_IO is not defined, the wording relies heavily on the conformance document to tell the user what can be expected from the system. It is explicitly intended that a null implementation is permitted.
Compare this to e.g. the wording for write(2):
> The write() function shall attempt to write nbyte bytes from the buffer pointed to by buf to the file associated with the open file descriptor, fildes. [yadadada]
This actually specifies that an action needs to be performed. fsync(2) sans SIO is merely a request form that the OS can respond to or not. And because macOS does not define SIO, you have to go out and find out what that particular implementation is actually doing and the answer is: essentially nothing for fsync.
But, the reality is that all operating systems provide some way to make writes to persistent storage complete, and to wait for them. All of them. It doesn't matter what POSIX says, or that it leaves crashes and power failure out of scope.
POSIX's model is not a get-out-of-jail-free card for actual operating systems.
An fsync that does not require the completion of an IO barrier before returning is inherently broken. This would be REQ_PREFLUSH inside Linux.
> fsync() might or might not actually cause data to be written where it is safe from a power failure.
The history is also interesting. It's not that "macOS cheats", but that it sincerely inherited the status quo of many years, then tried to go further by adding F_FULLFSYNC. However, Linux since got better, leaving macOS stuck in the past and everybody surprised. It's a big problem.
Here's Dominic Giampaolo from Apple discussing this back in 2005, before Linux fixed fsync() to flush past the disk cache: https://lists.apple.com/archives/darwin-dev/2005/Feb/msg0008...
And here's TigerBeetle's Twitter thread with more of the history and how projects like LevelDB, SQLite and various language std libs were also affected: https://twitter.com/TigerBeetleDB/status/1422854779009654785
> Note that F_FULLFSYNC represents a best-effort guarantee that iOS writes data to the disk, but data can still be lost in the case of sudden power loss.
[1] https://developer.apple.com/documentation/xcode/reducing-dis...
Note that the man page for `F_FULLSYNC` itself doesn't mention that it is not reliable: https://developer.apple.com/library/archive/documentation/Sy...
Having a separate syscall is annoying, but workable. Having a scenario where we call flush and cannot ensure that this is the case is BAD. Note that handling flush failures is expected, but all databases require that flushing successfully will make the data durable.
Without that, there are no way to ensure durable writes and you might get data loss or data corruption.
https://github.com/mysql/mysql-server/commit/3cb16e9c3879d17...
You didn't report the full reasoning.
At first glance, it seems to make sense - if someone shuts down while there is still uncommitted data because MySQL has tried a fsync(), it could leave the files on disk in a weird state when the power is cut. Am I missing something?
> Without that, there are no way to ensure durable writes and you might get data loss or data corruption.
No, not without that. Even with that, you can't have durable writes; Not on a mac, or linux or anywhere else, if you are worried about fsync()/fcntl+F_FULLSYNC because they do nothing to protect against hardware failure: The only thing that does is shipping the data someplace else (and depending on the criticality of the data, possibly quite far).
As soon as you have two database servers, you're in a much better shape, and many databases like to try and use fsync() as a barrier to that replication, but this is a waste of time because your chances of a single hardware failure remain the same -- the only thing that really matters is that 1/2 is smaller than 1/1.
So okay, maybe you're not trying to protect against all hardware failure, or even just the flash failure (it will fail when it fails! better to have two nvme boards than one!) but maybe just some failure -- like a power failure, but guess what: We just need to put a big beefy capacitor on the board, or a battery someplace to protect against that. We don't need to write the flash blocks and read them back before returning from fsync() to get reliability because that's not the failure you're trying to protect against.
What does fsync() actually protect against? Well, sometimes that battery fails, or that capacitor blows: The hardware needed to write data to a spinning platter of metal and rust used to have a lot more failure points than today's solid state, and in those days, maybe it made some sense to add a system call instead of adding more hardware, but modern systems aren't like that: It is almost always cheaper in the long run to just buy two than to try and squeeze a little more edge out of one, but maybe, if there's a case where fsync() helps today, it's a situation where that isn't true -- but even that is a long way from you need fsync() to have durable writes and avoid data loss or corruption.
"The sun might explode so nothing guarantees integrity", come on, get real. This is pointless nitpicking.
Of course fsync ensures durable writes on systems like Linux with drives that honor FUA. The reliability of the device and stack in question is implied in this and anybody who talks about data integrity understands that. This is how you can calculate and manage error rates of your system.
I think most people understand that there is a huge difference between the sun exploding and a single hardware failure.
If you really don't understand that, I have no idea what to say.
> Of course fsync ensures durable writes on systems like Linux with drives that honor FUA
No it does not. The drive can still fail after you write() and nobody will care how often you called fsync(). The only thing that can help is writing it more than once.
> No it does not. The drive can still fail after you write() and nobody will care how often you called fsync(). The only thing that can help is writing it more than once.
It does to anybody who actually understands these definitions. It is durable according to the design (i.e., UBER rates) of your system. That's what it means, that's always what it meant. If you really don't understand that, I have no idea what to say.
> The only thing that can help is writing it more than once.
This just shows a fundamental misunderstanding. You achieve a desired uncorrected error rate by looking at the risks and designing parts and redundancy and error correction appropriately. The reliability of one drive/system might be greater than two less reliable ones, so "writing it more than once" is not only not the only thing that can help, it doesn't necessarily achieve the required durability.
Unfortunately few databases besides maybe blockchains have been engineered with that in mind.
Unless a failure mode you are concerned about include being cut off from the internet, or your system isn't network connected in the first place, in which case maybe not eh?
Anyway surely the point is clear. "Durable" doesn't mean "durable according to the whims of some anonymous denizen of the other side of the internet who is imagining a scenario which is completely irrelevant to what I'm actually doing with my data".
It means that the data is flushed to what your system considers to be durable storage.
Also hardware failures and software bugs can exist. You can talk about durable storage without being some kind of cosmic-ray-denier or anti-backup cultist.
What's the difference between the sun exploding and a single machine failing?
I have no idea how to answer that. Maybe it's because many people have seen a single machine fail, but nobody has seen the sun explode? I guess I've never had a need to give it more thought than that.
> It does to anybody who actually understands these definitions. It is durable according to the design (i.e., UBER rates) of your system.
You are wrong about that: Nobody cares if something is "designed to be durable according to the definition in the design". That's just more weasel words. They care what are the risks, how you actually protect against them, and what it costs to do. That's it.
> You are wrong about that: Nobody cares if something is "designed to be durable according to the definition in the design".
No I'm not, that's what the word means and that's how it's used. That's how it's defined in operating systems, that's how it's defined by disk manufacturers, that's how it's used by people who write databases.
> That's just more weasel words.
No it's not, its the only sane definition because all hardware and software is different, and so is everybody's appetite for risk and cost. And you don't know what any of those things are in any situation.
> They care what are the risks, how you actually protect against them, and what it costs to do. That's it.
You seem to be arguing against yourself here. Lots of people (e.g., personal users) store a lot of their data on a single device for significant periods of time, because that's reasonably durable for their use.
One need not even assume no device failure, since that's the point of RAID: to make up for some not-insignificant device failure rate. We need only assume that not too many devices fail at the same time. A pretty reasonable assumption. One relied upon all over the world, across many data centers.
I believe drives that do have capacitors are aware of it and return immediately from fsync() without writing to flash. Thats the point of this API
Since neither Macs nor any other laptops have SSDs with capacitors, this point is kind of moot.
Laptop batteries are irrelevant - battery failure, freezin or cutting power to the curcuitbord by holding the off buttons are the failrue modes you have to protect against.
Yeah you can definitely write a transactional database without having to rely on knowing you've flushed data to disk. Not only can you, but you surely have to otherwise you risk data corruption e.g. when there's a power-cut mid-write.
The best the OS can do is to trust the device that the data was, indeed, written to durable storage. Unfortunately, many devices lie about that. If you do a `F_FULLSYNC`, you can say you did your best, but the data is out of your hands now.
Sure, that will be slow, but there is a way!
edit: sorry - not 'standards compliant' (whatever that is - does linux declare support for SIO?), but probably what you are looking for.
Here is Apple’s documentation:
https://devstreaming-cdn.apple.com/videos/wwdc/2019/419ef9ip...
F_BARRIERFSYNC: fsync() with a barrier
F_FULLFSYNC: Drive flush its cache to disk
This sounds like the Linux fsync() and Linux syncfs() respectively. What you say is that F_FULLFSYNC is the same as Linux fsync() and your performance numbers back that up. Unfortunately, you would only see a difference between Linux fsync() and Linux syncfs() if you have files being asynchronously written at the same time as the files that are subject to fsync()/syncfs(). fsync() would only touch the chosen files while syncfs() would touch both. If you did not have heavy background file writes and F_FULLSYNC really is equivalent to syncfs(), you would not be able to tell the difference in your tests.
That said, let’s look at how this actually works on Mac OS. Unfortunately, the apfs driver does not appear to be open source, but the HFS+ driver is. Here are the relevant pieces of code in HFS+:
https://github.com/apple-oss-distributions/hfs/blob/hfs-556....
https://github.com/apple-oss-distributions/hfs/blob/5e3008b6...
First, let me start with saying this merits a faceplam. The fsync() operation is operating at the level of the mount point, not the individual file. F_FULLSYNC and F_BARRIERFSYNC are different, but they both might as well be variants of the Linux syncfs().
For good measure, let us look at how this is done on the MacOS ZFS driver:
https://github.com/openzfsonosx/zfs/blob/master/module/zfs/z...
The file is properly synced independently of the mountpoint, such that other files being modified in the file are not immediately required to be written out to disk. That said, both F_FULLSYNC and F_BARRIERFSYNC on MacOS are mapped by the ZFS driver to the same function that implements fsync() on Linux:
https://github.com/openzfs/zfs/blob/master/module/os/linux/z...
For good measure, let us look at how syncfs() is implemented by ZFS on Linux:
https://github.com/openzfs/zfs/blob/master/module/os/linux/z...
It operates on the superblock, which is what MacOS’ HFS+ driver does.
From this, I can conclude:
Linux syncfs() == macOS F_FULLFSYNC on HFS+
Linux fsync() == macOS fsync()/F_FULLFSYNC/F_BARRIERFSYNC on ZFS
Also, MacOS F_BARRIERSYNC is a weakened Linux syncfs() and Apple’s documentation is very misleading (although maybe not technically wrong). POSIX does allow fsync to be implemented via syncfs (sync in POSIX, but I am saying syncfs from Linux to be less confusing). However, not issuing and waiting for the completion of an IO barrier on fsync is broken behavior like you claim.
I am not sure how MacOS APFS behaves. I imagine that additional testing that takes into account the nuances in semantics would be able to clarify that. If it behaves like HFS+, it is broken.
Edit: Upon further examination and comparing notes with the MacOS ZFS driver team lead, it seems that HFS+ is syncing more than requested when F_FULLFSYNC is used, but less than the entire filesystem. You are fine treating it as a Linux fsync. It is close enough.
It is still dumb that there's a definition of fsync() that does not sync :-/
It seems reckless to me to not do this when you're interacting with the filesystem using low-level APIs (i.e not via Swift/Obj-C).
Welcome to C APIs in general, and POSIX in particular.
Only if the standard where anything else is a "surprise" is 2022 Linux.
Many (all?) other unices and macOS itself since forever work like that. Including Linux itself in the past [1]
Maybe it not being added to OSes when drive caches came into the picture was arguably a bug, and Linux has been the first OS to fix it properly. macOS instead introduced new, non-buggy behavior, and left the buggy one behind :-)
You mean in the 1980s? Linux wasn’t used before this wasn’t a concern for sysadmins and DBAs. This concern has been raised for years - back in the PowerPC era the numbers were lower but you had the same arguments about whether Apple had made the right trade-offs, or Linux or Solaris, etc.
Given the extreme rarity of filesystem corruption being a problem these days, one might conclude that the engineers who made the assumption that batteries covered laptop users and anyone who cares about this will be using clustering / UPS were correct.
Also re Linux here's eg PostgreSQL 9.0 documentation saying ext4/zfs + scsi used the "SYNCHRONIZE CACHE" command with fsync even back then, and a equivalent SATA command being used by the storage stack with SATA-6 and later drives: https://www.postgresql.org/docs/9.0/wal-reliability.html
Apple doesn’t need to defend anything.
Do lawyers use Apple computers? Do they work on important documents relating to life and death?
Some people have literally been executed because developers couldn't do their job properly. People have been sent to jail for decades because developers fucked up in the british postmaster scandal.
Average people life in a dangerous world- work with documents about their financial wellbeing. They live in opressive countries where being gay is punishable by death. They drive 2 ton death machines. And now that we have put computers in places where life and limb depends on them, we are responsible for doing the job properly, that's why we get paid.
Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss?
I think I heard some enterprise motherboards/controllers/computers did just that, given the upside in normal operation.
But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown.
This does require a last-gasp indication from the PSU to the rest of the system, so if they don't have that, it's not something they could add in a firmware update.
That's unfortunate. My Mac Mini crashes every other night during sleep. I guess I'm going to have to shut it down to avoid any data corruption.
Edit: Here's the first line of the crash log (which I'm sending to Apple every time):
panic(cpu 3 caller 0xfffffe0023be8be0): [data.kalloc.16]:
element modified after free (off:0, val:0x0000000000000030, sz:16, ptr:0xfffffe2fffc9bb00)
I don't think that quite works for the purpose. What you'd want is a second signal that goes low as soon as possible after loss of AC power.
My reading here is that PWR_OK going low is an indication that the PSU has stopped providing good power, and the CPU must shut down immediately, or it might miscompute something due to low voltage. At this point you absolutely don't want to do any last-minute writing, you'd be risking corruption.
What you need here is an early warning signal that you can react to while the PSU is still coasting on the internal capacitors.
I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss.
Of course there is industrial hardware that can detect quicker than this but it would add a LOT of cost for arguably little gain, or something that could be solved in another manner.
Doubtful. 16ms is an awfully long time these days. There's no reason why you couldn't detect power loss much sooner, given a good input signal. The concept also gets used quite often, in the form of SSRs with zero crossing detection. Those are used for dimmers.
The reason is likely related to the awful waveforms produced by some UPSes and inverters:
https://www.christidis.info/images/blog/scope_20.png
Unlike a nice sine wave, those spend a good while hovering near zero volts, so the PSU has to be able to tolerate that. Detecting loss of power sooner in this case isn't a question of cost, it's a question of that you don't have a good signal to do the detection on to start with.
That wave chart was atrocious. I wonder if the extra load on the DC-side caps leads to them having lower life expectancy than the ones in a PSU attached to a proper power grid?
OTOH, if you have watchdog timeouts (I've seen this from bad drivers), those would certainly not give the kernel a chance to do that.
Does Apple implement the NVMe spec on their controller, i.e. do they indicate "Volatile Write Cache"?
Or just add a UPS?
This is one of those things that SCSI was much better at, SYNC CACHE had a range option which could be used to flush say particular files/database tables/objects/whatever to nonvolatile storage. Of course out of the box Linux (and most other OSs) don't track their page/buffer caches closely enough to pull this off, so that fsync(fileno) is closer to sync(). So, few storage systems implemented it properly anyway.
The choice of ignoring flushes vaguely makes sense if you assume the mac's SSD is in a laptop with a battery. In theory then the disk cache is non-volatile (and this assumption is made on various enterprise storage arrays with battery backup as well, although frequently its a controller setting). But i'm guessing someone just ignored the case of the mac mini without a battery.
But that would be awesome, especially with these ever growing cache capacities.
Indeed, I would very much like to know what on earth the ANS firmware is doing on flushes to make them so hideously slow. We do have the firmware blobs (for both the NVMe/ANS side and the downstream S5C NAND device controllers), so if someone is bored enough they could try to reverse engineer it... it also seems there's a bunch of debug mode options, so maybe we can even get some logs at some point.
On NVMes I wonder whether this really matters, but it's a serious issue on spinning disks: do you really need to flush everything to the disk (and interrupt more efficient access patterns)?
Btw, are you sure those spinning disks are actually flushing to rust? Caches all the way down... ;-)
That depends on the drive having power loss protection, which comes most of the time in the form of a capacitor that powers the drive long enough to guarantee that its buffers are flushed to persistent storage.
Consumer SSDs often do not have that, so flushing is really important there, at least if your data, or no FS corruption is important to you.
Enterprise SSDs almost always have power loss protection, so there it isn't required for consistency’s sake, albeit in-flight data that didn't hit the block device yet is naturally not protected by that, most FS handle that fine by default though.
Note that Linux, for example, does by default a periodic flush every 30s independent of caching/flush settings, so that's normally the upper limit you'd lose, depending on the workload it can be still a relatively long time frame.
But this isn't specific to Macs and iDevices. Some non-PLP drives also struggle with sync writes on FreeBSD [1]. Most enterprises running RDBMS mandate PLP for both performance and reliability. I understand why this is frustrating for porting Linux, but Apple is allowed to make strong assumptions about how their hardware interoperates.
[1] https://www.truenas.com/community/threads/slog-and-power-los...
You have wear leveling trying to keep things from blowing holes in certain physical pages. In certain cell architectures you can only write to pages that have previously been erased. Once you do write the data to the silicon... it's not really written anyway, because the tables and data structures that map that to the virtual table the host sees on boot also have to be written.
It is entirely reasonable that a system that does 100k honest sustained write I/O per second would come to its knees if you're insistent enough to actually want a full, real, power cycle proof, sync.
To do an actual full sync, where it could come back from power off... requires flushing all of those layers. Nothing is optimized to do that. I'm amazed that it can happen 40 times per second.
It's possible that you could speed this up a bit, but somewhere there's an actual non-wear leveled single page of data that tells the drive how to remap things to be useful... I strongly suspect writing that page frequently would eat the drive life up in somewhere between 0.1 and 20 million cycles. After that point, the drive would be toast.
I agree with the other thread that actually flushing is likely to be a very, very well guarded bit of info.
Curious, what's the real world risk of full OS level corruption and not just data loss?
But to be honest, if I end up really bricking a machine for science, that will be worth it for the information it gives us. Obviously I'm not trying to destroy my hardware, but I'm very grateful that I can afford it if it happens thanks to all the support I'm getting from folks for the project.
What usually happens is battery internal resistance is too high to sustain a given power load, so once load crosses a threshold the system goes into a spiral of doom increasing current as battery voltage decreases and you end up in a shutdown. That's the "30% and suddenly 0% or a shutdown" scenario. But if you catch it before it's too late, you can just stop consuming power and let the NVMe controller flush.
“30% to 0” and “Pull AC and it instantly dies” are typically a combination of load and device temperature. High CPU/GPU usage, high brightness, 3G/LTE usage, and cold temps and the device doesn’t have a chance.
It’s been somewhat fascinating to monitor power usage in this really crude way. TikTok on iOS, for example, uses so much power that it’s the most likely to cause the device to shut off. FB Messenger is in the top 5. Some of Apple’s background processes will also cause it, as will paging memory to disk.
There’s another bit of information that will not surprise many people on HN: high-amperage charging will cause the battery percentage to be “more wrong”. Devices will report 45% or higher and still die as if they were reporting 30%. Charging at 500mA will not only make it “more correct”, but will typically mean that a device will not suddenly die until it’s in the single digits.
This is still n=1 of course.
All of my Macs are either laptops or have a hardware backup device, so unlikely a write would be lost due to power failure (unless backup device failed which could happen).
Back when I still used a UPS down here, it was usually the UPS that died and triggered the power failure. So I stopped investing in a UPS.
I even lost a MBP to a light flickering event with 0 power loss. Fried the charging circuit straight through the original power brick.
Although, I also have the seemingly rare opinion here that ECC ram doesn't really matter on a laptop or desktop.
The most common situation where this would affect laptops, in my experience so far, would be a broken driver causing a kernel lockup (not a panic) which triggers a watchdog reboot. That situation wouldn't allow for an NVMe flush.
I mean I grew up diligently turning off my PC by parking the disk and using the various operating system level shutdown procedures. Nowadays I smack the off button, but that still just triggers the OS shutdown procedure. I don't turn my Mac off as a rule, its sleep mode actually works. ish.
REISUB for a somewhat safe EMERGENCY reboot and O instead of B at the end for shutdown.
At least re: this issue; it's still a bad idea because it's only safe if all software is written following data integrity and flush rules to the letter, and most software isn't. You're eventually going to run into issues on any OS by doing that, because most software doesn't get this right unless it's a database. And you're still going to lose data that's in buffer cache, I'm pretty sure that won't get flushed.
2) Why are you rebooting a laptop daily? My uptime on my MacBook Pro averages 30-60 days. There's zero reason to reboot any modern OS daily.
- I use Arch, I like to avoid accumulating too much major updates between reboots. - For a time I was facing a bug that resulted in a black screen of death after resuming sleep.
For a database this means that every transaction will take a minimum of 1 second, otherwise you can't guarantee durability.
Sure, you could write a program that periodically checks the battery rate (you'd have to poll since there's no ACPI notification like with a "device battery") and sends an email to the admin or something. However that's a tool that doesn't "exist" (as in, there isn't notable program that does so) which possibly hints that this isn't something system admins often do.
The above also requires there to be an interface available from userland, not only in the management firmware or BIOS/UEFI. That exists for HP, but I'm not sure all other OEMs do so.
1) Not everyone wants to use iLO or whatever equivalent another OEM provides.
2) Whilst such systems do support sending warnings about system components via email, dashboards, etc. that doesn't mean they'll necessarily warn about a RAID controller's battery being depleted. If I remember correctly, iLO4 doesn't.
3) What about RAID cards like the P420 (*not* the P420i) that either aren't hooked up to a management engine or are from an entirely separate OEM?
Then you aren't an enterprise because they're absurdly useful for managing dozens/hundreds/thousands of systems.
>3) What about RAID cards like the P420 (not the P420i) that either aren't hooked up to a management engine or are from an entirely separate OEM?
There's a reason enterprises standardize on a common infrastructure from an OEM that supports everything in the box even though you could go on Newegg and build your own systems for thousands of dollars less.
Getting all that right sounds so hard it is probably better to just have enterprise SSD's have a built in supercap to give 5 seconds or so of power to do all the necessary flushing, and for laptop/desktop grade SSD's they only need to offer barriers for data consistency. Laptop and desktop users don't care if they lose the last 1 second of data before a crash as long as what is on the drive is self consistent.
To be fair though, I sidetracked from the discussion at hand. The issue Marcan described was regarding the OS -> Disk rather than a "power loss situation". The latter does play in with the former, but solving the latter doesn't necessarily solve the former.
Apple may not implement such a durable cache, that's fine it's not an enterprise device and it's a cost tradeoff. So they might have to flush to NAND on any FUA, and that's slow as we've said, but not 25ms slow. Modern QLC NAND tPROG latency is more like 2.5ms-5ms, which could just about explain the EVO results when you include the OS and SATA stack and drive controller.
There's pretty close to 0% chance Apple would have messed this up accidentally though, in my opinion. It would have been a deliberate design choice for some reason. One possible reason that comes to mind is that some drives gang a bunch of chips in parallel end you end up with pretty big "logical" pages. Flushing a big logical page on a 4kB write is going to cause a lot of write amp and drive wear, so you might delay for a short period (20ms) to try pick up other writes and reduce your inefficiency.
Some of the basic NAND guides they put out are simple enough to understand the basics of operation
https://www.micron.com/-/media/client/global/documents/produ...
The details get very complicated and proprietary. NAND wears out as you use it. But it also has a retention time. It gradually loses charge and won't read back if you leave it unpowered for long enough. This is actually where enterprise drives can be speced worse than consumer. So durability / lifetime is specified as meeting specified uncorrected error rates at the given retention period. The physics of NAND are pretty interesting too and how it translates into how a controller optimizes these parameters. Temperature at various stages of operation and retention changes properties, time between erase and program does too. You can adjust voltages on read, program, erase, and those can help you read data out or change the profile of the data. Reading can disturb parts of other pages (similar to rowhammer). Multilevel cells are actually interesting some of them you program in passes so that's a whole other spanner in the works.
I don't know of a good place that covers all that, but much beyond "read/program/erase + wear + retention" is probably beyond "what every programmer should know".
The way you turn a bunch of NAND chips that have a "read/program/erase" programming model into something that has a read/write model (the flash translation layer or FTL) is a whole other thing again though. And all the endurance management and optimization, error correction... Pretty fascinating details really. The basic details though is that they use the same concepts as the "log structured filesystem", turns out a log structure with garbage collection is about a perfect it for turning the program/erase model into a random write model. That's probably what every programmer should know about that (assuming you know something about LSFs -- garbage collection, write amplification, forward and reverse mapping schemes, etc).
> There's pretty close to 0% chance Apple would have messed this up accidentally though, in my opinion
There's pretty close to 100% chance Apple would not have cared/optimized for this when designing this SSD controller, because it was designed for iOS devices which always have a battery, and where next to no software would be issuing flushes.
And then they put this hardware into desktops. Oops :-)
Lots of things about the M1 were rushed and have been fixed along the way. I wouldn't be in the least bit surprised if this were one more of them that gets fixed a couple macOS versions down the line, now that I've made some noise about it.
How are you measuring that and how do you figure it means the NAND writes are not being held off? Clearly they are by one means or another.
> The firmware is doing something dumb when issued a flush command, it's not just sitting around and waiting.
> There's pretty close to 100% chance Apple would not have cared/optimized for this when designing this SSD controller, because it was designed for iOS devices which always have a battery, and where next to no software would be issuing flushes.
Yes. It is clear the hardware was never optimized for it. Because it is so slow. I'm almost certain that is a deliberate choice, and delaying the update is a possible reason for that choice. It's pretty clear the hardware can run this much faster, because it does when it's streaming data out.
NAND and the controller and FTL just isn't rocket science that you'd have hardware that can sustain the rates that Apple's can and then through some crazy unforeseen problem this would suddenly go slow. Flushing data out of your cache into the log is the FTL's bread and butter. It doesn't suddenly become much more complicated when it's a synchronous flush rather than a capacity flush, it's the same hardware data and control paths, the same data structures in the FTL firmware and would use most of the same code paths even.
Pull blocks from the buffer in order and build pages, allocate pages in NAND to send them, update forward map, repeat.
powermetrics gives you DRAM bandwidth per SoC block, before and after the system level caches.
> how do you figure it means the NAND writes are not being held off? Clearly they are by one means or another.
I mean they're not just being held off. It's doing something, not waiting.
> Yes. It is clear the hardware was never optimized for it.
This is a firmware issue. The controller runs on firmware. I can even tell you where to get it and you can throw it in a decompiler and see if you can find the issue, if you're so inclined :-)
> I'm almost certain that is a deliberate choice, and delaying the update is a possible reason for that choice.
Delaying the update does not explain 10MB/s of memory traffic. That means it's doing something, not waiting.
> It's pretty clear the hardware can run this much faster, because it does when it's streaming data out.
Indeed, thus it's highly likely this is a dumb firmware bug, like the FLUSH implementation being really naive and nobody having cared until now because it wasn't a problem on devices where nothing flushes anyway.
> NAND and the controller and FTL just isn't rocket science that you'd have hardware that can sustain the rates that Apple's can and then through some crazy unforeseen problem this would suddenly go slow.
Yup, it's not rocket science, it's humans writing code. And humans write bad code. Apple engineers write bad code too, just take a look at some parts of XNU ;-)
> Flushing data out of your cache into the log is the FTL's bread and butter.
Full flushes are rare on devices where the cache can be considered persistent anyway because there's a battery and the kernel is set up to flush on panics/emergency situations (which it is). Thus nobody ever ran into the performance problem, thus it never got fixed.
> It doesn't suddenly become much more complicated when it's a synchronous flush rather than a capacity flush, it's the same hardware data and control paths, the same data structures in the FTL firmware and would use most of the same code paths even.
The dumbest cache implementation is a big fixed size hash table. That's easy to background flush incrementally on capacity, but then if you want to do a full flush you end up having to do a linear scan even if the cache is mostly empty. And Apple have big SSD caches - on the M1 Max the NVMe carveout is almost 1 gigabyte. Wouldn't surprise me at all if there is some pathological linear scan going on in the case of host flush requests, or some other data structure issue. Or just an outright bug, a cache locality issue, or any other number of things that can kill performance. It's code. Code has bugs and performance issues.
> Indeed, thus it's highly likely this is a dumb firmware bug, like the FLUSH implementation being really naive and nobody having cared until now because it wasn't a problem on devices where nothing flushes anyway. I don't think that's highly likely at all. I think it's highly unlikely.
> Yup, it's not rocket science, it's humans writing code. And humans write bad code. Apple engineers write bad code too, just take a look at some parts of XNU ;-)
I'm not some Apple apologist. I think their fsync() thing is stupid (although very surprised you didn't know about it and took you so long to check the man page, it's a old and well known issue and I don't even use or program for OSX). The hardware is clearly not very good for the task of a non-battery PC (even on batteries I think it's a questionable choice unless they can flush data in case of OS crash or low battery shutdown. I also think their kernel is low performing and a poor Frankenstein mishmash of useless microkernel bits. So you're not getting me on that one.
> Full flushes are rare on devices where the cache can be considered persistent anyway because there's a battery and the kernel is set up to flush on panics/emergency situations (which it is). Thus nobody ever ran into the performance problem, thus it never got fixed.
I never said the hardware was suitable for this type of operation.
> The dumbest cache implementation is a big fixed size hash table. That's easy to background flush incrementally on capacity, but then if you want to do a full flush you end up having to do a linear scan even if the cache is mostly empty.
I can think of dumber. A linked list you have to search.
This approach is really bad even if you don't have any syncs because you still want to place LBAs linearly even on NAND otherwise your read performance on large blocks
The fact you can come up with stupid thing that might explain it isn't a very good argument IMO. Sure that might be the case I didn't say it was impossible just didn't think it was likely. You're saying it's certainly the case. Don't think there's enough evidence, at best.
And if it was a strange forward map structure that takes a lot of time to flush but is fast or small or easy to implement, that actually supports my statement. That it was a deliberate design choice. Not a firmware bug. Gather delay was one example I gave, not an exhaustive list.
By that logic, every time a programmer uses an inefficient data structure and introduces pathological performance it's not a bug, it's a "deliberate design choice".
At this point we're arguing semantics. My point is it's slow when it shouldn't be, and it can be made faster. Whether it's a "bug" or not comes down to whether Apple fixes it or not. I consider it a bug in my book because you don't normally design things to be 10-100x slower than the competition. It's too blatant not to be an oversight.
And I've seen Apple make many oversights in the past year and fix them in an update. There is plenty of evidence the platform was rushed and lots of things were full of jank in early macOS 11 that got fixed on the way to 12, and more are still being fixed. This would be one of many and completely in line with history so far. It's why we're requiring 12.1+ firmware as a baseline for Asahi going forward, because many things were fixed and I don't want to deal with the buggy versions.
Not if it's efficient for it's primary use cases. Lots of data structures have pathological behavior including in the Linux kernel.
I disagree with this - my Apple is an enterprise device. It's a Macbook Pro, issued by my employer, to do real work. I wouldn't give Apple a pass on this dimension. I get that the "Pro" label doesn't mean what it used to, but these aren't toys either.
In short - no, you'll still see corruption.
ZFS (and btrfs) is not "journaled", it's copy-on-write.
You will not see any data corruption on ZFS as long as the underlying hardware implements REQ_PREFLUSH correctly and the software uses proper POSIX semantics. If no filesystem corruption is happening, then the stuff under ZFS is doing its job correctly and your problem is in userspace.
Following a crash, ZFS returns to a past good state. Any completed synchronous writes or writes protected by a completed fsync will be there. Any of those that did not complete can be expected to be lost (unless it was moments away from returning to userspace) and any non-synchronous IO that occurred in the last several seconds is allowed to disappear.
By default, non-sync IO is flushed to permanent storage every 5 seconds. A past good state is not something that I would call corruption and software is expected to be able to resume from the past under POSIX.
In any case, ZFS should be fine as long as REQ_PREFLUSH is working properly. You can read a little about that here:
https://github.com/openzfs/zfs/blob/453c63e9b74cea42d45e0bd3...
https://elixir.bootlin.com/linux/v4.18/source/include/linux/...
"Linux Fsync Issue for Buffered IO and Its Preliminary Fix for PostgreSQL"
https://news.ycombinator.com/item?id=19238121
"PostgreSQL used fsync incorrectly for 20 years (2019) [video] (fosdem.org)"
As such, they share the same storage infrastructure as other EC2 instances.
I'm pretty sure, whatever it is, Apple could fix it in a firmware update.
Can someone explain what "flushing write cache to stable storage" means? Isn't that the same as "writes to the drive". I am obviously not well versed in this area. Also what is stable storage? Never heard that term before.
Some storage tech is just slow at that, and manufacturers muddy the water by rating some (SSDs|Micro SDs|whatever) in GB/s overall when much of those big numbers are a combination of caches and trickery.
I would not be surprised if Apple is using a tech that just has slow write speeds in trade for fast read speeds since most Apple users will be happy with faster read speeds.
This was for enterprise SSD though a few years back.
I would think when the last of Apple’s hardware moves to ARM they’ll ensure there’s enough onboard battery to ensure the flushes happen reliably across form factors even if there’s a power cut.
If anything, now that the reason for the performance difference has been identified, I’d hope to see numbers for Linux and Windows storage access come up to par with these numbers as they go down this road too (e.g. via the NVME flush toggle mentioned in the article).
Trading correctness for performance without shouting at the users "YOUR DATA IS NOT SAFE WHEN YOU DO THIS" multiple times a day in a storage is benchmark-snake-oil. Period.
So there's that.
It's rather Apple's slow drive firmware checks though, that is problematic.
My point above was that the same “cheat” (to use your word) could be applied to the unbranded SSD too, with similar performance gains.
I’m not giving Apple a pass for low flush performance, I’m saying there’s nothing I can see here that’s uniquely available to Apple that would prevent others from deferring flushes in the same way for similar performance gains - which would make sense in many cases.
That's not the reason for the M1 performance differences (as a CPU).
Just for the disk writing (which isn't the fastest around to begin with anyway).
They just direct people to use Time Machine or iCloud, then look quizzically at you when have issue with writing off lost hours of work as cost of doing business.
“You can lose some of your file changes in case of hard-reboot” is more correct.
It was a given truth for me all the time and I can tolerate some data losses if power was accidentally turned off for my desktop, or if OS panicked (it happens ~ once per year to me).
If this is a price for a 1000x speed increase - I’m more than happy they have implemented it this way.
That's a problem. It means e.g. transactional databases (which cannot afford to lose data like that) have a huge performance hit on these machines, since they have to use F_FULLFSYNC. And since that "no really, save my data" feature is not the standard fsync(), it means any portable software compiled for Linux will be safe, but will be unsafe on macOS, by default. That is a significant gotcha.
The question is why do other NVMe manufacturers not have such a performance penalty? 10x is fine; 1000x is not. This is something Apple should fix. It's a firmware problem.
The point here is that on the apple systems if you do the correct thing your performance drops to that of spinning disks.
It's ridiculous to think that in case of power loss you expect 100% data integrity - it might happen in the middle of the command execution. If the system should be unkillable, it should have an unkillable power source in the first place.
A properly designed transactional database will only ever "fail ahead". If power fails a transaction that was in the process of committing might commit without an ack, but will never return an ack and then be lost on the next startup. The ack means the data is safe, regardless of what happened afterwards.
Nobody expects 100% data integrity on power of. What is expected is that data that what was fsynced has 100% data integrity once that system call returns. This information is also used when moving files across the network, the file gets deleted on the sender when the receiver said fsync is completed. This means you could loose entire files of data when moving things over the network onto a Mac
#!/usr/bin/python
import os, sys, time, datetime
t = datetime.datetime.now().isoformat()
print(t)
for i in range(5):
time.sleep(1)
print(4 - i)
fd = os.open(sys.argv[1], os.O_RDWR|os.O_CREAT)
os.lseek(fd, 0, 0)
os.write(fd, b"test: " + t.encode("ascii") + b"\n");
os.fsync(fd)
print("done!")
time.sleep(100)
This isn't some hypothetical thing, this is a trivial test you can do. fsync() on macOS does not guarantee data is on stable storage. And this is actually well documented.
Then if you want to see the performance problem, make it a loop instead and use `fcntl.fcntl(fd, fcntl.F_FULLFSYNC, 1)`. You'll get 40-odd IOPS, but at least your data won't disappear after power loss.
marcan@raider:~/tmp -$ echo "very important data" > file.txt
marcan@raider:~/tmp -$ rsync --remove-source-files file.txt macmini.lan:
marcan@raider:~/tmp -$ ls file.txt
ls: cannot access 'file.txt': No such file or directory
marcan@raider:~/tmp 2$ ssh macmini.lan
Last login: Thu Feb 17 23:45:26 2022 from 192.168.3.10
marcan@Mini-M1-2020 ~ % ls file.txt
ls: file.txt: No such file or directory
This is real, please stop pretending it isn't.
I had no doubts that you can lose your file if you are moving it and some very lucky power outage hits. I was not “pretending” it's not real.
What I have doubts about, is that it's a real concern for 99.93% of the users. As we’ve found here, it's not even a rare case in other kinds of OS, so users would definitely notice it.
It is theoretically possible, of course. In practice, it's just too rare to consider.
But still, I hope this topic will be noticed by Apple and they fix that low performance issue for fullsync. Also, I hope they will not make fullsync as the default behavior - it doesn't worth the risks (and those for whom it does - should use some Linux for sure).
> No, it’s not a problem, it is expected. If you are running a transactional database on your desktop - at least add a UPS to your system.
This misses the point--PostgreSQL on my machine is either lossy or slow. A UPS (or battery, in my case) doesn't fix that.
> But still, I hope this topic will be noticed by Apple and they fix that low performance issue for fullsync
It's silly, but you can :-)
You shouldn't have to add a secondary UPS at all, period, and still get that.
Databases are designed that way (for integrity under sudden power loss) - the OS just needs to provide a standard call for the sync that they can use.
Now, fsync not guaranteeing a write is one thing -- and it's common in other OSes, even Linux behaved like that.
The non-commital fullsync on the other hand (and the slow speed) are problematic, and that's not an excuse for the user having such a bizarro case as wanting to run a DB on their Mac Mini without 2 UPS, that's you excusing Apple.
Not to mention that 2 UPS wont solve the problem if you're not there to shut down the computer gracefully as they, themselves, are depleted (e.g. at night) when there's a powerloss.
For any mission critical stuff, I have it behind a UPS.
The last time I saw a modern filesystem eat itself on sudden power loss was when I was evaluating btrfs in a datacenter setting, and that absolutely told me it was not a reliable FS and we went with something else. I've never seen it happen with ext4 or XFS (configured properly) in over a decade, assuming the underlying storage is well-behaved.
OTOH, I've seen cases of e.g. data in files being replaced by zeroes and applications crashing due to that (it's pretty common that zsh complains about .zsh_history being corrupted after a crash due to a trailing block of zeroes). This happens when filesystems are mounted with metadata journaling but no data journaling. If you use data journaling (or a filesystem designed to inherently avoid this, e.g. COW cases), that situation can't happen either. Most databases would be designed to gracefully handle this kind of situation without requiring systemwide data journaling though. That's a tradeoff that is available to the user depending on their specific use case and whether the applications are designed with that in mind or not.
UPSes can and do fail.
Why this never happened to me? Why I don't know anyone which had this problem? Why nobody is complaining as it happened with the previous gen keyboards?
I think we might be missing something in this analysis. I don't think Apple engineers are idiots.
I've seen APFS filesystems eat themselves in production (and had to do data recovery), twice. Apple don't have a perfect data integrity track record.