Comment by VogonPoetry | Hacker News Reader

Comment by VogonPoetry | Hacker News Reader

Parent Full thread

Opinion. The bizarro thing with this is that O_<xzy> flags are for a per file open, but the guarantees can only be supported by an underlying filesystem.

On Linux you can mount a filesystem and invalidate some metadata updates using options such as noatime. This is also depends on the mounted filesystem, such as FAT, which might not even support the updated attributes. Are these constraints honored in these environments with these flags? (sorry, too lazy to write code and check this)

There is no data on what is actually "broken" or why it might violate POSIX.1-2008. What metadata is actually important or impactful?

I am very, very skeptical of claims made by some database folks, due to past experience. Some Oracle products, and from what I understand, still do not understand that monotonically increasing means greater than or equal to (>=). The equal to is the problem, because calling an absolute value timestamp function to derive a "unique" value will always fail (with the same return value) when you get faster hardware and more cores. In a past life (~2000), I had to add code to bork an operating system to appease this very misguided notion. Last month (May 2022) I got (2nd hand) info that after a hardware update to faster hardware, duplicate unique row_ids were being created in an Oracle DB.

Without additional details, I find it hard to understand what the issue actually is. Is this a detectable temporal violation of order of operations for a live system, or some sort of fail situation where an abort / reboot occurs -- would love to understand how the external logging of this was carried out, because how was that synchronized / timestamped?

Details and verification are in the quoted thread (you need to click through).

TLDR: https://twitter.com/jorandirkgreef/status/153231416960472678...

You can see that, for those block sizes, the throughput of O_DSYNC is the same as F_NOCACHE, e.g. 3121 MiB/s vs 52.05 MiB/s for libuv's durable fdatasync() which is really actually using fcntl(F_FULLFSYNC) under the hood here.

In other words, O_DSYNC is only going as far as the disk's own internal cache. It's giving nearly the same performance as a buffered write.

So the reason it is “completely broken” and not simply “broken” then, is that setting the O_DSYNC bit provides no durability at all. It's a no-op.

Also, to be fair, if you're familiar with the space, it's been known for some time that Apple's fsync() is not durable—it's not like this is coming out of left field. What's new here, is that O_SYNC/O_DSYNC have the same issue. That's why we're awarding the bounty.

What you appear to be saying here is that the "test" is based solely on comparing write performance, not on the conformance to what was written, when it was written and what was written when power was pulled and what was read back later. Is that a fair assessment?

This test seems to be based on how "understood" technology works and an expected outcome. Any technology that works better or differently than what you expect will fail.

Intel has already proposed MRAM -- aka persistent memory. If I am understanding correctly, your tests would claim that MRAM backed disks would fail your tests, they would perform way too fast.

These privatives seem to be super specialized and very dependent on what actually happens. The system call you reference, fsync() was introduced in BSD 4.2 -- 1983. There were no journaling filesystems in 1983. I think it is entirely reasonable to apply ~40 years of filesystem research to invalidate past system call practice and call it out as voodoo / blog fodder without real measurements on current filesystems and hardware and to also document if it fails or succeeds when the plug is pulled.

I acknowledge that doing this is hard work and time consuming. You are not going to get a 10 minute tweet from this, but really, isn't that the point of a robust filesystem?

No, I don't believe that the phrase “based solely” would be a fair assessment.

That there's a $1,024 bounty being awarded should be reason enough to suggest more than “10 minutes” went into this. While the issue was first reported in a tweet, you should know that it came to us from an experienced database engineer who worked on both Google Spanner and FoundationDB.

The initial triage was to compare O_DSYNC with F_FULLFSYNC on the same device, because the results should be within the same order of magnitude, relative to the same device—we already knew that APFS doesn't make the same distinction between O_SYNC vs O_DSYNC that Linux does.

"Intel has already proposed MRAM -- aka persistent memory. If I am understanding correctly, your tests would claim that MRAM backed disks would fail your tests, they would perform way too fast."

Again, I think you're missing that O_DSYNC was compared relative to the same device, with Darwin's custom F_FULLFSYNC as baseline.

Finally, to assume good faith, you can also imagine that once we had the triage in hand, the issue would have been confirmed independently as “correct” by a third party in the best position to make that assessment.

Reply on news.ycombinator.com