PostgreSQL used fsync incorrectly for 20 years (2019) [video]

It's very difficult to read this accusation as having been made in good faith, since the person you're accusing of intentionally omitting part of a quote already quoted that very text in their previous comment:

>>>> The tl;dr (and blunt truth) is that Linux doesn't implement POSIX's semantics of fsync() correctly. The standard says "the fsync() function is intended to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk".

No, that was not my intention. I just started the sentence "the job of fsync() is" and then went and looked for the portion of the quote that fit. I don't see a semantic difference, but I edited that back in, since you think I'm pulling a fast one on you.

> You're quoting a sentence fragment of a non-normative section of the spec, in other words you're half quoting a part of the spec that is not intended to be interpreted formally, literally, or even required to be adhered to. The rationale section of the POSIX spec is strictly informative.

Would be nice if you could include which part of the standard claims the rationale is non-normative. Moreover, the justifications against it are themselves based on "history" rather than anything in the text, so even if we assume it's non-normative, you can't really use this as an excuse to dismiss it as if it's somehow less valid than arbitrary tidbits from history. At that point you could try to agree that the (purportedly) non-normative text in fact does contradict the current behavior, but argue that you should have the latitude to ignore it. That's quite a bit different from denying its meaning in the first place.

> "The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined."

The "nature of the transfer" is implementation-defined, yes. They don't care if you do it all at once, in parallel or sequentially, in bursts or uniformly, by boat or by rocket, or whatever. The question of which data is to be transferred is completely unrelated to that. If I say "please take my family to the airport, the nature of the transportation is up to you", you don't get to leave half my family behind and then justify it by claiming I left the choice of family members up to you. I did not leave that up to you. I just said you can choose the nature of the transport. Not the whether or whom of the transport!

> It would be unwise to dismiss the wealth of resources that have been provided to you in other comments to better understand this issue, all of them quite extensive and written by experts on this subject, just because of an informal section of the standard

If experts claimed the standard was clear but perhaps inconsistent or self-contradictory, that would be very different than what's going on right now. Or if they argued "we agree this behavior is actually inconsistent with the rationale, but we don't believe the rationale is normative", then that would also be a different scenario. But when people are just saying the standard says something other than what it actually says, what am I supposed to do? Agree with them because they're "experts", despite the fact that the standard says the opposite in plain English?

In practice the intentional vagueness is indistinguishable from obfuscation, and doing so to please every vendor for conformance's sake is straightforward intellectual dishonesty. In this regard, the informational section is tantamount to an extended reprimand from the editors.

If you really want to trade language-lawyer broadsides, then leaning on the phrase "implementation-defined" was a tragic mistake, because this makes the Linux documentation normative for the circumstances, yet the behaviour occurring is not described in the Linux fsync(2) manpage, nor are there cross-references to anything FS-specific. It gets worse: the engineering solution to this schmozzle is switch to direct IO, but the documentation of Linux's open(2) then describes the behaviour of O_SYNC|O_DIRECT in terms of fsync(2) (and similarly for O_DSYNC). All this simply highlights that the whole line of thinking is an unproductive pissing contest; the root cause of the postgres issue is vendor politics.

Does "nature of the transfer" cover "the extent of the data set"? To me, the nature of the transfer refers to the technical implementation of the writeout mechanism. Especially given that the rationale explicitly mentions "since the concepts of "buffer cache", "system crash", "physical write", and "non-volatile storage" are not defined here, the wording has to be more abstract" it seems to me that "nature of the transfer" should not be interpreted as "the data to be written out".

But even going by the formal Description only, it does not follow to me that an implementation can choose to selectively ignore part of the data. These are the phrases used in that section:

- The fsync() function shall request that all data for the open file descriptor [..] to be transferred to the storage device

- [It] shall force all currently queued I/O operations [..] to the synchronized I/O completion state

- All I/O operations shall be completed

Each sentence contains the word "all".

The return value description reads "Upon successful completion, fsync() shall return 0". Based on this thread, the Linux kernel should never return 0 if a previous fsync failed, because the kernel has no intention of honouring the request.

As far as I understand, the answer is "definitely not" on Linux before 4.13, since failed async writes to the file could clear the dirty bit long before you called fsync(). There are some improvements since then, so arguably fsync() can now be used for something meaningful, though you have to take great care, and the only recourse on error is probably to start rewriting the file from scratch, even for transient errors (since the buffers may or may not be lost after the error, but future calls to fsync() will report success regardless).

Whether this behavior is a good idea or not is the only question that matters, since POSIX compliance hasn't mattered to Linux anyway for who knows how many years.