Hacker News Reader

Comment by mehrdadn | Hacker News Reader

> Any dev who actually RTFM'd and followed POSIX specs would have been fine.

So I'm trying to do that [1] and it seems to me the documentation directly implies that a second successful call to fsync() necessarily implies that all data was transferred, even if the previous call to fsync() had failed.

I say this because the sentence says "all data for the open file descriptor" is to be transferred, not merely "all data written since the previous fsync to this file descriptor". It follows that any data not transferred in the previous call due to an error ("outstanding I/O operations are not guaranteed to have been completed") must now be transferred if this call is successful.

What am I missing?

[1] https://pubs.opengroup.org/onlinepubs/009695399/functions/fs...

View on HN

Technically, there is nothing besides "all data written since the previous fsync" since the previous fsync already wrote all the outstanding data at that point in time. (Even if it actually failed.) I.e., there is never any to-be-written data remaining after fsync returns. Everything that was queued to be flushed was flushed and dequeued. Whether any particular write failed or not doesn't change that fact.

> the previous fsync already wrote all the outstanding data at that point in time. (Even if it actually failed.)

I'm sorry, but this... just doesn't make sense. An event can't both happen and also fail to happen.

Also, the queue business is an implementation detail. Our debate is over what the spec mandates, not how a particular implementation behaves.

> I'm sorry, but this... just doesn't make sense. An event can't both happen and also fail to happen.

Consider a multithreaded application where a and b point to the same file. Note this isn't exactly what Postgres does:

    T1                   T2
    -------------------- --------------------
    write(a,x) start
    write(a,x) success

    fsync(a) start
                         read(b,x) start
                         read(b,x) success
    fsync(a) failed
                         write(b,y) start
                         write(b,y) success

                         fsync(b) start
                         fsync(b) success

y has a data dependancy on x (for example: table metadata), and the fsync() for y succeeded, so is T2 given to expect that x was recorded on disk correctly?

No? It says "all data for the open file descriptor", not "all data for all open file descriptors".

The file descriptors point to the same file and can even be the same descriptor.

Maybe this is clearer/simpler, writing x and y to the same offset:

    T1                   T2
    -------------------- --------------------
                         write(b,y) start
                         write(b,y) success
    write(a,x) start
    write(a,x) success

    fsync(a) start
    fsync(a) failed
                         fsync(b) start
                         fsync(b) success

At this point, T2 is probably surprised to discover the original contents of the file on the disk. Even after a successful fsync()!

I'm confused, where is the reading happening that is causing T2 to be surprised? But in any case, I'm saying that if a and b are the same descriptor, then fsync(b)'s success should imply both a and b are written to disk, by the specification of fsync. I don't see where you're seeing a contradiction.

I understand what you're saying. If a and b are the same descriptor, then fsync(b)'s success DOES NOT mean that it was written to because the error was already reported by fsync(a)'s failure report. I think you're missing that there are two fsync() calls, one that fails and the other that doesn't.

And FYI: Even if they're not the same descriptor, you still have risk.

You're telling me what's actually happening, but that's not where we're disagreeing. I agree that's what happens. What I'm saying is that it's wrong. It goes against the spec of fsync. It all data is to be written, not merely all data written since the last call to fsync. Your example didn't show any contradiction in this reasoning.

> It [says] all data is to be written, not merely all data written since the last call to fsync.

No it doesn't.

The spec† says: If the fsync() function fails, outstanding I/O operations are not guaranteed to have been completed. That clearly permits the Linux behaviour; There is no "data" that hasn't been sent to the "transferred to the storage device" at this point. You can see it from the perspective of the Linux kernel:

    write() = success
    fsync() = error: outstanding I/O not guaranteed††.
    fsync() = success: no outstanding I/O

†: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs...

††: It would've been a different story if the spec said, If "the fsync() function fails, all future fsync() to the same file descriptor must fail." but it doesn't say this.

No, the spec is longer than that one sentence. That sentence of the spec is merely saying that if fsync fails then the I/O operations haven't necessarily been completed. That's completely true and obvious and yes, it's also consistent with Linux. That's not the problem. The problem is the other sentence I already quoted which is that "all data for the open file descriptor is to be transferred to the storage device". This includes any data that hasn't been written, which includes data that has failed to write on a previous call to fsync. Like I already said once, you can't claim data that was sent to the device not failed to write to persistent storage is considered "transferred" since by that logic fsync would never guarantee any data was actually written to persistent storage (read the entire spec). I already explained all this but I'm repeating myself since it seems like every time you respond you're only choosing to read half of the spec and half of what I'm writing, but I'm tired of continuing.

But the "spec" uses that line, so you can't very well ignore it either. Most unixes also have this behaviour, so this isn't some nonconformant outlier, this is the standard behaviour. It might not be ideal, but most of the UNIX IO model and heritage isn't.

Apologies for assuming you didn't understand the behaviour when you said it "doesn't make sense": I'd assumed you meant that you were having trouble understanding the behaviour, rather than "all the application developers and unix implementors" that expect this behaviour, not understanding "the spec".

  The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to
  the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined.
  The fsync() function shall not return until the system has completed that action or until an error is detected.

All of the data was transferred to the device. The device may have failed to persist some or all of what was transferred, but it all got transferred.

There's no mention of the OS retrying, or leaving the system in a state that a subsequent fsync can retry from where it left off. So you can't assume anything along those lines.

So you're suggesting "storage device" here isn't the same thing as "storage medium"? i.e. that merely transferring the data to a volatile buffer on the device is also considered being transferred to the "storage device", rather than to the (nonvolatile) storage medium?

By that logic even after a successful call you can't rely on the data being persisted, which goes squarely against the entire point of the function and its documentation: "The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk."

So this interpretation is wrong...

Nonsense. If the device returns "success" then everything was persisted. If it returns an error, then some/all of it was not persisted. There is no way for you to determine whether it is some or all or which. The only safe action for a user is to assume all of it failed.

And by the way, there are devices out there that lie, and claim the data is successfully persisted even though it only made it into a volatile cache.

Everything you said in this comment is consistent with what I've been saying. Nowhere did I suggest the user can assume data was written if an fsync call fails. I'm saying if an fsync call fails, the documentation of fsync (which I quoted multiple times) implies the next call will attempt to write data that wasn't successful it written on the last call.

Yeah, and we still disagree there. Nothing in the doc implies that just calling fsync() again will do anything useful, there is no implication of retryability.

> Nothing in the doc implies that just calling fsync() again will do anything useful

"Nothing" in the doc? More like everything in the doc? It literally says "all data for the open file descriptor is to be transferred" and "If the fsync() function fails, outstanding I/O operations are not guaranteed to have been completed." Together these are literally saying that all data that wasn't transferred due to a failure in one call will be transferred in the next successful call.

I could maybe buy it if the doc described a notion of "dequeueing" that was separate from writing, but it doesn't. It just talks about queued operations completing. So either they complete successfully (and are subsequently dequeued because that is common sense) or they don't.

Like if your boss had assigned you tasks A, B, and then C, and then he ordered you to finish all your tasks, and you failed to do B, and then he made the same request again, you wouldn't then suddenly turn to him and say "I have nothing to do because I already popped B and C off my to-do list". You'd get fired for that (at least if you persisted) because it literally wouldn't make sense.

You're going in circles.

https://news.ycombinator.com/item?id=19133714

You are equating "transferred" with "completed" but they are clearly not the same.

All the data was transferred. After transfer, the data on the device may have been written or may have been lost. The OS doesn't remember what has been transferred after the transfer - there is no language about this anywhere in this text.

Reply on news.ycombinator.com