Why `fsync()`: Losing unsynced data on a single node leads to global data loss

Using fsync() to flush data to disk is, generally speaking, the wrong way to do things. fsync() is like using a piece of wood to hammer a nail into the ground. It'll work under very limited circumstances, but will fail horribly to do the heavy lifting required in other cases. fsync()'s semantics are complex and poorly defined, doubly so if multiple threads or processes are involved. Who gets the i/o error for a failed write? Nobody knows! fsync() is probably one of the worst APIs warts in Linux/Unix.

Developers competent in matters of storage generally use other methods / APIs like open()ing a file with O_SYNC and then checking the results of write() or pwrite() which will clearly identify when data is or is not successfully written to disk. If you want more performance, use async writes and it'll perform just fine. Seeing fsync() in code is a sign that the code has a bad smell and probably needs to be thoroughly reviewed.

> One thing that comes to mind when reading things like this is 'Distributed journaling for distributed systems', but since nobody seems to be doing that, it's probably a silly idea

That's exactly what Redpanda, BookKeeper, LogDevice, Pravega, and countless other proprietary systems at big tech companies do.

> Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer without an NDA with the disk manufacturer and a hefty dose of proprietary IOCTLs (and even then, I would not bet my life on it).

The answer is easy in all cases: No.

Media can always fail without notice.

There is no way to know for sure that a Byzantine system is actually operating in a way that replicated copies of your data are actually safely written. Both due to the exact same issue of the drives themselves, and also that Byzantine system software is liable to have a variety of bugs and invalid states that will keep it operating as normal, even though the nodes are actually in a fault mode. (the problem is twofold: 1) the node and/or system not refusing writes when in a fault state, 2) the system not actually knowing that it's in a fault state) Even if you do all kinds of Jepsen simulation and mathematical proofs of the software (including the operating system!), you still can't trust the drives.

I think the only way to solve the problem is new storage firmware and hardware that is open and guarantees a write is done. I'm sure some companies may claim such functionality but we need an open source architecture and code to be sure.

In the meantime I think synchronous writes to multiple nodes is the safest option. Avoids complexity and bugs in fancy software, and the hardware is what it is.

Right. But that line of thinking gets you to a place where one is like “how does anything work” haha.

In general this was a response to Confluent attempting to dismiss fsync() as a neat trick rather than an actual safety problem and why when we benchmarked we showcase the numbers we did.

> you need to be absolutely, positively, 100% sure that `fsync()` has done The Right Thing would be: "Good luck with that!"

But it's safe to assume the right thing is not done if we haven't even called `fsync` to start with and just hope and pray that the dirty page flushing mechanism does it for us.

" the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer"

Well, kind of. fsync()'s job is only to ensure caches are written to durable storage. Its job is not to ensure the integrity of durable storage. Your idea of fsync()'s "Right Thing" is not quite correct, because these types of durable storage failures can happen outside of the context of a write -- so it's a bit silly to point a finger at fsync().

For example, you can write bytes to disk today and the drive might experience corruption on that sector next week.

"(and even then, I would not bet my life on it)"

All hardware fails, so of course you shouldn't bet your life on it. That's why we have both onsite and offsite backups. The durable media itself can always fail and lose data -- even if there aren't any writes occurring. Data loss can even occur on a powered off system.

Again, fsync()'s job is only to ensure the data is moved to durable storage. Solving the problem of data loss is done in other ways -- for example raid5 addresses the problem of durable storage failing by using parity and extra copies.

The real takeaway from the blog is perhaps that Raft as a protocol is inadequate, in the same way a raid1 mirror system is inadequate in terms of protecting from data corruption on durable storage (which of the two mirrored drives has the correct version of the data?). I'm frankly surprised that this situation is undetectable. It's a solved problem in other storage layers.

"I'll probably stick to SQLite write-ahead logs, at least I can wrap my mind around those"

Well, keep in mind: those have the same problem in the event of durable storage failure.

> Even in a relatively simple modern storage stack, say a single NVMe Flash device on a local PCIe controller, the question 'are these exact bits irrevocably committed to the media?' is pretty much impossible to answer without an NDA with the disk manufacturer and a hefty dose of proprietary IOCTLs (and even then, I would not bet my life on it).

This would limit throughput quite a bit and might significantly shorten the lifespan of the disk, but you could put the disk on a separate power supply that the computer could control. Don't consider data as written until you've power cycled the disk and been able to read that data back. :-)

I don't know what consistency Kafka is targeting but usually with 'consistent' systems in the situation where the cluster cannot serve valid data it should serve failure responses instead of giving back bogus data. if your system is 'eventually consistent' then that's a different game.

Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer.

Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie.

Kafka as it's described in the post(): any number of nodes may be down, at most one power outage is allowed (loss of unsynced data), as soon as the majority is available a replicated system is available and doesn't lie.

The counter-example simulates a single power outage

() https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

Yep, I still consider them to be cutting edge. Paxos was written in 1990 but the industry adopted it only in 2010s. For example I've looked through pBFT and it doesn't mention reconfiguration protocol which is essential for industry use. I've found one from 2012 so it should be getting ripe by now.

the blog posts mentions that it is for any protocol that is non bft

> Protocol Aware Recovery is really needed if you want Raft to tolerate disk corruption

I believe OP does not make any claims about arbitrary log corruption. Neither raft nor Kafka protocol can handle it. It is about losing tail of the log due to fsync failures or rather lack of fsyncs.