Hacker News Reader

No. Redpanda has made a lot of noise about this over the years [0], and Confluent's Jack Vanlightly has responded in a fair bit of detail [1].

[0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s...

[1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

View on HN

I think all modern system even scylla db do commit batch no fsync on every write, you either need throughput or durability both cannot exist together. Only thing what redpanda claim is you have to do replication before fsync so your data is not lost if the written node is dead due to a power failure. this is how scylla and cassandra works, if iam not wrong, so even if a node dead before the batch fsync, replication will be done before fsync from memtable,so other nodes will bring the durability and data loss is no longer true in a replicated setup. single node? obviously 100% data loss. but this is the trade off for a high tps system vs durable single ndoe system brings. its how you want to operate.

Similarly in regular SQL systems, the same is achieved by fsyncing to WAL.

Reply on news.ycombinator.com