Hacker News Reader

Comment by rocky1138 | Hacker News Reader

Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?

View on HN

Which source code? There's more than one implementation of all of the following: OS kernel, disk driver, and filesystem.

It only takes one combination to start the chain reaction. If someone identifies one combination of kernel, disk driver, file system, hardware, and syscalls that results in reliable durability then people who care about reliable durability will start using it, and that will eventually turn into a de facto standard which will eventually turn into an actual standard.

Until someone who doesn't care changes some code and that pattern is no longer durable.

All of it - these things aren't straightforward, you need to go knee-deep in this area of programming to have a good hnderstanding of it.

The problem isn't just the mainline source code, you also have drivers which of course vary for each device.

Then even completely beyond control of the kernel, devices have their own RAM caches, which means that even if the device reports that something was written, it might not have been. So there's no way to absolutely be sure.

Event assuming that you can look at the source code for your filesystem/kernel the results of a given write still depend on the conditions and orders under which your writes hit disk relative to other processes' writes.

For example, if two processes issue writes to disk sectors that are adjacent but one of the processes' writes also affects a different sector farther away on the physical disk the I/O scheduler may re-order processes' sector level writes. Normally journalling addresses this issue but if the journalling transactions are reordered then it's no help at he application level.

The Linux Documentation Project has this to say about flushing of buffer cache: "In traditional UNIX systems, there is a program called update running in the background which does a sync every 30 seconds, so it is usually not necessary to use sync. Linux has an additional daemon, bdflush, which does a more imperfect sync more frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes causes."

Wow, I'd love to know what they mean by "imperfect". In Linux, specifically, the sync_dirty_buffers syncs all the dirty write buffers (nope - see edit) and isn't imperfect or incomplete in any particular way.

Edit: Nevermind, I'm remebering now there is a dirty_background_ratio separate from a dirty_ratio. It will sync some of the dirty buffers as long as it's above dirty_background_ratio and under dirty_ratio. And it was actually bdflush that determined all of this; the sync_dirty_buffer function just took a specific buffer object and synced that.

Edit 2: And I'm finding out that information seems to be outdated. Here's the old documentation about bdflush [1]. That was either moved or removed in the latest version, but I did find this [2], which suggests that there was an old bdflush vs something else now. Looks like I missed something important and have some research to do.

[1] http://lxr.free-electrons.com/source/Documentation/sysctl/vm...

[2] http://lxr.free-electrons.com/source/Documentation/vm/active...

Reply on news.ycombinator.com