Hacker News Reader

> All filesystem operations on modern computers should be atomic

Well that seems a bit over the top. You would pay the price of full consistency for every write, while just one write every 3 years will fail. It would be insanely slow and wasteful.

Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases.

> the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them

This is a bit of wishful thinking. We've seen already on numerous cases (Nvidia?) that OS developpers have very little leverage on hardware manufacturers. The best they could do is issue some kind of "Linux certified" label, which honestly nobody would care about.

Also, I guess there is some form of optimum that users and providers reach in terms of stability versus performance. If one OS provides x10 performance for 99.99% of uses while an other one provides 100% safety for /10 performance, I'm not sure users would choose the latter.

> Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases.

Why would it disallow caches? Just issue writes to a new location on disk and switch reads across to it after the write transaction has committed.

In any case, most writes aren’t performance sensitive. I’m thinking things like saving a file when code editing. My editor shouldn’t need to do some obscure, fragile renaming dance just to overwrite a file safely. It wouldn’t write 10x slower. Just have the OS do what it needs to do so my editor doesn’t have to.

And big transactions (like installing software or OS updates) could just do the whole update to a new location and then atomically swap to it. It shouldn’t cost any performance at all beyond a few pointers being moved around at the end of the transaction.

Maybe doing it for all writes is too much. Let databases opt out if they want to do their own thing. But I think it would be a much better default. Right now all good software does some version of this themselves. An OS level solution would allow all those hacks to be removed. If anything, I’d expect performance to improve.

I think we have a quiproquo about the level of atomicity here.

You seem to be talking about _application level_ atomicity, like when you issue 4 write() to a file and want an "all or nothing" flush of these 4 writes.

I'm talking about OS level atomicity, as in "I want a write() (or fsync) to block until it is flushed to disk".

Am I understanding correctly?

Atomicity means something either happens entirely or it doesn't happen at all.

> I'm talking about OS level atomicity, as in "I want a write() (or fsync) to block until it is flushed to disk".

Thats not atomicity. Thats a blocking API. And we already have an fsync that does that.

Blocking and atomicity are unrelated properties. You could have a blocking write thats not atomic (so if your computer turns off halfway through the write, it might have only half completed). And you can have an atomic write thats non-blocking (async). Or both or neither.

> Thats not atomicity. Thats a blocking API. > Blocking and atomicity are unrelated

It's the same in this context. fsync uses a blocking API to implement atomicity. That is, after an fsync() all previous writes should have been written.

> we already have an fsync that does that.

Did you actually read the article...? The whole thing is about the fact that disk controllers can eagerly notify that a write was flushed to disk while in fact it is still in the disk write cache, and could have been either partially, fully or partly written, thus tricking fsync().

> It's the same in this context. fsync uses a blocking API to implement atomicity

No, it doesn’t.

In that context fsync is a write barrier. It doesn’t make writes atomic. The difference is this: If you issue a handful of writes then your computer crashes, some, all or none of the writes may have completed. And some writes might be “skewed” - where part of the write succeeded and part did not. Atomicity means a set of actions happens as one indivisible unit, and it either all happens or all doesn’t happen. Think database transactions. Filesystem writes are not like that. They are not atomic operations[1].

Fsync does not change the behaviour of the write system call. Fsync only stalls your program until all writes have been written to disk, at least assuming your drive isn’t lying to you. Just like a memory barrier. If your computer crashes before fsync returns, all bets are off.

[1] With the exception of aligned writes under the device block size. Allegedly. I’d still like to see this actually documented somewhere.

Windows Vista introduced in 2006 a transactional file system API for user applications.

It was recently deprecated because almost nobody uses it.

https://learn.microsoft.com/en-us/windows/win32/fileio/trans...

I wonder why it wasn't used. Any insights? IMO they should have just made the regular write calls atomic across the OS and have applications opt out if they want.

My guess would be that nobody seriously considers Windows servers for enterprise storage. I've worked for several storage companies and Windows support in that world is something that people usually want to overlook, or to provide a bare minimum of. I.e. think about companies like Dell EMC (their storage division) or Infinidat etc... it's mostly Linux with rarely some BSD in the mix.

Non-atomic stuff is simpler on the implementation. Even if the behavior is worse for the user, it's easier to write, and you have fewer requirements to satisfy. So, not having a real pressure from the customers, it's just easier not to do this.

They mention in the docs it was somewhat complicated to use.

Very few apps need more than the ability to atomically swap a file.

And for the complicated multi write cases, a lot of software just use sqlite which deals with it.

I think this kind of design would have crap performance (or at least enforce it). The reason is that the disk controller has its own set of RAM that it employs on the other end of the PCIe bus which could be battery backed and be designed to keep the controller alive long enough to flush everything. That’s not possible if the OS is in charge (the power requirements for the full OS are orders of magnitude higher). Additionally, you can’t have hybrid NVMe spinning rust disks with that kind of design.

Finally, it’s unlikely that implementing it in the OS would be faster because there’s a bunch of flash translation logic and garbage collection that has to happen - much faster if all of that happens on the other side of the PCIe bus with a dedicated coprocessor with a dedicated SPI directly to raw flash.

I’m imagining a software solution using some form of journaling / checkpoints. Instead of overwriting data in-place, send all writes to fresh, unused storage and then “commit” by updating a pointer to avoid write amplification. You’d need a bit of extra machinery to handle reads on the old data. As I understand it, NVMe drives already send all writes to unused space and then update pointers to reclaim old capacity.

We could start with a nice implementation in the OS. Probably with a way for applications to opt in and out. And then optimize over time. Optimizations could include better integration with the way NVMe devices actually work, rather than the way they pretend to work to the rest of your computer.

file systems already do journaling and a bunch of other fancy techniques. Drives themselves then have their own versions in addition to that

Additionally, apps can do whatever they want in custom partitions if they need better storage perf (databases for example do this). Getting lower level than that is probably doable, but you run the risk of becoming firmware that is tied to particular drive controller design

Remember that computer storage technology is over 70 years old, many many things have been tried/implemented in that time.

File systems only do journaling for their own metadata.

Has this been tried? It seems like an obvious idea to me, but operating systems have evolved remarkably little in the last few decades. Especially the filesystem.

https://en.wikipedia.org/wiki/USN_Journal

McKusick is an OS writer (4.2BSD and 4.3BSD), he is writing about the challenges systems programmers face with deceptive hardware.

Hello Llama, go back to your training data and try to get it right this time.

> There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state.

No structural reason no, but a lot of other reasons. You can install your favourite atomic file operations library in your PL of choice and run some benchmarks to identify reason #1.

This is like saying all network traffic should be tcp. For many applications there is simply no good reason and nobody would be willing to pay the additional costs.