Files Are Fraught with Peril (2019)
danluu.com
danluu.com
It seems obvious to me that we need a transactional filesystem API with methods like:
fbegin()
fwrite() / rename / etc
fcommit()
This would dramatically simplify the code in every program that wants to safely interact with the filesystem. And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing. Because fsync() is so coarse, I suspect performance would improve with this API, too. (So long as the transactions are mostly small.)
But why don't we have this API already? My theory is that kernel developers don't write enough userspace software to understand how bad the status quo is. And most people writing software in userland don't understand the kernel well enough to feel empowered to change it. Adding an API like this might also need changes in the filesystems.
But wow, it would be a huge boon to humanity to have a filesystem API which isn't so braindead. How many more decades do we need to have our files corrupt by default when errors happen? I wince every time I'm playing a video game and it has that "saving - don't turn off your console!" animation. Or during OS updates. Data corruption happens because we implemented the wrong abstractions 50 years ago and somehow forgot that code is mutable.
> Microsoft has deprecated TxF and stated that it may be removed in a future version of Windows
> While TxF is a powerful set of APIs, there has been extremely limited developer interest in this API platform since Windows Vista primarily due to its complexity and various nuances which developers need to consider as part of application development.
> As a result, Microsoft is considering deprecating TxF APIs in a future version of Windows to focus development and maintenance efforts on other features and APIs which have more value to a larger majority of customers.
"TxF is presently deprecated, but the functionality is still available. We don't recommend disabling this feature on the C: volume."
Microsoft supports APIs which were declared deprecated for decades.
However, I totally agree in "Basically nobody is using it" part. It's because it was not properly implemented in .Net Framework. And now, when .Net Core is multiplatform, i.e. worst of all worlds, chances it will be implemented are very close to zero. ReFS does not support TxF.
Windows has a lot of awesome APIs which are accessible only to C/C++ and are not advertised well.
Of course it is possible to mitigate these issues more than most file systems do, but at the cost of reduced space or performance efficiency. Thats what using a further abstraction like SQLite buys you, but its not for free. Thats why all the FS as database efforts in the early 2000s like Longhorn failed.
According to the article, if I use ext4 with journaling and have a file with "a foo" and write "bar" at position 2, possible resulting states are: "a foo", "a boo", "a bao" or "a bar". This is the best scenario - without journaling, I can also end up with uninitialized junk.
The authors of the filesystem APIs consider this to be correct behaviour. But I can't think of any use case for this nonsense behaviour. Real data, in every file on your computer is structured. And structured files simply can't handle this kind of data corruption. Maybe this API has slightly higher throughput, but by the time you've peppered your code with fsync() calls to make it correct, you've killed performance. And working around that takes whole philosophies of threading and thousands of lines of code.
I want that write call to either complete completely or fail completely. Actually, I want more than that - for a bunch of successive write calls touching multiple files to collectively succeed together or fail together. Which is not what the article is saying.
> Its like saying, well software just shouldnt have bugs
The kernel shouldn't have bugs in its filesystem. We don't tolerate bugs in the scheduler, or in OpenSSL or in the CPU. If Postgres randomly corrupted its database every month or two, we would fix the bug. Or call it broken and use something else. If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel. And in the kernel, that code would be easier to optimize, and would be useful everywhere.
If it was a simple bug, we would fix it. The problem is that the bug is that we have the wrong abstraction. Why do we have the current filesystem API? The story I heard is it was written by some intern at Berkeley decades ago who thought it would be neat to have files and network streams use the same API. We know more now, and its past time to fix the mistake.
> Thats why all the FS as database efforts in the early 2000s like Longhorn failed.
I don't buy the cynicism. Just because microsoft overengineered longhorn (and successfully shipped TxF btw), doesn't mean the problem is unsolvable.
And legacy software can just keep using the current unreliable APIs with all the performance that entails anyway.
It makes a lot more sense if you consider it shared, concurrently accessed memory. If you try to access it without atomics you can get funny results. And if you kill one process in the middle you might be stuck with reordered accesses indefinitely. Something like ext4 provides more than that.
If you want ACID then you'll have to opt into that. The same as you have to do with concurrent programming (e.g. via std::atomic). But there are many cases where none of this is needed, where all you need is a bag to hold a few bytes for a while. E.g. cache files or large media blobs.
The only issue is ubiquity of the necessary tooling, or the lack thereof. Many standard libraries now ship with concurrency-safe data structures while you often have to reach for 3rd-party libraries for safe file abstractions if you don't want to code your own.
> If SQLite can implement transactions on top of the current filesystem APIs, then obviously its possible to put that behaviour in the kernel.
If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side? Userspace needs many different levels of granularity. Sometimes integrity doesn't matter at all. Sometimes it only matters that writes make it to disk in correct order but it's ok for the last ones to get lost. And sometimes you need multiple processes memory-mapping a file and doing shared byte-level access and only random byte-ranges need to be committed to storage at any given time.
But its not concurrently accessed memory - unlike memory you often want filesystem operations to run asynchronously.
And even if you want to think about it like memory, we're missing all the tools we have for managing concurrently accessed memory like atomics and granular load and store barriers. And even then only about 1 in 10000 programmers know how to use those primitives to implement correct software anyway, so thats not exactly what I'd reach for when I think of good API design.
But even if you want to use that analogy, the filesystem API doesn't have APIs for granular memory barriers. Until recently all we had was fsync. Peppering your code with fsync calls destroys your performance even though it doesn't change your program's behaviour. And if you batch up your changes and then fsync returns an error, you have no idea which writes succeeded and which failed. inotify helps, but oh goodness does it get complicated.
> If userspace can already implement it (when needed) based on provided kernel primitives, why should the kernel maintain two APIs side by side?
This is a good question. I guess I don't really think much about caches, or processes sharing a pseudofile to communicate. For those programs, sure - the current behaviour is fine. I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk. Those programs shouldn't randomly end up in a corrupt state if power goes out at the wrong time. This is the core use case for the filesystem and I think its pretty reasonable for the kernel to make that case easy and correct by default, without resorting to storing all user data in every application in sqlite.
From Linus talking about this issue ( https://lkml.org/lkml/2009/3/25/632 ):
> The problem is not that we have a lot of fsync() calls. Quite the reverse. fsync() is really really rare. So is being careful in general. The number of applications that do even the _minimal_ safety-net of "create new file, rename it atomically over an old one" is basically zero. Almost everybody ends up rewriting files with something like open(name, O_CREAT | O_TRUNC, 0666); write(); close();
Basically, the reality on the ground right now is that:
- Its almost impossible to reliably update a saved file right now without sometimes corrupting data. From the article, researchers found sqlite was the only correct database that existed based on their survey.
- Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used.
Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct. A good API makes simple things simple and complex things possible. The posix API fails here. It makes the simple task of updating a save file almost impossible.
I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data. But nobody seems to be making that argument, and thats not what reality looks like. So long as the filesystem API is advertised (and used) as the standard way to persist user data, it should do that in a simple, safe and correct way.
Well, but it is. Files can be opened multiple times and file descriptors can be shared between any number of processes and threads. mmap, atomics and msync also give you granular access.
Most people might not use it that way, but the thing is that it can be used that way. Just like RAM can be accessed randomly from many threads but higher-level languages limit what you're allowed to do concurrently. It's the same fast-but-unsafe-guts, use safe abstractions thing.
> I'm thinking about the 95% of applications out there which want to use the filesystem to save user data to disk
Those 95% of the applications aren't necessarily those doing 95% of the write IO though. Looking at the activity on my system then most of the write traffic is a) logging (including messaging) b) firefox occasioally showing 1GB/s places.sqlite spikes (hah) c) temporary files d) docker e) p2p storage.
For logging O_APPEND's atomicity promise may be good enough. I don't think text editors need transactions either, they only need file-level atomic replace which can be handled by a single library function. Such functions should be available in more libraries, I can agree with that.
> Why should the kernel maintain an API for this? Because the default, obvious way people use the filesystem should be correct.
Why the kernel and not libc? Or some other userspace lib for that matter.
> I'd be happy for an argument that the posix filesystem is too low level to be used to directly store user data.
That might be a matter of perspective. open/write/close is perfectly fine if all you do is move 100s of RAW photos from your SD card to your file server. There's no rewrite-in-place happening, it's a one-time thing that at worst would leave you with one incompletely copied file if the PC crashes shortly after copying. And the SD card wasn't a bastion of reliability anyway. Docker layers and containers are quite discardable and I wouldn't even mind if they had 0 durability. git, sqlite and postgres already seem to be doing the right thing. Whether document processing software does, I don't know but I assume they put some effort into it with all their autosaving features. But, as mentioned above, all they really need is nicer standard libs not a new kernel API. That doesn't leave that many things I deem critical. Perhaps it is a little annoying if some of the configuration text files of various GUI applications get lost in a crash but on the other hand I wouldn't want them to fsync my preferences every time I tick a checkbox so I'd only want atomicity but not synchronous durability.
> - Most programmers don't even bother to use fsync. Anything which starts with "you can solve the trivial problem of saving a file in this super complex way" is a nonstarter and will almost never be used.
If people can't even be bothered to fsync then why would they be bothered to wrap IO into a transaction? After all APIs still would have to continue to work in non-transactional mode for performance and backwards-compatibility reasons so there would be no pressure on people to use transactions.
Maybe this also is a documentation problem since many developers don't seem to be aware of the non-durable-by-default behavior and different steps needed to get atomicity or durability. Man pages and standard libraries lack a high level overview of filesystem semantics. Some of it is noted in fsync() for example but not in rename()
It works ok most of the time.
And if you really care about some data, you will stick it some database anyway.
And if you are going to stick data into database, you want to be able to select database that suits your needs best, and not be stuck with one provided by os.
B I think you are underestimating how hard is it to implement good and performant transactional database. I am not sure it belongs into kernel.
C. A transaction based api would fail anyway occasionally.
D. Note that some OSs (MVS/OS360) that are famous for transactions have the reputation that locking and so-forth are done in the kernel. Which is not actually true.
We put things in the kernel for several reasons - performance, but also consistency and interoperability. If every program embeds its own database and its own semantics, its much harder to have programs share data.
You can do some work in advance with sync_file_range. fsync is still needed to write out the metadata, but it'll have to do less work and you may be able to get better pacing and backpressure that way.
> Then the fcommit() call would atomically flush the locally buffered changes to the filesystem. If an error happened, the API should guarantee that the final state either contains all changes or none of them.
Many existing filesystems simply cannot support such an API because they're not journaling or CoW. Another issue might be that multiple transactions could overlap and be of unbounded size, so now you would need MVCC, exclusive locking or the ability for the kernel to provide feedback to userspace that it will immediately rollback the transaction due to resource limits or conflicts. This suddenly looks more and more like a database than a posix filesystem.
There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead. Lazy writes that the kernel can flush to disk whenever it feels like provide a lot of the performance that we take for granted when working with large amounts of data.
> And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing.
I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace.
Are you talking about copy-on-write or redirect-on-write, which seems to be the real copy on write ?
> Despite their names, copy-on-write usually refers to the first technique.
Thats fine. Legacy filesystems can just use a best-effort approach like they do now with 95% of programs. I already wouldn't put data I care about on ext2. Programs could detect transaction support with a syscall then decide whether to fallback to old behaviour, lose reliability or for a database maybe just refuse to run entirely.
> This suddenly looks more and more like a database than a posix filesystem.
POSIX filesystem apis aren't fit for use when you care about correctness. And as I understand it ZFS, btrfs and APFS already look more like databases internally than traditional posix filesystems. If that ship is sailing anyway we may as well get all the benefits we can.
> There are many use-cases that don't need such hard guarantees and want throughput, minimal overhead and latency instead.
Are there? I can't think of many. Maybe copying big files? We'll need to keep supporting the current unsafe-by-default posix API forever for current software anyway. I just don't think it should be the default for 95% of applications. When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes.
If the recommendation is that everything that matters should be in a sqlite database because posix is so hard to use, well, that sounds like we agree more than we disagree. It seems like a pity though, and I think the filesystem is worth salvaging.
And with the API I'm suggesting, I don't think large operations need to get much slower. The transactional API could just optimistically write at full speed to unlinked sectors on disk, then atomically update pointers to reference the new sectors when the transaction is committed. That wouldn't add much amortized overhead. I'm sure there's more complexity than that in practice, but it sounds pretty tractable and fast in the common case of a single writer. (And in the slow case where multiple programs try to fight over the same file, slow would be an improvement over the current behaviour of "probably everything breaks".)
> I think the postgres/fsync error saga raised awareness of that and there were some proposals how to provide a better error channel to userspace.
Its still almost impossible to write a database correctly on top of the POSIX APIs. Especially if you want it to be correct. That seems fixable. We should fix it.
I don't mean everyone should use sqlite. We just need more safe abstractions over posix APIs that cover different use-cases. If you need to concurrently query and modify many small records, then sure, go with sqlite.
> Are there? I can't think of many. Maybe copying big files?
It's not just the big files. Millions of small files would cause some overhead if you actually tried commiting them instead of relying on best-effort behavior (see git case below). Temp and cache directories. Archival. Compiler artifacts. Often you don't even want them to actually hit the disk at all if you have enough RAM. A file can act as nothing more than named swap space.
> When I do a git commit, its plenty fast enough. But I definitely don't want my repository to get corrupted if my computer crashes.
Sure, that's for the canonical git storage, which rarely touches many files anyway. But if you checkout a worktree (e.g. by switching branches) then putting all the thousands of files from the .git object storage into the work tree does not need durability. You want to be able to work on those files as soon as possible, even if they're just in the page cache and not written to disk yet. The command shouldn't block on fsync.
I reach for lmdb for a lot of small projects. It feels like failure though. We shouldn't need things like that just to deterministically save data to disk.
> The command shouldn't block on fsync.
Sure - some operations can and should discard durability for higher performance. Sounds great to me. But maybe you should opt in to that. It certainly shouldn't be the only behaviour the kernel provides. If Inkscape crashes halfway through saving my work, it shouldn't by default end up in a half-saved state where my file is permanently corrupted.
But we can have our cake and eat it too here. It can be as simple as, if you don't wrap your write() calls in a filesystem transaction, the behaviour stays the same as it always was. Or maybe the default behaviour is that a write call entirely happens or entirely doesn't happen, but you can optionally ioctl(NO_DURABILITY) or something for extra performance in git checkout.
I hear you that git checkout performance (and things like it) are important to you. Keeping the current performance of those operations sounds good to me too!
So let's get rid of them. And that leads to "but what about all the software that hasn't updated to the new APIs?" Guess what?
> Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
I think he's wrong. Compiler / PL theory works. Distributed systems theory (TLA+, etc) allow us to write correct software. I've used fuzzers to find and fix bugs in databases and concurrency algorithms, and the result is robust, working code.
Theory can add complexity, but complexity added in one place can remove complexity from another. (Eg TCP is complex but reliable, in order delivery in the network layer makes the application layer much simpler and free of bugs.)
Linus's argument is that users will keep writing open(); write(); close();. I think he's half right - thats the right level of complexity the API should have, with no tricky, buggy fsync barriers. But we've known how to write simple, correct APIs for modifying data like this for decades. The answer is transactions. Keeping the kernel as it is just keeps us in a dark age of buggy software.
(And the same thing happened to GNU/Linux, so Linus would know something about it !)
Yes, it is. It even has its own (four-layer) model but the IETF People aren't nearly as interested in layering as the OSI people were:
Layering Considered Harmful
https://tools.ietf.org/html/rfc3439#section-3
Point being, this is why people who know history get a bit perturbed when people insist on shoving TCP/IP into the OSI Model, or, worse, insist that TCP/IP "implements" OSI somehow. OSI had its chance, it lost, and now we use POP and SMTP for email, not X.400.
Are transactional databases useless theory or simple and pragmatic? Hash tables? Nagle's algorithm? CRDTs? SSA compilers? Your answer probably says more about your experience than it does about the technologies themselves.
And thats why I don't trust Linus's rant. He's familiar with the garbage fire of the current posix filesystem APIs. I suspect deep down he's just arguing from status quo bias and doesn't like the "useless theory" of transactional filesystems because he hasn't seen the practical side of them and he hasn't thought through how they could be implemented efficiently in the kernel. And thats fine; but it might mean if we want transactional filesystems to happen in linux, it might make sense to start by implementing them in a BSD.
Another Linus example would be git, which he allegedly threw together in a couple of weeks, compared to pijul, which seems to have required years of work, but now is going to have a very hard time to catch up...
Linus has tunnel vision about this, and it wasn't true in 2009 and is becoming increasingly untrue.
The people who need better tools to deal with this are the language implementers who almost always abstract over this. And since most language implementers target Windows and OS X as well as Linux, they are generally more aware of what better could look like than Linus.
Do they? What languages abstract away the filesystem or the OS's file abstraction?
That seems harsh. Even a transactional FS would have this message. Saving takes time, and the best case scenario for shutting off the machine before it's complete is still data loss. What alternative interface do you propose?
The behaviour should be that if power goes out while saving, the game either updated the save file or it didn’t. Saving should be atomic. That’s the best we can do, and what the user should expect if power goes out before it’s finished saving the game.
I'm quietly hopeful it will de-depreceted at some point. :)
So, if you're writing a database, there's already a bullet proof and fast backend to store things that comes with the OS and is maintained. No need to evaluate various layers on top of a stream (RocksDB, LMDB, etc).
The UNIX way would be to have a filesystem that does all the internal logic of checking line width, and allow the user to mount it. One API, different functionality
It's much easier to just do everything inside a single software as a monolith than to distribute functionalities, I don't think one needs to look further than that
https://news.ycombinator.com/item?id=20509533
A temp file plus rename seems to be close but not fully enough.
This is exactly the sort of thing that should be abstracted by a standard library so that whatever needs to be done is correctly done.
> I was also (incorrectly) assuming that renaming a file after having fsynced the contents is a valid mechanism
Where does anyone say that this doesn't work?
I read the article twice and it suggests that open+write+fsync+close+dir-fsync is safe ("We've now seen what we have to do to write a file safely").
I've also read the reference "Bornholt et al., ASPLOS’16" mentioned in the "Rename" section; it says "One possible fix is to add an fsync(fd)". I cannot find another place in that paper that says otherwise. The later analysed Atomic-replace-via-rename (ARVR) and Atomic-create-via-rename (ACVR) approaches skip the fsync.
That said, user "tfha" replied to your linked thread
> I can confirm experimentally that the rename trick is not good enough.
but unfortunately without any data on whether the directory was fsynced, or on which file system.
So I'd like to ask again:
Is there any evidence that open+write+fsync+close+dir-fsync does not work?
The article gives a concrete example of a failure mode: fsync can silently fail. If this happens then "open+write+fsync+close+dir+fsync" does not work.
Naturally I'm not talking about the "if nothing goes wrong case", but the power failure case, and thus obviously you shouldn't omit the fsyncs.
> fsync can silently fail
I don't think that's what the article says, nor do the linked references such as the LWN article about postgresql's "fsyncgate", where the problem is about re-opening files, and not re-doing writes when fsync returns failures.
In what I am talking about, and what "Bornholt et al., ASPLOS’16" are talking about, you do `fd = open(); write(fd); fsync(fd), close(fd)`, and I have not found any report of that silencing errors or not working as expected, except the linked HN comment by "tfha".
A different failure mode: the drive reorders writes and doesn't respect flush commands.
Edit: of course, you can't avoid the problem by changing the way you write the file. Files stored locally are inherently fragile.
> Where does anyone say that this doesn't work?
The article explains it here:
> This trick doesn't work. People seem to think that this is safe becaus the POSIX spec says that rename is atomic, but that only means rename is atomic with respect to normal operation, that doesn't mean it's atomic on crash. This isn't just a theoretical problem; if we look at mainstream Linux filesystems, most have at least one mode where rename isn't atomic on crash. Rename also isn't guaranteed to execute in program order, as people sometimes expect.
According to the article, open+write+fsync+close+dir-fsync is indeed safe on ext4 on any new-ish Linux kernel (on older kernels, fsync() wouldn't report errors in some cases, e.g. if it happened after the FS attempted and failed to write the data before the fsync).
Not sure what it has to do with rename though.
There are too many people writing software, who have no business doing so.
> There are too many people writing software, who have no business doing so.
Sounds like something a hipster would say ?
Please don't sneer.
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely.
Re the latter: could you please stop creating accounts for every few comments you post? We ban accounts that do that. You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...