File handling in Unix: tips, traps and outright badness | Hacker News Reader

File handling in Unix: tips, traps and outright badness

rachelbythebay.com

134 pointsjanvdberg72 comments

> Put it another way: if your idea of avoiding races in your temp file creation is to just have a wider namespace, what keeps an evildoer (me) from just grabbing every possibility first? Disk space is cheap, right? You can only have so many pids, and the time of day is obviously predictable, and your random number space is also similarly limited. I could just set up every one of them first and when you hit it, I win!

A UUID, generated by the system's CSPRNG. Ideally through a syscall if the OS has one, so I don't need to burn a FD. You can't pre-fill all UUIDs without running out of space, and the CSPRNG prevents prediction, short of using gdb to attach to the process mid-name-generation, at which point you're on the other side of the airtight hatch.

I do wish linkat could do an atomic replace of an O_TMPFILE file descriptor. It'd avoid the whole random-name business, which is an ugly wart.

This sounds to me like seriously bad API design. I'm not saying the low-level functions that allow handling fancy situations aren't important but shouldn't the default be "it just works" and not "you must write 1000s of lines code exactly correctly or else your program will fail and you'll be powned"

This reminds me of Jonathon Blow's rant on similar issues on XBox and PS2/3 where the API, instead of doing the right thing by default required the developer to spend days/weeks using it correctly and having their app rejected over and over and in this particular case there was zero good reason for it not to handle the issues itself rather than pass it all on to the developer.

I think interrupting I/O can be tricky for an application to handle. If write(2) didn't have this behavior, it's likely that a common outcome would be frozen processes that you can't quit with ^C.

There either needs to be a notion of partial writes (what the article describes) or a write that gets rolled back if interrupted (probably really hellish to implement ... Maybe impossible for something like a socket where the remote machine may have already seen some packets).

EINTR or SIGPIPE are famously criticized as poor mechanisms for this. However even without that, interrupting a blocked write would remain a tricky problem.

Writing to a file should be transactional. It either succeeds, or gets rolled back. Journaling filesystems implement it as a crash protection mechanism, but an API where you can commit changes without closing, and can roll back explicitly, would be nice.

Writing to a socket is unlike writing to a local file, and cannot offer such guarantees.

Seems a lot easier said than done, and would probably incur some costs for something most programs would be largely uninterested in.

Case study: Windows has a feature called "transactional NTFS" which I think may have grown out of the WinFS experiment. They duplicated a lot of filesystem APIs with "transacted" versions. I think the ntdll APIs were a little cleaner than the win32 wrappers - maybe at that layer you can set a transaction on a handle and issue normal fs calls? Anecdotally I heard it was flakey. I believe it's in a sort of deprecated state today.

Journalling protects metadata writes (file names, dates, sizes, permissions, root block allocations) not file content writes, such that a FS can be brought up quickly after a crash. If you want transactional/atomic file ops, you need to implement it yourself on top of existing mechanisms, like transactional databases have been doing for ages.

> Journalling protects metadata writes […] not file content writes

As I understand it, the ext4 file system can have full-data journaling with the “data=journal” mount option (instead of the default, “data=ordered”, which is as you describe).

are there measurable performance hits that you take with transactionality?

Certainly there should be. Not always large; I suppose creating and sequential writing into a new file should be about as expensive as non-transactional.

But it sounds from the article that write(2) is more complicated than necessary. Why do some errors cause an error return while other errors cause a signal? I would think consistency in how errors are propagated would be better.

The article I remember reading said that to do it right like most other operating systems of the era did requires a lot careful design and work to get it right, so Unix developers punted and left it to the application layer.

You get bagged on for saying this but it's just inexcusably shitty. And that's that.

You are thinking of "Worse is Better". But the take away of that article, even if the authrs would have preferred otheewise, is that paractical considerations, time to market and ease of implementation beat purity and elegance every time.

A lot of it had to do with code that historically does not check for errors.

> interrupting a blocked write would remain a tricky problem.

Would it? I believe kill(pid, SIGTERM) interrupts it all right. Or do you want to do this without terminating the program? But then we are back to square one: the programs have to handle failed writes somehow. How? "Abort, Retry, Ignore"? And then again, most of this stuff happens deep inside libraries, and every library makes different choices in handling such errors, and when you combine them in a single program, you end up with a mess. As always.

It doesn't... or rather, it doesn't unless the kernel can safely unroll the write from it's insides. Famously NFS mounts can get eternally stuck processes when they die because the other end of the file descriptor doesn't exist and there is nothing the kernel can reasonably do about it.

Uh, return EIO? Or EINVAL?

EINVAL would mean the fd is invalid, which isn't true. The application already opened the file and possibly wrote to it, so EINVAL is not a valid error code.

EIO sounds like it would make sense but can only be returned for errors relating to modifying inodes, the EIO that makes sense for NFS to be gone would be occuring when you call close(2) on the fd, or fsync(2) which has very unpredictable behaviour when it returns errors anyways. Either way, NFS doesn't handle inodes, so an EIO cannot be returned.

That's an overly pedantic reading of the Linux manpage, IMHO. Even then, a) any of those error conditions may be extended to cover NFS failure; b) EINVAL's description already says "fd is attached to an object which is unsuitable for writing" — who says that such unsuitability is an immutable property? After all, writing into read-only fds is EBADF, and c) a new error code could be invented, if one is so inclined.

But then again, POSIX merely states that while EIO shall be returned if "a physical I/O error has occurred", it "may also be returned under implementation-defined conditions". Heck, if you squint at the right angle, even ENXIO could be somewhat appropriate. I don't think that extending the meaning of EIO is worse than making unkillable zombie processes, to be fair.

No, when the process is waiting on read/write to a file, it cannot be interrupted. The signal literally will not reach it until the syscall finishes. This is a deficiency of Linux design inherited from Unix and something that is being addressed over time. However it stems from the real requirement that a device can't just have its operation cancelled, once you've asked the HDD to read a sector, it won't respond until it has finished reading. So the process is stuck in the kernel HDD driver which literally cannot be interrupted. When it's a network or other removable storage obviously interrupts should happen at the media level, but the design did not permit it originally. Currently you can kill an fuse process and NFS has bits to allow signals in the middle of a read, but cifs for example doesn't.

Mmm, but don't modern HDDs use DMA+IRQ for data transfer? The disk driver issues the read instruction and then it's done until the interrupt arrives?

Anyhow, what are the peripheral devices that support only synchronous access? I don't believe there are any such ones in the modern PCs, but maybe there are some in mobile devices? Old floppy drives?

Isn't that the same as EINTR with a different name?

And what happens if some bytes have made it to disk, network, or pipe? Don't you want to tell the caller how many, in case they want to recover somehow?

Congrats, you reinvented the status quo.

One improvement would be to separate the error indication from bytes written. Then you could have a write that fails, with nonzero output. Windows' API allows for this, but I don't know how common it is. It would probably be a source of bugs the same way these write(2) corner cases we are talking about are.

I was discussing a different scenario, "NFS mounts can get eternally stuck processes when they die because the other end of the file descriptor doesn't exist and there is nothing the kernel can reasonably do about it" — the kernel absolutely can do a reasonable thing and return from write(2) with whatever error code.

And yes, Windows API's WriteFile does actually return less lpNumberOfBytesWritten than was passed in nNumberOfBytesToWrite, but only when writing to a non-blocking pipe; the disk writes are either fully written or fully failed. Sockets use their own special API that generally follows the write(2) semantics (although there are some hilarious corner cases with internal buffers: e.g., you can make a 1 GB write to a write-ready non-blocking socket and the internal buffer will grow to accomodate it, but further write requests will return EAGAIN).

Oh, absolutely. The Unix system API is what was easy to implement in 1970, not what's sane for developers nowadays.

Don't invoke system calls directly. Use a library that handles all this stuff for you properly. Better yet, use a language that gives you a better interface than C does.

Advice (which I agree with) that conflicts with the author's previous post referenced in the first paragraph.

I see that post as a double standard. Yes, libraries are mostly crap; that's because code is mostly crap. Fixing the file-writing implementation in a library may take more patience and tact than this author has, but it can be done; fixing what dozens of mediocre programmers wrote on their own is completely hopeless. If you want to talk about an attractive nuisance that lets people write code that sort-of works most of the time but's actually fundamentally broken, that's Unix - and C - all over.

> This sounds to me like seriously bad API design

this illustrates the sad irony of unix's genius. All the people who think like Dave Cutler (or whoever) are eager to bring Cutlerian ideas to unix. But unix is where they are bringing these ideas because unix is what won, and unix won because it did not make Cutlerian design choices.

I'm not saying that an API call that did the right thing wouldn't be great for a programmer; just that there are a lot of (simple) things going on in the kernel and tradeoffs for a simple userspace security model and some of this comes with that territory. And neither am I saying that a Cutlerian model is theoretically unsound. But things play out practically and this is how things have played out, and observers along the way discussed it the whole time.

More to say for sure, but please forgive me. My comment is for people who understand what I'm talking about. Explaining to people who don't understand is a useful and noble thing, but this ground has been covered so many times over ("worse is better"), and it's very detailed and historical and somewhat subtle, and I have limited time atm.

I think the post is dramatized a little. Mentioned SA_RESTART should fix the EINTR problem for most synchronous functions and is used for default signals.

"Doing the right thing by default" is a weird concept.

Who's default are we talking about here? The developers desired state of default, or the users?

Wasn't that the point of the whole O_PONIES thing?

I enjoyed this post and the one before it (linked in the article). +1 for a resurrection of the old PC loser-ing situation made semi-famous by the seminal "worse is better" article (also linked therein). Really this all comes down to, "Where should the complexity live?" People who enjoy languages, tools, and OSes which cater to systems programming (myself included) eat this stuff up. I can definitely appreciate the dismay this causes in folks who simply want to glue things together to produce higher-level results more quickly.

I really don't see how this post supports the one before it. Much of the complexity here can and in many, perhaps most, cases should be abstracted into a library, whether that is the standard library or a well-established high-quality third-party library. Given how easy it is to make mistakes, common operations, like atomically writing a buffer to a file, or creating a new file and ensuring it didn't already exist should not have to be re-implemented in every application.

You need a signal handler to eat it and do something about it, or you have to explicitly say that you don't care and set it to "ignored", but you can't just pretend it won't happen... because it sure will.

The default action of SIGPIPE is to kill the process, which is exactly what you need for pipelines to work correctly. Related article: https://news.ycombinator.com/item?id=22647539

Not all programs are meant to be used in pipelines.

It’s not even entirely accurate to say that that’s EXACTLY how pipelined programs should work; maybe they want to do some cleanup before they die. SIGPIPE was essentially created as a hack for naive programs that didn’t properly check the return value of write(). EPIPE should have been enough, it’s a perfectly fine solution.

> Not all programs are meant to be used in pipelines.

I would argue that programs that aren't meant to be used in pipelines aren't well designed programs.

> It’s not even entirely accurate to say that that’s EXACTLY how pipelined programs should work; maybe they want to do some cleanup before they die.

If you need to do some cleanup before you die, then that's exactly what a signal handler is there for you to do. Nothing stops you from exiting immediately after a signal handler.

> If you need to do some cleanup before you die, then that's exactly what a signal handler is there for you to do.

I challenge you to clean up a temporary directory, like doing "rm -fr $TMPDIR/tmp$SECRET/" except not by running another program, in an async-signal-safe manner inside a SIGPIPE handler which then exits.

Hint: You're not allowed to call system(), readdir(), malloc() or any of exec*().

If you'd check your errno when write returns -1 on standard in or standard out, SIGPIPE would not be necessary.

That would mean duplicated logic across all applications and all places in them.

SIGPIPE is like an exception. If you really need to, you can catch it (to do cleanup or similar, as a sibling comment notes), but otherwise the default behaviour is fine and likely what you want.

I've made use of a similar technique when reading input: in code where an EOF is a rare case, but reads occur in multiple places, you can simplify the main logic considerably if you create a function that contains the read and a longjmp() upon EOF to a specific handler which then uses state variables to decide what else needs to be done; it gets rid of all the "if EOF" duplication and puts all the special-case EOF-handling logic in one place.

To be honest, your solution doesn't sound more maintainable, because now I have to write a complicated longjmp handler and have global state variables that interact with all my read calls.

That only maybe works if you write most of the application code yourself but it would be unrealistic in larger projects that have to do more than read from stdin and pop out data on the other end.

SIGPIPE should not exist, no matter what hacky ways you found to abuse it for your advantage in some edge cases.

> Let's say it returns a value of 8192 instead, setting errno to EINTR.

errno is meaningful only after the function failed. Short write doesn't count as failure.

That's a derp on my part. Fixed. Thanks. Clearly, this stuff is incredibly complex.

Indeed, errno is not cleared by successful calls, so you can't check it for success.

After a short write your only legitimate course of action is to advance the pointer and write again.

You can set errno to 0 and see if it changes.

I'm not sure there is any guarantee that a function that completes successfully hasn't changed the value of errno anyway.

Yeah, I didn't mean to imply that you can ignore the return value.

The syscall is allowed to set it even in case of success.

This is sort of right, but the behaviour is a bit subtle. errno should never be set to zero by any POSIX-compliant syscall or library call. However it may be set to something (which is non-zero) on a successful call. Ref: https://pubs.opengroup.org/onlinepubs/9699919799/functions/e...

note, while this might be true for write it is not true for fwrite

> The fwrite() function shall return the number of elements successfully written, which may be less than nitems if a write error is encountered

> if a write error occurs, the error indicator for the stream shall be set, and errno shall be set to indicate the error

Sometimes you want to use a temp file named ${target_name}.new, then rename it after writing to it -- the point of this is to avoid leaving garbage if you have unclean exits, but it requires using flock(2). Even better is to create an unlinked file using O_TMPFILE (Linux-only) and then use linkat(2) and then rename(2) (renameat2(2) doesn't quite support what's needed, and it's a darned shame). If you're doing this in a /tmp/ directory, well, you do have to worry about attacks by other users if you try to do the .new thing.

Wanting a few more details I googled that and found: https://lwn.net/Articles/559147/

Second time I googled something you mentioned in this thread, eager to know more. This guy filesystems, folks.

The scenario described in the article suffers from one more problem, not mentioned in the article. Quote:

"Create a file adjacent to the target path using mktemp or similar.

Write your data to it

rename() it from the temporary name to the final name."

I believe this is prone to data loss when the system goes down and the cause is that POSIX guarantees on write reordering are loose. The rename can make it to disk before the data gets written.

I think it caused actual data loss with one of the fancier filesystems which actually pushed the limits of write reordering and exposed this (pretty common) scenario as buggy.

Edit: I found an article about this problem. "Ext4 and data loss": https://lwn.net/Articles/322823/

Of course, between finishing the write and doing the rename you have to call fsync() on the fd (as that article pointed out).

Also reminds me of this talk by Dan Luu: https://www.deconstructconf.com/2019/dan-luu-files

He doesn't mention ZFS at all.

So...basically use SQLite for this then as recommended. https://www.sqlite.org/aff_short.html

It has its own share of legacy and poor tradeoffs. Transaction log is an overkill for most applications, but the only alternative is full cache in RAM, there's no middle ground like you have with file system.

The first half of the article basically complains about the write()'s syscall possibility to fail. Errors can occur and you need to handle them, there is no way around it. Given that "Everything is a File", the OS shouldn't try to make any guarantees around that. There are cases where you want to handle an early return by write().

This is a syscall after all. First priority is to provide a very general interface and remain compatible, not developer ergonomics.

I wonder if EINTR can also happen on macOS? macOS is certified UNIX, and the post talks about Unix in general, that's why I wonder.

I've inspected the family of fopen() functions (fread, fwrite), and they don't handle EINTR. Do they delegate this task to the caller? Or that EINTR doesn't exist there? I mean, the error code itself is used, and by quickly skimming XNU sources I can see it's used in some NFS code, but can it happen when reading from or writing to local storage (hard disk)?

Edit: some minutes later, I can see that e.g. libstdcxx supports EINTR, so maybe it can happen on macOS as well (the xwrite() function):

https://opensource.apple.com/source/libstdcxx/libstdcxx-104....

I just tested it on macOS and Linux with the two main sources of EINTR: signal handlers and attaching a debugger. I tested by interrupting a read from stdin, but the behavior for disk reads and writes should be similar.

Signal handlers: POSIX specifies that, after a process runs a signal handler which was triggered when it was in the middle of a system call, either the system call will automatically be resumed or it will fail with EINTR, depending on a per-signal-handler setting. If the signal handler was set with sigaction(), this setting is the SA_RESTART flag; otherwise it can be set by siginterrupt(), but the default seems to be implementation-defined(?). Anyway, it seems that both macOS and Linux enable restarting by default. (But well-written library code should take into account the possibility that something else in the process could have decided to disable it.)

Attaching a debugger (using ptrace): This interrupts system calls because the debugger expects to see a userland state. On Linux it seems to automatically restart the system call, but on macOS it produces EINTR. This behavior doesn't seem to be configurable on either OS.

On both OSes, when EINTR does occur in read(), fread() does not cover up for it by automatically retrying, regardless of whether it successfully read any bytes beforehand.

Which libc were you using on Linux, or were you making raw system calls? Because I think glibc’s behavior is to EINTR unless you define _GNU_SOURCE, which seems to me to be incompatible with your description…

glibc, Debian package version 2.31-3, without any special defines. Here is the program I used:

https://gist.github.com/comex/58ba394588478bb1d506a998850b94...

If I run the program and press Ctrl-C, it prints "interrupted" but then keeps going. However, if I add `siginterrupt(SIGINT, 1)` after the call to signal(), interrupting the read() produces EINTR as expected.

Interestingly, the glibc manual used to state that the default behavior depended on the presence of _BSD_SOURCE or _GNU_SOURCE, but in 2014 it was changed to state that the default is always to use EINTR:

https://sourceware.org/git/?p=glibc.git;a=blobdiff;f=manual/...

Oh, huh, that’s really interesting! I didn’t realize that glibc had changed their behavior; I wonder if this broke any applications…

Does write situation apply to linux as well?

Since 99% of the disk I/O is cached, write either success or fails completely. It's beyond scope of the userspace application to know what and when data arrives on the physical disk. Same applies to Windows as well.

Using direct I/O is different story where this is the correct behaviour.

NFS. Also the gdb/strace case still holds. It's really rare so many programs get away with not doing it, and then when it happened once in a honeymoon your shrug it off and restart the program.

I hope you’re not fixing short writes during your honeymoon…

Maybe I have skimmed too much, but I didn't see anything about the problems with using O_EXCL to resolve races over NFS. I believe (correct me if I am wrong) it doesn't work.

No, O_EXCL works on NFSv3 and NFSv4 just fine. NFSv4 is stateful and solves a number of problems that NFSv3 had.

Googling around has this for Linux: http://nfs.sourceforge.net/#faq_d10

Tldr: it used to not work. It was fixed in Linux in 2004.

No idea about other unix-like OSs.

In my experience, most developers ignore the "partial write" scenario with files. Then they get experience working with sockets... where this happens all the time.

> Then they get experience working with sockets...

Ha. I have seen TCP sockets code which breaks in a "partial write" scenario, and even a "partial read" scenario.

Code worked fine on their own LAN. Broke for someone else, they couldn't see why...

They were very surprised when I explained that you don't always get one message per read(), because they had for years. Or maybe they hadn't, but assumed something else caused their occasional application glitches.

If you're doing async IO, I strongly recommend using a library that abstracts much of the pain away, like libuv. Deal with just the libc functions (or system calls) is a way too low level API for many applications.

And this is still only scratching the surface. For more file related horror:

Files are hard - https://danluu.com/file-consistency/

I remember being bitten by write doing only partial work early in my career.

This definitely felt like an API shortcoming, ended up writing a wrapper for write that did what I expected.

Reply on news.ycombinator.com