Filesystem devs should aim to make “badly written” app code “just work” (2009)

It would be nice to be able to have a process tree own a temporary file, such that when the last process in the tree exits (not necessarily the process which created the file), the file is automatically deleted, rather than having to wait for the next reboot.

We have the temporary files you're asking for. https://lwn.net/Articles/619146/

With O_TMPFILE, you can also write new data, and then automatically replace a file on disk.

Classic MacOS-9 had the PBExchangeFiles call which did this perfectly. Before call:

    dirEntryA -> fileContentsA     
    dirEntryB -> fileContentsB     

    dirEntryA -> fileContentsB     
    dirEntryB -> fileContentsA     

So when saving a new document you wrote it to a new hidden temp file, and when everything was written, you called PBExchangeFiles to swap the contents of the old and new files. After this you deleted your temp file which now contained the old document content.

http://mirror.informatimago.com/next/developer.apple.com/doc...

Consider that filesystem data cannot be synced to underlying storage in a such way that it can be reliably read later and backups by definition are asynchronous and provide eventual consistency. This means that the best meaningful guarantee an application can get after machine crash is being able to read data written up to some point in the past, but not after every successful fsync before crash. Usually, though, even that guarantee is hard to achieve, as bad blocks happen and redundancy is not there. Given all that it's ok to relax filesystem APIs behavior to proper physical constraints. Like fsync is only meaningful as an ordering operation, no need to actually flush anything to disk immediately on fsync or any operation.

Next is multi process and multi threaded scenarios. Should O_APPEND only work correctly from a single thread, should each write be atomic and to what size, we certainly can't have gigabytes in an atomic append or should there be some synchronization mechanism that blocks others? Same for temporary files and unit files.

And what to do on bad blocks? Should there be redundancy within a single disk, should block device underneath be log structured storage with block remapping and scrubbing and provide reliable storage layer to the outside by sacrificing space? Maybe for desktop machines it should, but not for servers, at least not all of servers, they need a different API.

I'm not even touching performance considerations here, that depend on performance-friendly APIs a lot.

I wonder if this would ideally also provide a rotation API. Rotating log files in the presence of multiple writers is messy, and maybe shouldn't be reimplemented at the application layer every time.

> Temporary files

Windows is not always a model of filesystem elegance, but it has a "delete on close" flag in their equivalent to open, which makes it go away on the last close (handle can still be duplicated or inherited, so you get some reference counting through that).

Actually I think you can do similar to this on Unix by unlinking after the open, but keeping an fd open for the lifetime of the temp file.

Compare that with a GPU. It took decades, but (at least for nvidia GPUs) they have finally reached the point where you can be at least reasonably certain that high-performance is the default. Sure, you can ruin your performance by doing some stupid things, but it's at least harder to do those stupid things. The defaults are high-performance.

It's exactly the opposite for Cloud TPUs. By default, it only uses one core. You have to use their TPUEstimator API, which is a byzantine mess. Since two weeks ago, I've spent roughly 3 solid working days of effort solely trying to read and understand (a) what is the TPUEstimator actually doing? (b) why is it supposedly so much faster?

I have some half-hearted justifications -- the answer seems to be "you have to colocate your gradients with the device; you have to scope your tensorflow graph to a specific device; and each TPU core is a separate device." But there are unanswered questions. For example, you're supposed to pass your tensorflow computation to tpu.rewrite(). Yet I've never done that. The defaults seem to just work. So does that mean it does the rewriting for me automatically? Is it being emulated in software, and I'm ruining my performance? Tensorflow, why don't you just crash instead of being so damn slow on TPUs? That would at least let me aim my optimizations!

If only I had low level access to the actual TPU operations, I could just write a compiler that specifically emits instructions to give me the performance I need. But this tensorflow graph abstraction makes everything "easy" yet exponentially more complicated.

Anyway. Yes. More of your mindset, please. Simplicity is such a lovely metric seldom optimized for.

(Anyone who's curious can see a dramatic 11x difference in performance on GPUs vs TPUs in Colab: https://twitter.com/theshawwn/status/1196593451174891520 ... this was very surprising, since Google's marketing would have you believe that TPUs are the bee's knees. Yet by default a TPUv2 is 11x slower than a K80 GPU for some basic matrix multiplications.)

As the complexity of the projects I work on has increased (especially in the time since I became a part of a professional dev team), I've come to realize that most of that tricky, complicated-looking code I had read years earlier was actually kind of the easy way out. When you are just trying to get stuff done, the complexity of your code mirrors the complexity of the project. There's no effort put in (and often no time to do so) to create a smoother interface to shelter the code from the complexity of the task at hand.

It's much more interesting to me now to be faced with a complex task and to figure out how to make the code simple and clear. Elegant, well-designed libraries/APIs belie the challenge in writing them. The code looks so simple that it feels immediately obvious as you're reading it. I've come to realize that reading a code base that seems dead simple -- so simple anyone could instantly understand it -- is actually often inversely related to the difficulty in creating it.

Yes, necessary complexity does exist, and sometimes there's no way around doing something fairly nasty in your code. But as an ideal to strive for, I find 'simple' to be fascinating.

A perfect C++ program that outperforms a simple looking python program which does the same thing isn't worse because it's harder to read or to write, or better because it's faster, it's just different.

If you need to optimize for readability, so be it. If you need to optimize for performance, so be it.

If you write code executed trillions of times in a trillion places, where power use and the expense of hardware is dependent on the performance of the software, I think it's okay for it to be as complex as it needs to be to be very optimal, at the expense of readability and other things. It should be well annotated, sure. That's always a reasonable thing to expect except in rare, extreme cases where you don't even have the time to type out the comments.

It's important to weigh the relative importance of simplicity vs speed vs memory frugality vs portability vs maintenance difficulty (dependencies and tool choice factor in here) and so on to determine the best approach.

Dogma about how best to code in a general sense is only helpful if you don't want to think or talk about what's the best fit.

To some people, C is simple. To other people, C is full of "complex and subtle" sources of pain.

To some people, a VM with a GC and a JIT is "complex and subtle". To other people, using a HLL with those features enables them to write programs which are much simpler and clearer.

These positions also change over time. A JIT is a fairly standard technique in 2019, but in 1973 it was still very much in the research phase. Even the idea of writing a kernel in such a HLL as C was once revolutionary.

I have an SDET friend who calls these types of universal truisms "apple pie", as in, you hold a meeting and say "Apple pie is good, right?" and everybody nods and says "Yes!" and then goes back to work and absolutely nothing was learned or decided.

POSIX, and by extension, the classic 1960s-1980s era UNIX way of doing things just needs die a long overdue death.

This stuff was designed at a time when every CPU instruction mattered, everything was optimised to death for frugality, and commands were abbreviated from "copy" to "cp" because ermahgerd two bytes is a huge saving! That mentality got us Y2K. This is an era where latencies were not the bottleneck, CPU cycles and memory bytes were.

A lot of stuff in filesystems is just plain stupid. For example, why do applications install their files. one. at. a. time? Like... what the fuck? How does it make any sense for an application to be partially installed? Who actually codes their application with 500 modules and dynamic libraries to be able to handle the scenario where one of them is inaccessible due to an ACL or a mismatched version because of an overwrite by something or someone else? NOBODY, that's who. Meanwhile, I can make a cup of tea while Adobe Lightroom launches on an SSD drive because it is 99% OS API overhead and 1% usermode action.

This is why Docker is popular. Not because Docker is good, but because OS APIs are retarded.

Every application install should be a union fs. This union fs should be entirely user-mode, so that if an application has 10,000 files, it doesn't take 10,000 round-trips to the OS kernel with the Intel mitigations, context switches, and cache flushes that all brings with it.

Copying a file shouldn't require a user-mode buffer to feed the data through, forcing it to come down WAN links just to go back up the same WAN link again on the way out.

Overwriting a file shouldn't require more than a single API call, because it's nearly 2020, and we should have long since realised that kernel transitions are expensive, so we should optimise to minimise the number of round-trips. open(), write(), write(), write(), flush(), sync(), close(), poke(), prod(), jesusfuckingchrist(). Just take a buffer bigger than 4KB, or better yet, standardise an API to take a stream from user mode.

Just take a buffer and a filename, and atomically replace. Done. Bang. No lost data, not torn writes, just DO IT. How hard can this be? Is it impossible to do this? Are we forever stuck with POSIX, which was created in 1988, before most of its modern users were born?

But if most people think that this doesn’t apply outside of FS, I have the will to change the title :-)

That's why this idea is popular, and your comment is controversial. You're taking away a crutch that many of our peers cling desperately to as a way to justify their shortcuts and poor decision making.

maybe abstractly

in practice, this never happens because the code will cease to be considered useful first

You mean its assumption that the API wasn't lying about the integrity of the data it claimed to be writing? Very broken indeed.

I'd think "everybody" knows, certainly I would expect Linus to know, about Postel's law and the subtle ways it ends up causing problems. Whenever you make things easy or difficult, you shape the evolution of how people do those things. There's no simple universal answer to "do we make things easy or difficult". Or "do we blame the user or the toolmaker?"

I don't really understand the psychology of going around arguing one side of an insoluble problem, observing that others are totally convinced of the other side, and occasionally flipping sides, but never acknowledging the meta-problem of integrating both or deciding when to apply each.

I think that's why overcommit exists. It's generally not easy to re-design all userland applications to deal with difficult memory management problems in a complex system, but it is easy to just lie to them all so they can continue to do stupid things and not crash. If you have a system with only "100% correctly written" software, just disable overcommit (echo 2 > /proc/sys/vm/overcommit_memory).