In Praise of Plan 9
drewdevault.com
drewdevault.com
However if it had become mainstream, it would be just as cluttered up with inelegant stuff as Linux is now, in the endless pursuit of, say, graphics bandwidth performance, first for videos, then for 3D immersive games and now to merely draw your desktop. Mount an audio device remotely. Lovely. But try to get that working with Bluetooth, something mainstream desktop Linux has only just recently managed (i.e. use bluetooth headsets reliably and without fuss).
Ditto for 10GB ethernet or what have you. Elegance is quickly sacrificed at the altar of efficiency and expediency. At the risk of inciting disagreement, look at what happened to the originally relatively elegant and simple X protocol.
The fact that the lowest level of interaction you can have with a computer is writing memory. When writing device drivers, or interfacing with microcontroller hardware, the way you give commands and transfer data is by writing your command to a memory-mapped device register, or an area of memory that will be copied to the target device. This, coupled with the ability to map a file to memory, allows this paradigm to compete with the most efficient bare-metal implementations. One particular example is the /dev/draw interface mentioned in the article, where anyone could literally write graphics program just by fiddling with bits in memory, just like in DOS or the C64.
The other thing is, modern computers are networks in a box. For example, what if your GPU is a separate computer networked over a high-speed link. What if I could upload a texture just by opening a 'file' on the GPU, mmaping -it and memcpy-ing the data into it? I don't see anything particularly inefficient here, particularly if the data transfer can be handle by a special network protocol that takes advantage of PCIExpress.
AIUI, you need something like CXL to implement a proper concurrency model that works seamlessly for both local and remote memory. So we're kinda close to what you're describing but not quite there yet.
CXL 2.0 and earlier just aren’t it because of a coherency strategy that is absolutely dreadful. CXL 3.0 is better though, but that’s a long time away from hardware still.
While this is true as far as it goes, it's worth keeping in mind that not all memory controllers are equally low-level. Some of them expose cache architecture or the rectangular organization of DRAM in interesting ways.
This is somewhat true, but usually these devices are not designed for multiple processes to come in and start writing memory to them. On Linux, the job of the DRM kernel driver for graphics is to mostly facilitate command buffer submission, scheduling, and results gathering. We'd really just see the same exact situation on Plan 9: user-space GPU drivers which do the work of translating graphics APIs into command buffers, and talking to a kernel-side component by submitting dedicated command buffers.
That they talk over a file interface isn't really relevant here, since it's just used as userspace -> kernel IPC.
> The other thing is, modern computers are networks in a box. For example, what if your GPU is a separate computer networked over a high-speed link. What if I could upload a texture just by opening a 'file' on the GPU, mmaping -it and memcpy-ing the data into it?
Well, first, you'd need to figure out the layout of the texture you want (e.g. AMD has these layouts https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/amd... ), allocate the full mip chain size, allocate your texture descriptors, then swizzle the texture data into a form that the GPU can understand.
Oh, and if you wanted a fast texture on desktop GPUs, you'd have to store it in host-inaccessible memory (host-accessible memory is often slower). So first you'd need to allocate a scratch buffer in slower, host-accessible memory, copy your texture data there, and then allocate the real texture storage in the faster host-inaccessible memory, and submit a copy command. That's a lot more work than a memmap and a memcpy.
Maybe we put all those smarts in the driver, but the driver is going to need a lot more info about the intended usage of the texture, and by that point we're basically creating /dev/vulkan rather than a low-level GPU API.
> usually these devices are not designed for multiple processes to come in and start writing memory to them.
Multiplexing devices as needed can also be done in userspace. That's what, e.g. pulseaudio and pipewire do for audio devices, even those that don't support outputing audio for multiple processes at the same time.
You can use virtual functions for this purpose, but you only get access for a dozen-ish procceses. Also, GPUs have support for this already.
Of course, virtual machine managers and hypervisors also provide good solutions to this problem.
With VFIO, it is at least an order of magnitude easier to get to 90-95% of what the underlying hardware can do.
I use linux on my desktop but have an ancient Thinkpad that runs OpenBSD as well. I love following the changelog for each new release.
I've considered a combination setup of a terminal system for my unix needs and a tablet for web and video. ChromeOS would work well for this too, I think.
This is a community and distro problem. If you accept patches for the sake of adding stuff with no good reason then you're going to get linux. For example just the other day on the 9front mailing list someone submitted a patch for an ethernet driver they were working on to enable tcp/udp checksum offloading. Another user replied along the lines of "do we really need this? first thing I do as an admin is turn off weird/buggy offloading" which the submitter agreed with and that was that. No one really needs that patch; its code for ONE specific ethernet device few if any other users own.
The community strives to keep the OS small and practical as it was intended. That's a big part of the culture behind plan 9: Keep it simple, stupid!
Another thing is porting software for the sake of porting software is also discouraged. If you need it, then you port it and maintain it. Dont port things you don't need.
> But try to get that working with Bluetooth, something mainstream desktop Linux has only just recently managed (i.e. use bluetooth headsets reliably and without fuss).
Not really a fair statement. BT is a horror show with its batshit layercake of drivers, protocols and profiles. Plus the Linux audio situation has never stabilized and invents a new audio subsystem every decade: OSS, ALSA, Jack, pulse, pipewire, etc. The BT situation on Linux is the fault of the constantly moving linux dev community who's focus seems to favor server side stuff and DE colors.
Linux kernel is used on billions of devices by millions of people. If you don't see a good reason for some feature in Linux, it doesn't mean there is no such reason.
I'd even argue it is impossible to have something small, simple and practical, but simultaneously supporting such a vast sea of hardware and software combinations.
This was not so previously. I tried every release or two, and always ended up with a locked-up driver within 10 minutes of playing with it. This is totally Bluetooth's complexity's fault of course.
That’s... true, but what “file” means in that sentence is a bit tricky. The graphics protocol (or at least I think it was the graphics protocol) requires each command to be written in a single write() call, so a Plan 9 file is neither an array nor a stream of bytes, it includes those implicit boundaries as well. The interface used for impersonating users, IIRC, looks like a kernel-implemented file server but that file server essentially uses its kernel nature by referencing the process that opened the file, so this interface only deserves being called a file if /dev/stdin and /proc/self in classic Unix do as well.
I like Plan 9, mind you, but I also think we ought to be careful in treating it as an existence proof for what the pure everything-as-a-file model can do. Even outside of things Plan 9 doesn’t and can’t implement (e.g. modern bandwidth-limited 3D graphics), it also has some hacks in parts it does.
I'm sure you can encode command separators in binary without the need to rely on write flushes.
> https://marc.info/?a=111558719100068&r=1&w=4
> "Computers should feel instant whenever possible. This involves the event path, whatever processing is done, the speed of drawing, and the way the drawing is displayed."
This was 27 years ago and still no progress on this field.
By the late 1990's Linux supported tear free with lower latency than compositors currently support.
Anyway, computers have been getting worse at output latency since the mid 1980's. Compositing is just yet another ratcheted performance and usability regression.
Even then, there are better and worse ways of coping with limited video memory bandwidth, and what's "best" depends on the given application.
This matches my memory of configuring tear free before y2k.
I know for sure that I had tear free working at 1600x1200x24 (or 16?) by 2001.
I don’t think it knows anything about the widgets inside those windows, either (again unlike classic X and like Wayland). (On the other hand, like classic X and unlike Wayland, it makes the client funnel every drawing operation through the window system.)
Memory, power and compute efficiency have also regressed badly vs. those systems. Carmack explains how if would be possible to have tear free, framerate level updates "if only" people would upgrade to 8MB of video ram and 100MB/sec busses.
I'm old enough to remember the DOS days. On average, iOS apps are significantly less responsive than old 386 DOS, SDL or X11 programs.
With 1280x1024x8@75Hz and a single blit per frame. For 32-bit colour like you have virtually everywhere these days, multiply that bandwidth by four; for a 2560x1600 screen like the one I’m writing this on, multiply further by three and a bit; (admittedly,) for 60Hz instead of 75Hz, multiply by 4/5.
All in all, this laptop I’m sitting before needs a gigabyte per second to just barely sustain one blit per frame. And it needs a full framebuffer of video RAM (15M bytes) for the pretty security-message crossfade, probably at least a couple framebuffers per tab for the tiled renderer in the browser, something in that ballpark so that I can smoothly scroll through a PDF I’m reading and not wait for it to rasterize... If I open a Chinese webpage, how much video RAM for the subpixel-positioned glyph cache, I wonder? And remember, none of that OpenGL floating-point-colour nonsense, that would multiply both memory and bandwidth by three again.
Carmack could have probably squeezed a smooth UI into a 8M main + 8M (16M?) video RAM, 100 MB/s machine at 800x600, if the UI you wanted was the one from 1995. And that would honestly be grand, I’d be happy to see that on a 2005-specced machine, even. But let’s not kid ourselves—we do in fact demand much more from our computers now than we did then.
There are lots of other factors, for sure, but that's a pretty big one!
(I still don’t remember whether I looked at one of the /dev/draw implementations or at one of the 9p implementations when I was curious about the fiddly buffering question, sorry.)
Sort of like how we now have USB-C everywhere, but you have to know what kind of cable is actually in the middle in many cases. Maybe it would better to split major feature sets across a few different physical plugs, so you know exactly what you have just by looking at it.
Inferno and Limbo, which tend to be ignored with too much focus on Plan 9, a middle stop in their whole experience designing OSes and programming languages after being done with UNIX and C.
Was there any reason for that? Were the CPUs not powerful enough, or were the compilers not quite there yet? Or did it have more to do with the business side of things?
Perfect is the enemy of good, and good enough is the perfect enemy of better.
Unix/Linux were/are good enough for vast majority of world's needs. So anything better is not worth the hassle.
There have been better ideas since then (including Plan 9) but nothing able to knock *n*x off its perch.
IMO, LMI and Symbolics might have done it, if their offering hadn't required enormously expensive machines (by the standards of the time), while *n*x would run on cheap hardware (again, by the standards of the time).
Perhaps we'll eventually see an innovative OS written in WASM. That's about the only way I see to get around the vendor lock-in (I mean, even Microsoft appears to be converging on a "Windows UI wrapped around a *n*x kernel" model... Apple, of course, has been using a "Mac UI wrapped around a *n*x kernel" for a couple of decades now).
For anyone who hasn't read it, I recommend Richard Gabriel's "Worse is Better" essay: https://dreamsongs.com/WorseIsBetter.html
(original essay and several followups at this link)
On "Cloud OS" the underlying kernel only matters for classical workloads being pushed onto the cloud.
Any language that doesn't depend on POSIX and has a rich library ecosystem, can happily run on top of type 1 hypervisors, which are in a way, the revenge of microkernels.
That's interesting. Can you run for example Golang programs on ESXI? Go isn't exactly POSIX-free but it also doesn't need all of it.
https://github.com/solo-io/unik
https://github.com/icexin/eggos
"Pure Go Unikernels - go metal with TamaGo"
My main side project at the moment involves trying to use QEMU+WHPX on Windows to sandbox selfhosted services. Long term I think it would be awesome to support unikernels, but I get the feeling a lot of the momentum for them has died down the last couple years.
Also, from parent comment, ops, can deploy to esxi/vsphere perfectly fine.
I hope that doesn't happen. No body wants a Nuclear War and 40 million refugees off the coast of Australia.
For thise wondering about this reference:
https://www.destroyallsoftware.com/talks/the-birth-and-death...
Also note that these ideas are quite old, IBM and Unisys mainframes and micro-computers are quite different from their original versions, yet most applications keep running thanks to their language environments instead of shipping pure native code.
1. They were primarily research OS', Singularity was never even meant to be used at all. Its successor Midori was, but again it never made it outside of MS. JavaOS was sort of targeted at embedded devices but again never had any real usage.
2. Operating systems are ideally very tightly programmed with minimal overhead. Given a choice between elegance and performance, people pick the latter. A slow app can be optimized but if your kernel/drivers are slow then you're often stuck. Whether it's microkernels or operating systems written in managed languages, these better designs sell performance to buy convenience for the implementors, but convenience for users is more important.
3. Midori tried to fix this but ended up spending all its design budget on trying to turn C# into C++ and do fashionable things with concurrency/thread safety, which isn't a compelling basis for an OS. Plan9 at least had features that were interesting for the actual users of the computer, but the Singularity/Midori approach was mostly just a PL nerdout.
Single address space operating systems have some design issues that have never been well addressed, or at least the solutions didn't seem convincing. Unfortunately OS research is stagnant so very little has changed over time.
For example, they need everything to be written in a single (GCd/managed) language, and so existing C/C++ libraries can't be used (without a win3.1 style stability model at least). But the VM itself is written in C/C++, so the first step is to either switch the entire model to AOT compilation (the MS approach) or to write the language VM in the language itself (the Sun approach, which eventually surfaced in real products in the form of GraalVM). The Graal approach seems to solve this problem by being able to run any kind of language on top fot the JVM, even C/C++ code, and it can run it safely/GCd.
Another problem is the question of what exactly replaces a process. Processes in classical operating systems do a lot of different things - sandboxing, data isolation, scheduling, but also things like defining a failure domain. The big advantage of a single address space OS is you can allocate objects and then just pass them around freely without needing all the IPC/serialization/socket bindings/etc stuff that classical kernels require. So the obvious approach is to just not have such a thing as a process - but if you do that, you hit the question of how to manage executing code. What's the equivalent of SIGKILL if you don't have a process? What does a CPU usage graph look like? What happens if a driver does a callback into a piece of code which then deadlocks? This problem combined with the difficulty of finding a one-size-fits-all garbage collection algorithm pushes SAS operating systems towards reintroducing a process-like concept, for instance the Singularity process equivalent was actually way more constraining than a UNIX process, and objects could no longer be passed around freely between subsystems but instead required a convoluted exchange heap + quasi-RPC approach.
Java and Inferno were designed to fix the abysmal portability problems other operating system vendors created.
"Alef appeared in the first and second editions of Plan 9, but was abandoned during development of the third edition.[1][2] Rob Pike later explained Alef's demise by pointing to its lack of automatic memory management, despite Pike's and other people's urging Winterbottom to add garbage collection to the language;[3] also, in a February 2000 slideshow, Pike noted: "…although Alef was a fruitful language, it proved too difficult to maintain a variant language across multiple architectures, so we took what we learned from it and built the thread library for C."[4]
Alef was superseded by two programming environments. The Limbo programming language can be considered a direct successor of Alef"
Drew, who do you think is reading your blog if not the sort of people who know what Plan 9 is :p
Yeah, plumber is way better than xdg-open in that it is extensible and it can be invoked from a script more easily via 9P. Given how hard it to customize xdg-open, I use a xdg-open wrapper that gets the caller's name using procfs and sends it to the plumber service. That way I can open links on a different browser depending from where the link was clicked/opened from .
#!/usr/bin/env bash
# this is named xdg-open and placed in a directory that is
# before /usr/bin in the $PATH
PARENT_COMMAND=$(cat "/proc/$PPID/comm")
case "$PARENT_COMMAND" in
slack)
/home/puercopop/src/plan9/bin/9 plumb -s slack "$@"
;;
zoom)
/home/puercopop/src/plan9/bin/9 plumb -s zoom "$@"
;;
*)
/usr/bin/xdg-open "$@"
;;
esac
type is text
src is zoom
data matches 'https?://.*'
plumb to firefox-trunk $data
type is text
plumb to xdg-open $dataThis feels so ham fisted to me. Why not just open /net/tcp/127.0.0.1/80? I am sure there are reasons, but if the goal is to make everything a file, this feels like a more natural representation.
Sure, you could script opening each address of the entire address space and have an unfortunate situation, but you don’t get that by default when you simply query what exists.
That said, it’s been a veeeery long time since I tried out Plan9 and Inferno.
Magic indeed.
Yes, though this could have been achieved with a bunch of sysfs-like entries under /net/tcp/127.0.0.1/80/<port>/ with the added benefit of being easily discoverable.
> It would also be unfortunate to run ls in /net/tcp and dump the entire IPv4/6 space into your terminal.
A reasonable semantics here would be to only list hosts to which there are active connections.
The approach would be Linux-y but probbly not enough according to 9p standards.
They're optimizing for as few syscalls as possible, I suppose, but in Plan 9 you still create pipes with pipe(), not by writing to a special file. And if you already have pipe(), why not socket()?
But the important thing to not is this fixes the biggest issue of modern Linux - the lack of stable API/ABI - that requires everything to be compiled for every distro.
It also naturally documents each program's interface allowing them to be easily rewritten/mocked/logged/debugged etc.
There's no particular reason it must be so. The Java compiler has a --release flag that lets you say "I want to run the results on Java N" and then the compiler won't use any bytecode or library features that were added after that time, even if it otherwise would automatically and implicitly do so. The result is still binary and non-text based, but there's nothing magic about it. That sort of feature simply takes work that the Linux community is culturally opposed to doing thanks to the fully centralized model.
And making things work this way everywhere is a way to achieve network transparency.
So, it's pretty easy to defend if you care about network transparency above the highest possible performance.
The Unix renaissance in the ‘90s had a lot to do with open source source OSes such as (primarily) Linux and the BSDs. Until then we had a bunch of corporate unixes, no two alike but expensive and kernel hacking was basically off limits. With the advent of i386 we had affordable & powerful computers that could run “real” OSes and us hackers were hungry for an open source OS. Computing history might’ve been different if plan9 was made open source before Linux became available. But that was not to be.
The current mainstream platforms are as complicated as they are because of performance.
Abstractions are fine until you realize you're doing thousands of network round-trips to draw a window, so what if you short-circuited that?
Yes the 9P protocol sounds great and cleaner than FUSE but what's the IOPS?
The reason only some of Plan 9s concepts made it elsewhere was because those were the ones that could be implemented without too steep a performance penalty (or where performance didn't matter).
(No excuses for sockets though :p)
In contrast a custom syscall that takes a structure is a more direct interface, more suited for writing actual programs.
It also means the API is much more tightly defined. Everything-is-a-file can create a lot of edge cases just like how HTTP creates a lot of edge cases in web programming by pretending everything is a document. What happens if you delete /dev/draw, what does that mean? You need to define the semantics of that. Does it mean closing the window? What about trying to move or copy it - does that make sense? Do you have time to think about all these operations and give them meaningful results?
Once you go down the road of saying that there should be a consistent set of operations you can perform on conceptual 'objects' using a generic set of tools and commands, you may start to wonder why you'd pick a file API for that. Why not just invest in objects as a core tech - why not allow binding directly to an object in a remote process or over the network and then have methods and functions actually work? Why pretend it's a file and force everyone to constantly invent mini-protocols and formats, when types and functions are what you need 90% of the time anyway? That's why Microsoft ended up going down the DCOM path, why Apple ended up with XPC/Mach, why Android/BeOS ended up with the Binder and so on. The Plan9 approach wasn't taken up in any big way because if you're going to make a big effort to unify everything it might as well be around objects, not files.
1. OOP can at least in principle abstract over whether code is in-process or in a remote process, with efficient stack based calls for in-process calls and serialization/RPC for remote calls. Yes you can't make them be identical, but you can bring them very close together. But files are fundamentally a kernel controlled object. For a pseudo-file system to hold together at all, you end up having to talk to the kernel all the time, which is slow (ish).
2. OOP has the notion of a queryable set of interfaces. UNIX style file objects don't have any equivalent, which is why you suggest that operations that don't seem to make sense should just return a generic error code. But this is bad. If we saw a junior programmer both design and then implement an interface in which most methods returned error codes we would school them (it can of course be acceptable if you don't control that interface but even then, really not ideal). A much better approach is to define interfaces better so that your objects only implement operations that make sense, and a generic client can use type casting / IUnknown::QueryInterface style operations to figure out what an object can do.
3. Just as you can't opt-out of generic file operations, you also can't add more. That's why you end up having this split world in which some toy operations are just basic file IO and then the moment you need anything complex enough for the real world, your file becomes just a sockety thing speaking some ad-hoc protocol and you can't use the file directly anymore, e.g. you can't shell script it, you need some ad-hoc library that wraps it. There are tons of files in /dev but how often do you see code that directly uses them? Almost never - apps use libraries that wrap the file interface.
4. The lack of proper interfaces and type safety means that evolving stuff is way too hard. Doesn't matter for a research OS but matters in reality. Look at the docs for /dev/draw, it's all ad-hoc protocols with no proper versioning or evolvability at all, just stuff like "open the device and read 12 strings each 11 characters wide" (!).
I think that even though they suffer from execution issues, Microsoft was heading down the right path with COM and PowerShell. COM provides you with objects, which is way more commonly what you actually want. DCOM attempts to abstract over location, so the serialization and sockety/filey stuff only gets involved when necessary. And PowerShell attempts to give you a shell-like syntax for accessing them. For various reasons it doesn't quite work as well as it could, hence why concepts like Plan9 have enduring fascination, but the core ideas are more general and robust.
I'm still unconvinced by "everything is a file". Writing `connect 1.1.1.1!80` to a file is a particularly shitty completely untyped, unchecked, fragile and slow alternative to an actual ABI.
It's obviously easier to use from shell scripts but I don't think that's what you should optimise for.
IMO there should be a proper API with a typed IDL and then you can automatically make it easy to access from a shell without compromising other languages.
I think maybe Fuchsia works like this.
The fact that you can paper over a bad interface doesn't mean it isn't a bad interface.
There are ways around this ofc like the Android Runtime that requires any code that interfaces with it to be written in Java adjacent bytecode that gets compiled to machine code (and cached) by the runtime (after being checked I imagine).
Right, mistakes are possible in both cases. But why pick the design that is more likely to lead to mistakes (on both sides of the interface) and is slower? There's just no good reason to do it like that.
"Everything is a file" is a nice idea, but we almost have that in Unix, and it is something that can be emulated perfectly in a programming language or a library. There is no need for the low level stuff to be neat. It has to be performant, secure, and support my hardware. Everything else can - and I think should - be built as abstractions on top. That I can basically run the same applications on macOS, Linux, and Windows shows that it is possible and works well.
One thing where Plan 9 is lacking IMHO is in the GUI department. ACME is novel but where are the really new radical (G)UI concepts? I wonder what we would have got if BeOS or Longhorn would have been successful. Both played with the idea that the filesystem is a database. Your file system browser morphs into a mail client, a MP3 player, a photo browser depending on the circumstances. You don't deal with "video files" anymore but "episodes", for example. And I think it is not really a technical but a UX challenge to make something like that work well. I hope that people start experimenting with that stuff again!
The problem is that those abstractions cannot implement the interfaces that, e.g. native Linux programs will expect to use. Features like FUSE (filesystems implemented in userspace) are useful precisely as means of increasing the level of abstraction.
For example, I have been developing a static site generator where I implement the output folder as an fs.FS[1]. The output generating code is now a simple function that copies from folder A to folder B, without even knowing that A is a virtual filesystem. Now how do I implement a preview server? Simply pass said filesystem to the standard library's http.FileServer. Done. (okay you actually have to pass it through the http.FS() adapter, but that's only because http.FileServer predates io/fs)
Of course this kind of abstraction can be done in any language, but Go explicitly specifies this interface, which can already be used by multiple utilities in the standard library (e.g. http.FileServer, go:embed). This nudges people to the same interoperable interface, and I'm all for it.
[0]: https://www.youtube.com/watch?v=yx7lmuwUNv8 [1]: https://pkg.go.dev/io/fs#FS
FileSystem.Java: 12 abstract methods
FileStore.Java: 10 abstract methods
FileSystemProvider.java: 17 abstract methods
Total: 39 abstract methods
And having read through it, I’m still not sure what the role of these three systems are precisely. And there are more involved classes I ignored
Vs
fs.FS: 1 method
fs.File: 3 methods
fs.FileInfo: 6 methods
Total: 10
That’s 4 times less work for the developer. And 4 times less learning for users.
And it took me less time to look this up, and it’s abundantly clear that there is nothing I am missing here.
Generally in Go interfaces are much simpler than their Java equivalent.
In fact I’ll go further. The Java solution is so complicated that it is basically unused in practice.
Oh boy, it is used, starting by application servers.
It's closer to this. The actual content was already written to an sqlite db, and my read-only FS allows reading that data back out but as a filesystem.
The admittedly not that interesting (and probably bad - I'm new to Go) code is here[1], and this "BlogFS" is used in 2 places:
- During "export" where it generates the static site. "Folder A" is my FS and "Folder B" is the destination i.e. an actual folder on the real filesystem.
- In a preview server that just serves "Folder A" directly.
[1]: https://git.sr.ht/~nhanb/bloghead/tree/40b70cadb01c14f1e3bf5...
So many good ideas in so many areas didn't go off for this reason (some previous design is already too widely adopted)..
The internet existed for most of a decade before Berkeley Sockets. Surely there were other APIs out there. If none of them forced everything to go through a filesystem, maybe that means it's just a bad fit?