Your computer is a distributed system
catern.com
catern.com
Which abstractions are those?
Kernel namespaces are the building blocks for this, because an app that accesses all kernel-managed resources via separate namespaces is insulated from the specifics of any single node, and can thus be transparently migrated elsewhere. It enables the kind of location independence that OP is arguing for here.
Namespaces simply make the kernel lie when asked about sockets and users and such. It’s intended for isolation on a single server. They’re next to useless in distributed work, particularly the kind being discussed here (Plan 9ish). You actually want the opposite: to accomplish that, you want the kernel to lie even harder and make things up in the context of those interfaces, rather than hide things. Namespaces don’t really get you there in their current form.
Isolating processes from the specifics of the system they're running on is a key feature of the namespace-based model; it seems weird to call it a "side effect only". We should keep in mind that CRIU itself is still a fairly new feature that's only entered mainline recently, and the kernel already has plenty of ways to "make up" more virtual resources that are effectively controlled by userspace. While it may be true that these things are largely ad hoc for now, it's not clear that this will be an obstacle in the future,
The thing is when comparing plan9 and linux here, you have to recognize that linux has it backwards. On plan9 namespaces are emergent from the distributed structure of the system. On linux they form useful tools to build a distributed system.
But what's possible on plan9 is possible because it really does do "everything is a file," so your namespace is made up of io devices (files) and you can construct or reconstruct that namespace as you need.
Like, this[1] is a description of how to configure plan9's cpu service so you run programs on another node.
[1] https://9p.io/wiki/plan9/Expanding_your_Grid/index.html
Nothing in there makes any sense from a linux containers perspective. You can't namespace the cpu. You can't namespace the gui terminal. All you can namespace is relatively superficial things, and even then opening up that namespacing to unprivileged users has resulted in several linux CVEs over the last year because it's just not built with the right assumptions.
And besides how would you even implement all those features you listed without recreating k8s? A distributed “ps -A” that just runs “for s in servers; ssh user@$s ps; done” and sorts the output would be trivial, but anything more complex (e.g. keeping at least 5 instances of an app running as machines die) requires distributed and consistent state.
Distributed yes, but not necessarily consistent. You can use CRDTs to manage "partial, flexible" consistency requirements. This might mean, e.g. sometimes having more than 5 instances running, but should come with increased flexibility overall.
In terms of CAP, yeah it might not have been technically as reliable. But there's different levels of reliability for different applications; we could implement a lot of it in userland and tailor as needed
> we could implement a lot of it in userland
Yeah that’s k8s, etcd, ceph, and the distributed database of the week.
It worked great for forking apps. Trouble was hell would freeze over before the patches got merged and most people thought it wouldn't be widely adopted without shared memory and threads.
But the point is, it did run arbitrary apps across distributed nodes, you could see any node's processes and instrument them, you could see the filesystem of any node. This isn't some advanced mystic sorcery, it was there two decades ago. Clearly we could implement these features again in some new way - not as an SSI, but at least allowing an assortment of system-level RPC and some sort of distributed pluggable VFS.
And also my point is: sure, we have all these 3rd party userland solutions, and that is bad. It means nothing is supported until it's been "integrated". It means we have miles and miles of plumbing that schmucks like me are paid to set up before a JavaScript developer can run their piddly web app across 3 nodes. It should just be baked into the OS, batteries included. A lot less annoying bullshit, a lot more standardization, and the ability to get more shit done with less effort. That is the entire point of operating systems, to make it easier to run programs. Not to make it necessary to add 15 million new abstractions before you can run your programs.
Rob Pike:
> This is 2012 and we're still stitching together little microcomputers with HTTPS and ssh and calling it revolutionary. I sorely miss the unified system view of the world we had at Bell Labs, and the way things are going that seems unlikely to come back any time soon.
[0]: https://en.wikipedia.org/wiki/Single_system_image [1]: http://linuxpmi.org [2]: criu.org [3]: https://man.dragonflybsd.org/?command=sys_checkpoint§ion...
Not all units of computation are interchangeable, and a system that recognizes this and doesn't try to shoehorn everything down to the lowest common denominator actually gains some expressive power over a uniform system (else we would not have threads).
No, Plan 9 is not a SSI OS. The idea is all resources are exposed via a single unified file oriented protocol: 9p. All devices are files which means all communication happens over fd's meaning you look at your computer like a patch bay of resources, all communicated with via read() and write(). e.g.:
[physical disk]<-->[kernel: sd(3)]-----< /dev/sdE0/
[audio card] <---->[kernel: audio(3)]--< /dev/audio
[keyboard]-------->[kernel: kbd(3)]----< /dev/kbd
And since files are the interface you can bind over them to replace them with a different service like mixfs(4). /dev/audio is like the old linux oss where only one program could open a sound card at a time. To remedy this on plan 9 you run mixfs which opens /dev/audio and then binds itself over /dev replacing /dev/audio in that namespace with a multiplexed /dev/audio from mixfs. Now you start your window manager and the children programs will see mixfs's /dev/audio instead of the kernel /dev/audio. Your programs can now play audio simultaneously without changing ANYTHING. Now compare that simplicity to the trash fire linux audio has been and continues to be with yet another audio subsystem.
Keyboard keymaps are a filter program sitting between /dev/kbd and your program. All it does is read in key codes and maps key presses according to a key map which is just a file with key->mapping lines. Again, keyboards are files so a user space file server can be a keyboard such as a GUI keyboard that binds itself over /dev/kbd.
Now all those files can be exported or imported to other machines, regardless of CPU architecture.
Unix is an OS built on top of a single machine. Plan 9 is a Unix built on top of a network. It's the closest I can get to computing nirvana where all my resources are available from any machine with simple commands that are part of the base OS which is tiny compared to the rest.
Plan 9 was definitely ahead of its time, but it's also a far cry from the sort of distributed OS we need today. "Everything is a remote posix file" ends up being a really bad abstraction for distributed computing. What people are doing today with warehouse scale clusters indeed has a ton of layers of crap in there, and I think it's obvious to yern for sweeping that away. But there's no chance you could do that with P9 as it was designed.
The abstraction is sound. We ended up with TCP and HTTP instead of IL and 9P (and at scale, URLs instead of file descriptors), because of trust issues, but that's not surprising. Ultimately the interface of read/write sits squarely in the middle of all of them, and most others. To build a distributed system with different primitives at the core, for example, send/receive, requires creating significantly stronger constraints on usage and implementation environments. People do that all the time, but in practice they do so by building atop the file interface model. That's what makes the "everything is a file" model so powerful--it's an interoperability sweet spot; an axis around which you can expect most large-scale architectures to revolve around at their core, even if the read/write abstraction isn't visible at the point users (e.g. application developers) interact with the architecture.
Colossus and Spanner are both proprietary so there's very limited info on them, but both seem to be built for very specialized goals and constraints. So, not really on the same level as a general system interface like 9P, which is most readily comparable to, e.g. HTTP. In Plan 9, 9P servers are routunely used to locally wrap connections to such exotic systems. You can even require the file system interface locally exposed by 9P to be endowed with extra semantics, e.g. via special messages written to a 'control' file. So any level of compatibility or lack thereof with simple *nix bytestreams can be supported.
In case of any confusion, that sort of thing wasn't a generic Beowulf feature, but it sounds like Bproc. I don't know if it's still used. (The Sourceforge version is ancient.)
https://updates.penguincomputing.com/clusterware/6/docs/clus... https://sourceforge.net/projects/bproc/
Containers actually only make it harder to "orchestrate" your distributed processes in an HPC system.
The entire point of UNIX philosophy (Which seems to be something they aren’t teaching in software development these days) is to do one thing and do it well. We don’t need Linux operating operating as a big declarative distributed system with a distributed scheduling systems and a million half-baked APIs to interact with it, the way K8s works. If you want that you should build something to your specific requirements, not shove more things into the kernel.
> “A program is generally exponentially complicated by the number of notions that it invents for itself. To reduce this complication to a minimum, you have to make the number of notions zero or one, which are two numbers that can be raised to any power without disturbing this concept. Since you cannot achieve much with zero notions, it is my belief that you should base systems on a single notion.” - Ken Thompson
It's binary blob design is no good for security, as opposed to a byte code design like Forth. Its user security model was poor and doesn't help with modern devices like phones. Its multiprocess model was ham fisted into a multithreading model to compete with windows NT. Its asynchronous i/o model has always been a train wreck even compared to NT. Its design creates performance issues, especially in multiproc networking code with needless amount of memcopys. Now folks are rewriting the networking stack in user space. Its software abstraction layer was some simple scheme from the 70s which has fragmented into crazy number of implementations now. Open source developers still complain about how much easier it is to build a package for windows, as opposed to linux. It was never meant to be a distributed system either. Modern enterprise compute cannot scale by treating and managing each individual VM as it's own thing with clusters held together by some sysadmins batch scripts.
And that's just a single system call, albeit a very important one. People simply vastly overestimate how rose-tinted their glasses are and all the devils in the details, until they actually get into the nitty gritty of it all.
https://www.microsoft.com/en-us/research/uploads/prod/2019/0...
Ah, a bit meh. Haiku OS/Be did similar queries with BFS and yet it can be much faster on SSD's.
Ok, no proper permissions/ACL's, but NTFS it's on par on EXT3 performance with some additions.
It needs two news FS', one for desktops and another one for the enteprise. Linux' ones should be F2FS for flash media and BcacheFS for the professional storage needs.
I am also not sure why you insist on arguing about a topic you are not familiar with at all.
> Which seems to be something they aren’t teaching in software development these days
This is funny. Perhaps the thing you should have been taught instead is history, my friend.
[0] https://www.qnx.com/developers/docs/7.0.0///index.html#com.q...
[1] https://recon.cx/2018/brussels/resources/slides/RECON-BRX-20...
"LAN use" I think would qualify roughly 95% of the need for a "distributed OS," including a lot of usage of K8s, frankly. Systems with WAN latency impose a different set of challenges for efficient comms at the OS layer. But even then you also have to design your apps themselves to handle WAN-scale latencies, failover, etc too. So it isn't like QNX is going to make your single-executable app magic or whatever bullshit. But it exposes a set of primitives that are much more tightly woven into the core system design and much more flexible for IPC. Which is what a distributed system is; a large chattery IPC system.
The RECON PDF is a very good illustration of where such a design needs to go, though. It doesn't surprise me QNX is simply behind modern OS's exploit mitigations. But on top of that, a modern take on this would have to blend in a better security model. You'd really just need to throw out the whole UNIX permission model frankly, it's simply terrible as far as modern security design is concerned. QNet would obviously have to change as well. You'd at minimum want something like a capability-based RPC layer I'd think. Every "application server" is like an addressable object you can refer to, invoke methods on, etc. (Cap'n Proto is a good way to get a "feel" for this kind of object-based server design without abandoning Linux, if you use its RPC layer.)
I desperately wish someone would reinvent QNX but with all the nice trappings and avoiding the missteps we've accumulated over the past 10 to 15 years. Alas, it's much more profitable to simply re-invent its features poorly every couple of years and sell that instead.
This overview of the QNX architecture (from 1992!) is one of my favorite papers for its simplicity and straightforward prose. Worth a read for anyone who like OS design.
https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.p...
And the unix command-pipe philosophy is realized much better as ordinary functions in a functional programming language.
However, you don't ignore the distributed nature of even single HPC nodes unless you want to risk perhaps an order of magnitude performance loss. SMP these days doesn't stand for Symmetric Multi-Processing.
* a metrics dashaboard that gives you a ps -A for all our nodes
* intermachine pubsub without having to setup a third party message queue
* auto failover for all our microservices
* spinning up a microservice takes about as much effort as adding a controller in rails
* cronjobs where one node can trigger a job on another node in the network. Hell, we have crons scheduled where we don't even know which machine it'll run on. it just gets done
This is a neat line of thought, but I don't think it can go very far. There is a huge difference in reliability and predictability between small-scale and large-scale systems. One way to see this is to look at power supplies. Two ICs on the same board can be running off of the same 3.3V supply, and will almost certainly have a single upstream AC connection to the mains. When thinking about communications between the ICs, you don't have to consider power failure because a power failure will take down both ICs. Compare this to a WiFi network where two devices could be on separate parts of the power grid!
Other kinds of failures are rare enough to be ignored completely for most applications. An Ethernet cable can be unplugged. A PCB trace can't.
I used to work with a low-level digital communication protocol called I²C. It's designed for communication between two chips on the same board. There is no defined timeout for communication. A single malfunctioning slave device can hang the entire bus. According to the official protocol spec, the recommended way of dealing with this is to reset every device on the bus (which may mean resetting the entire board). If a hardware reset is not available, the recommendation is to power-cycle the system! [1]
Now I²C is a particularly sloppy protocol, and higher-level versions (SMBus and PMBus) do fix these problems, so this is a bit of an extreme example. But the fact that I²C is still commonly used today shows how reliable a small-scale electronic system can be. Even at the PC level, low-level hardware faults are rare enough that they're often indicated only by weird behavior ("My system hangs when the GPU gets hot"), and the solution is often for the user to guess which component is broken and replace it.
[1] Section 3.1.16 of https://www.nxp.com/docs/en/user-guide/UM10204.pdf
Sure, but physical damage can disconnect it.
Whereas in a distributed system, a single broken communication line, even or especially if it's extremely important, still means that the distributed system has to recover somehow, preferably gracefully.
Turning on a computer, compare the probability of an issue with the ethernet cable, ie not plugged in as suggested vs a pcb trace being broken. Given that probability do you want to handle one of those gracefully and not fry the machine? Both? Neither. And that's the point of identifying the difference. A thing that is rare and if it occurs everything is b0rked anyway doesn't need to be handled.
That individual components having to be robust on some level (what level? It depends...) to failure difference of distribution via a network connection and pcb traces is probably the reason we don't usually (but we might) refer to the latter as a distributed system.
I know this pain as an embedded software monkey. So many edge cases unthought of and no method to gracefully fail. Not to mention you always find some fault with the I²C implementation on the SOC or slave device you want to talk to.
I’ve spent weeks debugging bus lockup problems only to find erratas for the silicon, or find this particular chip needs an extra 100ms to startup than others.
It’s the one communication protocol that o always have problems with. Nothing else has caused me as many headaches.
I’m not familiar with Oxide, but the I2C physical layer is just a pair of open-drain signals, so I wouldn’t worry about a level-shifter too much.
[0] https://github.com/oxidecomputer/hubris/blob/master/app/giml...
The links between the components of your computer are solid and cannot fail like actual computer network connections.
In terms of "CAP" theorom, the system has no Partition tolerance. If one of the the links connecting CPUs/GPUs/RAM breaks, all hell breaks loose. If a single instruction is not processed correctly, all hell might break loose.
So I find the analogy misleading.
> The links between the components of your computer are solid and cannot fail like actual computer network connections.
I've personally had this disproven to me on multiple occasions.
That sounds like interesting stories! Can you elaborate?
Multiple bad disk cables (more common in IDE era, but happened once with SATA). Interestingly enough, Windows would reduce the drive speed on certain errors, so I had a drive that booted up in UDMA/133 and the longer it was running the slower it got, eventually settling in at PIO mode 2. Switching the drive cable fixed it.
A sound card that wasn't screwed in to the case, so if you pushed the phone connector in too hard it would unseat. I still don't know how that happened; it must have been me (unless someone pranked me) but the sound-card hadn't been changed in like 2 years at that point.
A DIMM wasn't fully clipped in, but the system worked fine for weeks until someone bumped into the case.
Things that were actually intentional:
We expect anything plugged in externally (e.g. USB, ethernet, HDMI) to be plugged and unplugged without needing to restart the system. This sounds banal, but wasn't always the case. I had a network card with 3 interfaces (10BASE5 AUI, 10BASE2 BNC, 10BASE-T modular plug) and you needed to power off the system and toggle a DIP switch to change which was in use.
I've seen server and minicomputer hardware with hotpluggable CPUs and RAM
Eurocard type systems (e.g. VME, cPCI) could connect all sorts of things, and could run without restarting. This sort of blurs the line as to what a "node" is. If you have multiple CPUs on the same PCI bus, is that one node or many?
eGPUs have made hotplugging a GPU something that anyone might do today. If you run this setup, then the majority of the computational power in your system can appear and disappear at will, along with multiple GB of RAM.
The problem is linux's monolithic model, doesn't work well for kernel checkpoint/restore despite it actually supporting hotplug cpu/ram it they have to be gracefully removed.
So, this is less about the machine being distributed, and more about the fact that linux is the opposite of a microkernel/etc that can isolate and restart its subsystems in the face of failure. Its also sorta funny that while these types of operations tend to need to be designed into the system, the last major OS's designed this way were done in the 1980's.
I agree with parent. The major reason why programming on a single computer is easier than a distributed system is that we assume total resilience of various components that we cannot for a distributed system.
From the article:
> This offers hope that it is possible to some day abstract away the distributed nature of larger-scale systems.
To do this is not a question of software abstractions, but hardware resilience. If we have a network which we can reasonably assume to have 100% uptime and absolutely no corruption between all its components then we can program distributed systems as single computers.
So it shouldn't be surprising that computers have the same lack of resilience.
edit: Googling yields few results that aren't actual books, Try this
https://books.google.com/books?id=wBuy0oLXEuQC&pg=PA218&lpg=...
That's incorrect.
There are plenty of machines/OSs which are (or can be) resilient to a CPU failing; Linux, for example. From the OS point of view, you just kill the process that was running on the CPU at the time and move on.
Resilience to spontaneous RAM failures is rarer but possible.
Which would be overkill on a single node, given that CPUs don't really fail all that often.
Attempting to account for such environments in user programs massively inflates their complexity, does little to enhance reliability, and the resulting behavior is typically brittle or outright broken from the get go.
This is why, for example, the C++ committee flirts with making allocation failure a UB condition.
A hydrocephalic demo!
But I'd be totally in to a 10% increase in IQ in exchange to being able to eat 10% more sugar.
Two flipflops interconnected on one wafer. Two flipflops inteconnected on one PCB. Two flipflops interconnected with a cable between two PCBs. These are all "distributed". They're all subject e.g. to the CAP theorem. Sure, the probability of one flipflop failing on the same wafer is quite small. The probability of one flipflop failing on one PCB is slightly larger. But fundamentally all these systems are the same. If you have two computers on a network you can make the probability of failure (e.g. of the network) pretty small.
distribution of signals within smaller systems (microcontrollers, ASICs, FPGAs, etc) are all distributed systems. Ask anyone doing any kind of circuit design about distributing clocks and clock skew, etc.
Highly recommend the post if you're into this and also sort of amazing how far single systems have come. You can basically do "big data" type things on this single box.
[1] https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...
IMO there are two things that make the current abstraction of a computer as a unit make sense:
- You (mostly) don't have to try to handle partial failures within a computer. Partial failures are what make distributed systems hard.
- The difference in communication costs between two cores in a single machine is several orders of magnitude lower than communicating with a separate machine using commodity technologies. So while yes, "it's all just distributed" and you can use a common abstraction, a large enough constant factor difference means that you still will have to look through the abstraction to build a performant system.
E.g. you have multiple CPU cachelines, caching different values of a main memory location. And there are different cache coherence protocols to keep them sane. But cache coherence protocols never need to worry about the failure mode when one cacheline is temporarily unavailable but the others are.
So yes, there's a distributed system in each multi-core computer, but it's a distributed system with an easier failure mode.
If you like more analogies between CPU caches and distributed systems, https://blog.the-pans.com/cpp-memory-model-as-a-distributed-... :p
As this link points out, it gets a bit more difficult with some of the larger machines we have to keep the abstractions useful. That said, it does mostly work. Despite being able to find and harp on the areas that it fails, it is amazing how well so many of the abstractions have held up.
Would be neat to see explicit handling of what features are basically completely hiding distributed nature of the computer.
But in the abstract, sure. It's a give and take. It's useful to know things and use that knowledge. It's also useful to know a detail is hidden and changeable without consequence.
A similar situation happens if my NIC/kernel buffers are to overloaded to send the packets I need out. Instead I can try in vain to push packets out and have almost no understanding how many packets the OS is dropping just to keep up. Media standards like RTCP were designed around scenarios like these, but that itself is complexity we wouldn't need if the OS could notify the application when their packet writes failed.
This kind of flexibility right now is really difficult because most OSs try to pretend as hard as possible that everything happens sequentially. This is just about opening up more complete abstractions to the programmer.
That said, there are times when it isn't hidden, but only taken out of your control. I guess the question is mainly in how to move them to first class objects to reason about?
Bluebottle active objects https://www.research-collection.ethz.ch/bitstream/handle/20.... with some discussion of DMA
Composita components http://concurrency.ch/Content/publications/Blaeser_Component...
Mobile Maude (only a spec) http://maude.sip.ucm.es/mobilemaude/mobile-maude.maude
Kali scheme (atop Scheme48 secure capability OS) https://dl.acm.org/doi/pdf/10.1145/213978.213986
Kali is probably the closest to a distributed OS, supporting secure thread and process migration across local and remote systems (and makes that explicit), distributed profiling and monitoring tools, etc. It is basically an OS based on the actor model. It doesn't scale massively as routing nodes was out of scope (it connects all nodes on a bus), but that can easily be added.
Extremely small (running in 2mb ram), it covers all of R5rs, and the VM has been adapted to bare metal.
I feel that there is more to do, but a combination of those is probably the right direction.
https://en.wikipedia.org/wiki/Sprite_(operating_system) was a project at UC Berkeley that ran building-scale networks of workstations as a distributed OS. It ended in 1992 but produced innovations such as the log-structured filesystem along the way.
Various commercial products like Domain/OS or SGI Irix also were distributed operating systems at different scales. It seems like these lost out to the more typical HPC solutions with distributed/parallel OS instances limited to each node. Or on the other extreme, mainframe systems continue to be a kind of distributed OS built for high availability and scaling in a very controlled environment.
What happens is enough people try to do something and can't quite get it to work quite right that it eventually becomes assumed that anyone trying that approach is naive. Then people actively avoid trying because they don't want others to think they don't know "best practices".
Remember the post from the other day about magnetic amplifiers? Engineers in the US gave up on them. But for the Russians, mag amps never became "unworkable" and uncool to try, and they eventually solved the hard problems and made them extremely useful.
Technology is much more about trends and psychology than people realize. In some ways, so is the whole world. It seems to me that at some level, most humans never _really_ progress beyond middle-school level.
The starting point for analyzing most things should probably be from the context of teenage primates.
That's not even remotely unique.
OP is grappling with "the map is not the territory" vs. maps have many valid uses.
Abstractions can be both not accurate in every context and 100% useful in many, many common contexts.
Also (before you get too excited), abstractions have quality: there are good abstractions -- which are useful in many common contexts -- and bad abstractions -- which overpromise and turn out to be misleading in some or many common contexts.
I'll put it this way: the idea that The Truth exists is a rough (and not particularly useful) abstraction. If you have a problem with that, it just means you have something to learn to engage reality more fruitfully.
Because, user level programs are all at one level of abstraction, and this distribution is distributed over many levels of abstraction.
So in desktop systems, mean mostly successors of business micro machines, access to other levels of abstraction intentionally hardened for measures of security and reliability. The same thing applied to crowd computing - there also vps's are isolated from hardware and from other vps's.
These measures usually avoided in game systems and in embedded systems, but they are not allowed to run multiple programs from independent developers (for security and reliability), and their programming magnitudes more expensive than desktops and even server side (yes, you may surprised, but game consoles software in many case more reliable than military, and usually far surpass business software).
To solve this contradiction, need some totally new paradigms and technologies, may be some revolutionary, like usage of GAI to write code.
And Erlang is based on relatively new syntax from Prolog, which also have cool ideas.
https://codahale.com/you-cant-sacrifice-partition-tolerance/
L1-blockchain entrepreneurs and people who got locked into MongoDB aside, I think most agree.
At the moment there are lot of such devices - exists for sure many full featured, like Raspberry; but also there are network connected ATA drives, network connected sensors, RAM, ROM (Flash); BTW IEEE 1394 FireWire is serial interface, could been used as networking bus; exists adapters ethernet-usb (and many commodity devices work well with such connection), so virtually anywhere could been considered as connected via network bus. Even exists USB 3.0 to PCIe adapter, to use PCIe device throw USB connection.
And in reality exists problem, that FireWire so distributed, that it where possible on Macs with FireWire interface, to read memory via this interface.
So hardware and software exists, but need some steps to make it's usage safe.