Inferno-OS: distributed operating system where everything is a file
github.com
github.com
In the 90s our sector started to look (again) at VMs as viable alternatives for software platforms. The most prominent example of that is Java. But Inferno was born at the same time at the other side of the Atlantic Ocean[0], sharing the same goals: an universal platform intended for set-top boxes, appliances and the like, with the same write once, run everywhere philosophy.
Inferno programs ran on top of a VM called Dis and programs were written in Limbo (designed by Rob Pike). If you use Go nowadays, you're using a descendant of Limbo, as both share a lot of syntax and concepts (and Limbo descended from Aleph, by Rob Pike too, but that's another story)
Going back to 90s. This space was in a very "exploratory" state then, and any technology had to probe it's superiority to gain adoption. For example, Inferno creators implemented Java on Dis[1] simply to demonstrate how fast was Dis vs the JVM.
I don't know exactly why things were the way the went, but I suppose Sun had more deep pockets to push it's technology and Inferno ended as a the rarity we have today.
From a pure tech standpoint, I've studied the Dis VM to write my own (incomplete) interpreter[1] and, IMHO, it's design is far less abstract than the JVM. It seems to be too low level and tied to that era processor design. That make it less future proof (but that's only my own point of view, of course)
[0] https://www.vitanuova.com/inferno/ [1] http://doc.cat-v.org/inferno/java_on_dis/ [2] https://github.com/luismedel/sixthcircle
1. Java,
2. Inferno,
3. Oberon System (Niklaus Wirth, ETH Zurich),
4. Squeak (Apple, Walt Disney Imagineering)
5. Erlang (Ericsson, Joe Armstrong) still going strong.
IMHO, Technologically Java was the worst and least innovative. Platform adaption is "Worse is Better" so Java won.
Oberon, Erlang and Squeak were are all amazing in their own way.
Squeak/Smalltalk had the issue of programs and runtime images being conflated, making version control awkward (at the time at least).
Erlang is a functional language.
All the above had relatively small and poorly documented libraries.
Java combined an imperative C-like and relatively simple language, with an evolvable bytecode-oriented VM, garbage collector, large and well documented libraries + lots of tutorials, and really intense VM engineering. Easy to forget but Cliff Click and colleagues were the first to show that a JIT compiler could produce code competitive with gcc, Sun also bought/implemented stuff like deoptimization which is still relatively advanced, and then gave that tech away for free. From the end user POV all these things mattered a lot.
Innovation loses to taking small steps to improve the worse alternative. Community is built around tinkering and fixing errors, getting around design failures.
Java was a pretty major clean break when it came out. It was probably right at the limit of what could be done innovation-wise whilst still being targeted at the mainstream.
Also agreed: Java and JVM are were not the "worse is better" options at the time of adoption. Hype alone did not create the massive shift to Java that occurred in late 90s. It was a genuinely positive development for the practice and people were getting results. It was Java's 2nd act of moving into "enterprise" (and Sun's mismanagement of that effort) that created its current sense of 'heaviness' of language.
The reason I always hated Java was because it was designed by good programmers for bad programmers in a looking down at them way, not a good programmers for themselves to use.
btw. Java design team licensed the Oberon compiler sources years before to study it.
The code has been available for years before Oak was born, it is even on the book.
Not only were CORBA and DCOM much worse, Java EE was the reboot of Objective-C framework for distributed computing at Sun. And those Objective-C lengthy methods are quite the pleasure to type without code completion.
While they certainly could have done better, it was already an huge improvement.
[1] https://en.wikipedia.org/wiki/Self_(programming_language)
In the words of Guy Steele: And you're right: we were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp. Aren't you happy?
* https://people.csail.mit.edu/gregs/ll1-discuss-archive-html/...
Java has only familiar surface syntax.
Semantics has almost nothing common with C++. Different object model, memory model, execution model, ...
That's the genius of both Java and Javascript. Make it look familiar without being one at all.
Now we are back at it with WebAssembly, as if it was something new.
it's also why WASM is so slow compared to native executables; security guarantees have a performance cost.
https://www.usenix.org/conference/usenixsecurity20/presentat...
https://www.usenix.org/conference/usenixsecurity21/presentat...
More devs should get security trainings.
On this specific case, it is quite a big deal to add write and execute controls to the WASM memory, so it requires further justification than "I can do stack underflow attacks on my C code". Even though "I can do stack underflow attacks on my C code" is relevant information.
my point stands though. WASM is not ActiveX where the whole applet has admin permission on the computer.
In fact, everybody is trying very hard to use all the experience we've got with the 90's VM, and getting annoyed here and there when it's different enough that they can't.
What I enjoyed about the 90's is that there was still plenty of meaningful work implementing published algorithms. I remember for example, coding Delaunay triangulation, our own Quartenions etc in C++. These days you just download and learn an API to do it.
What this means is now you can build comparable stuff much quicker. It's all been implemented and you're mostly gluing libraries.
Gluing stuff is a lot less fun than doing it from scratch and you learn much less but it's the only way to stay competitive.
If you want to build something rapidly to show off though, today it's much easier.
Web development seems to suck more with every year passing. I wasn't all that into it in the 90's as it was very nascent and primarily for text and images due to bandwidth limitations. But VRML was mighty impressive if you used it on a LAN.
Web devs seem to reinvent wheels more than other types of devs. Everything is hailed as a breakthrough in productivity that subsequently fails to materialize. They impress each other with how concise the framework du-jour makes the code, forgetting that troubleshooting ease is much more important.
Troublueshooting stuff was much easier in the 90's even with the simple tools. Not many systems were distributed. It was considered craziness to build distributed software unless you really needed it and had a massive budget.
Everyone builds distributed systems now, whether they need it or not.
AI/ML is amazing today. None of this was possible in the 90's we had nothing approaching the crunching power required to produce decent results. So a lot of ML work was deemed a failure even though it's being exonerated now.
I think with the resurgence of ML the software engineering field is exciting again. Things were rather boring for the last two decades when hipsters kept reinventing mundane stuff only to go back to tried tested and true (vide resurgence of SSR or RDBMS).
Overall, I think the 90's were fun. More fun than what followed. ML/AI might make things fun again.
I really wish more people could see this as you (and I) do.
Web development is really its own horribly sheltered community. they are driven to deliver at faster and faster paces and they can't do anything properly if they even wanted to.
there weren't entire categories of Wikipedia articles about things like this. there weren't widespread communities welcoming new users or lots of YouTube videos explaining everything and getting viewers excited.
yes, that was the time to be into Inferno and projects like it, but the entire internet was a different place.
Maybe I just wasn't looking in the right places for this stuff or maybe I was too invested in other things, but I recall these things being terribly hard to penetrate even if you did find out about them.
From what I remember reading, JVM is a stack machine, while Dis is a register machine, right? Is there any other significant difference that would make Dis less "future proof"?
If a future CPU has more registers than your VM, you have to either cripple your VM by not using all registers of the new hardware or write code to detect and remove register spilling that was necessary on the smaller architecture.
If, on the other hand, it has fewer or a different mix, you have to change your VM to add register spilling.
Either way, your byte code compiler has done register assignment work that it almost certainly has to discard once it starts running on the target architecture.
If you to start at the extreme end of “no registers”, you only ever have to handle the first case, and do that once and for all (sort-of. Things will be ‘a bit’ more complex in reality, certainly now that CPUs have vector registers, may have float8 or float16 hardware, etc)
You can also start at the extreme end of an infinite amount of registers. That’s what LLVM does. I think one reason that’s less popular with VMs because it makes it harder to get a proof of concept VM running on a system.
The rest of the world's CPUs have normal registers, which is one aspect of what makes register-based VM bytecode easier to JIT to the target architecture, which was one of Dis' original design goals (with an interpreter being a fallback). It also happens that we know a lot more about actually optimising register-based instructions (rather than stack-based), so even if you had to fall back on an interpreter, the bytecode could've gone thru an actual proper optimisation pass.
The sparc sliding register window turned out to be a very bad idea, but I guess you already know that.
I think the Itanium did register windows right: allocate only as many registers as the function needs, and overflow into a separate "safe stack". Also, the return address register was among them, never on the regular stack, so a buffer overrun couldn't overwrite it.
There is a third option to stack and registers: The upcoming The Mill CPU has a "Belt": like a stack but which you only push onto. An instruction or function call takes belt indices as parameters. Except for the result pushed onto the end, the belt is restored after a function call – like a register window sliding back. It also uses a separate safe stack for storing overruns and return addresses. Long ago, I invented a very similar scheme for a virtual machine ... except for the important detail of the separate stack, so it got too complicated for me and I abandoned it.
Yikes, that triggered flashbacks of 8086 segmented memory.
I'd very much like to understood what was going through the sparc designers minds when they did that. Looking back on it with my own current understanding of CPU designs and all that, they seem to have made some incredibly basic mistakes, including designing the hardware without talking to the compiler writers (a cockup the Alpha designers very definitely didn't make). It's all very odd.
Another mistake they made was apparently deciding to leave out instructions based on counting them in the code – if an instruction didn't appear very often they omitted it. Sounds reasonable but that meant they left out the multiply instruction initially, which might not have been so common in the code was actually executed quite often (e.g. in array lookups) and there were complaints about the new Sparc stations with their new superior chip, that they were slower than the 68000-based CPU that preceded it. Hardware multiply was later added.
Interestingly, Plan9 started moving in this direction in their later papers. They'd walk through all the many different file operations you'd need to get something accomplished, and then say "but we made a library which does all these things, so you don't need to do it yourself," which kinda defeats the purpose of having everything be a file--and brings you toward the Fuchsia approach.
Big shame though, really love the idea behind it :/
I wish Linux distros had this feature as well.
Its why rm evokes -i by default and requires -f to avoid interaction on some OSes. Users can and will make a mistake with it, especially new users. And everyone makes typos.
As you say, it's a bit scary that any typo can install random commands lol
I already installed several 1 character langauges by mistake
$ v == https://vlang.io/
If a distro were to appear with this feature enabled by default, I hope they would have the foresight to not include "sl", or at least put it in the default blacklist.
Can you point out which ones? I thought I was pretty familiar with the Plan 9 papers but don't remember that part. Or maybe I just missed it?
Lynxline's Inferno Labs on porting Inferno to the Raspberry Pi:
https://github.com/yshurik/inferno-rpi (the related web site http://lynxline.com/ is currently not reachable)
David Boddie's port of Inferno to the Ben NanoNote, a tiny MIPS-based handheld from 2010:
https://www.boddie.org.uk/david/www-repo/Personal/Updates/20...
I've run Plan9 in the paste on Raspberry Pi and found it to be a neat experience. Inferno I've only run under Windows, and which seemed kind of pointless.
https://github.com/yshurik/inferno-rpi/releases/tag/v0.6
Note that this Inferno port only works on the original Raspberry Pi 1 (probably also the 1B and the Pi Zero).
https://web.archive.org/web/20141216095808/http://lynxline.c...
> The Inferno Business Unit closed after three years, and was sold to Vita Nuova Holdings. Vita Nuova continued development and offered commercial licenses to the complete system, and free downloads and licenses (not GPL compatible) for all of the system except the kernel and VM. They ported the software to new hardware and focused on distributed applications. Eventually, Vita Nuova released the 4th edition under more common free software licenses, and in 2021 they relicensed all editions under mainly the MIT License. [1]
[1] https://en.wikipedia.org/wiki/Inferno_(operating_system)
gscreendata.data = (ulong *)(va+0x800000); /* Framebuffer Magic */
[2] https://github.com/inferno-os/inferno-os/blob/48f27553574bf5...
It’s proven easier to implement many of the features of Plan9 lineage and Erlang via containers and orchestrators, rather than porting software and reteaching developers.
It's interesting to me how channels in Limbo look and work very similar to Go's channels.
Great. But `open()` needs to be hyper-extensible. Think of URI q-params and HTTP request headers as `open()` options. Think of all the `ioctl()` APIs that have arisen because "files" are a relatively poor way to represent devices. But most importantly `open()` needs to be async.
I'd be very interested in an OS where all system calls that can block are async, and where something like io_uring is the only way to make system calls.
The "everything is a file" thing is fine, but we need more innovation around that if that metaphor is going to stick.
Welcome to WinRT application model as it was introduced in Windows 8.
Sadly it was yet another reason why the Windows development community rebelled against it.
I'd like to see an OS that drops the concept of files. Files are very low level (generally a stream of bytes) - the app has to interpret it as config, or data, etc "manually". An OS should be providing higher-level data management, and insisting that is what is used.
(And please not SQL. It's only a little higher level that raw data, and has serious interface problems that permit prompt injection ... the db equivalent of buffer overflows.)
The original/legacy OS is quite interesting and high level. It's Object Based. You can only use the objects built-in methods e.g. you can't WRITE arbitrary bytes to an Application object, or a User Object. They are accessed as a giant Single address space, and whether they are on disk or in RAM is transparent to higher layers, i.e. if an object isn't in ram it will page it in..
The designer of System/38 etc. Frank Soltis wrote a couple of interesting books on it - Inside The AS/400 and its 2nd edition Fortress Rochester: Inside the iSeries. They're out of print and expensive unfortunately, I wish I hadn't gave my copy away.
Indeed, Plan9 and inferno don't have ioctls. For special operations ("control"), there is often a file named "ctl" to which you can write commands. So you open the file, write some text, see if the write succeeds, and close the file. E.g. to make a tcp connection, to flush a write buffer to disk, etc. The commands are typically just plain ascii. That's easy for scripting and there is no need to import C struct types into that new programming language you are using/developing. Of course, the commands have to be parsed (for kernel devices, this happens in the kernel), typically done with some simple tokenization functions. When the commands become complicated, you could still choose to write binary data, or even straight C structs...
This assertion seems normative. Could you please expand upon how higher-level data management improve the overall performance and efficiency of the system? Or could you point me in the direction of some good sources? Also what are the benefits of an OS providing higher-level data management instead of relying on lower-level data management solutions? Doesn't abstraction lead to less fine control? That is to say, how does insisting on using higher-level data management provided by an OS affect the development and maintenance of applications? I've seen some object-centric systems adopt this approach and find it very interesting.
Rather than a loss of control, capabilities enable fine-grained control over authorization, because a capability is both designates a resource and provides authority to use it.
But pathnames are not useful as capabilities, because they can be easily forged. So at best a pathname could serve to discover resources for which you could then request a capability to access.
Everything in Windows is an object, on a centralized resource broker Ob.
Windows uses capabilities based access to enable fine-grained control. It is EAL4 - Methodically Designed, Tested and Reviewed.
This by itself doesn't prevent Windows from having security issues.
With proper capabilities, the capability itself provides the authority. There's no need to have separate access control lists or some kind of central resource broker. Each process manages its own capabilities, can create new capabilities and can delegate them to others. And importantly, capabilities can always be revoked, at any time.
See: http://www.erights.org/elib/capability/overview.html, https://en.wikipedia.org/wiki/Capability-based_security
Also see seL4 for an example of this done right.
Of course, "abstraction lead[s] to less fine control" - at the the lowest (assembly) level, you can do almost anything - and make all the mistakes imaginable. Sometimes you want to genuinely maximise performance, or do things otherwise difficult - fine, use assembly and hit the hardware. But most of the time, software is made to be read, to be trouble free, to build on other's work, and to be written easily - that's when abstraction is valuable.
the plan9 window system also worked this way, so you could access a window on someone else's display if you mounted it locally; this was how you would run a graphical program on a remote server, by mounting your desktop display in its container on the remote server
linux is not like this at all; you have to use separate protocols for graphics, vpning, and filesharing
could you somehow mount or pipe the openssl library on top of the network interface via a helper program and then access it that way to encrypt it?
Also, unrelated, do you have any advice for a very short-sighted and poorly thought out move to Argentina, assuming the person you're talking to is completely set on it?
i decided that probably write() and read() is not a good way to send a large volume of pixels to the display hardware because, say, 2048×1080 32bpp at 60Hz is 530 megabytes a second, and that's a significant fraction of typical bandwidths to main memory, so it is important to strictly minimize the number of memory copies. indeed, even today, i think this is a primary driving concern in the design of usably efficient graphics systems
if you write() some pixel data to a socket, the semantics of write() imply that you can overwrite it (for example with the next frame) as soon as write() returns; and the semantics of read() mean that the pixel data read from the socket by the display server needs to get put in the display server's memory space at a buffer aligned where the display server allocated the buffer. moreover, if the pixel data is intermixed in the same byte stream with control and framing data (such as the w×h dimensions of the following chunk of pixel data, for example) that will also tend to misalign it
so wercam allocates the pixel data in shared memory, on unix in the form of a memory-mapped file, and then transfers the ownership of the shared memory space from the drawing application to the display server — ideally this would be done in such a way that the display server automatically knows that the drawing application cannot continue overwriting it, but on unix that is impossible, so they share access. when the display server is done with the pixel data buffer, it returns it to the drawing application for reuse
writing this, though, i am struck by the realization that you can probably avoid the extra memory copy with write() and read() by the simple expedient of allocating the write() and read() buffers at page boundaries and in multiples of the page size, so that a sufficiently smart kernel can handle write() by marking the written pages copy-on-write rather than actually copying the data, and handle read() by adding a (copy-on-write) mapping for the same pages to the receiving process's address space. then the misalignment induced by the control and framing data is inconsequential, and possibly even helpful, if the framing data is a multiple of the cache line size and potential simd register size, so that the pixel data is cache-line aligned and less likely to create cache contention with whatever framebuffer the display server eventually copies it into
— ⁂ —
as for moving to argentina, it's a good idea to have officially certified copies, made within the last year, of all your immigration-relevant documents; apostilles for those documents; a lot of money, ideally in bitcoin or 100-us-dollar bills (smaller bills and the old bills with a smaller benjamin franklin face trade at a discount, and bitcoin trades at a wide spread); fluency in spanish; a small number of easily salable electronic gadgets that you nevertheless actually use, such as macbooks, iphones, and recent phones or tablets from xiaomi, samsung, or motorola (compatible with the frequency bands we use here!), along with powerbanks and bluetooth earbuds; and enough of your savings to live on the form of 18-karat gold jewelry worn under your clothes
you can't legally bring pepper spray on the plane, but you should probably buy some as soon as you arrive. expect to get robbed about once a week during the first part of your stay, including from your checked luggage before you arrive, so don't be too attached to your possessions
withdrawing money from a dollar bank account at argentine atms does work but you only get about 45% of the money you withdraw thanks to the fake exchange rate; some expats have reported success in sending themselves money with western union, which doesn't use the fake exchange rate, but only scales up to about 200 dollars at a time, and may trigger investigations by the tax authorities
because voip on cell data is illegal, voip accounts from companies with a nexus in argentina will not work, at least on cellphone data (which is quite cheap, but i think signing up for a cellphone line will require an argentine citizen or permanent resident to vouch for you). this means google fi doesn't work at all, but for example voip.ms works fine with sipphone, and so does jitsi. also most cafes, restaurants, hotels, etc., have free wifi for customers, secured with a wpa psk key that changes every year or two
once you have a place to live that isn't a hotel or hostel, leave your passport there so that if you get robbed at least you won't lose your passport. unless armed robbers escort you at knifepoint to your house and demand entry in order to loot your house, which is a thing that happened to a couple of friends of mine. still you can probably hide your passport somewhere that they won't find it, and tell them that you lost it and are waiting for a replacement if they ask. a usa driver's license is generally enough to satisfy cops who demand to see your papers, and in 17 years that has happened to me only once, while i've been robbed on the street several times
don't expect to get a job! we're in the middle of the worst economic crisis we've had since 02001, so you'll probably have to live off your savings (or earnings from working for overseas clients) indefinitely; also keep in mind that living in the middle of an economic crisis can be depressing and anxiety-inducing, which can exacerbate any mental health problems you may have
stay out of the villas and la boca
if you're trying to do something else on the computer, even on a different core, that's bottlenecked on memory bandwidth (as opposed to cpu or i/o or something) that's like using 50% of the computer, which means the whole computer is effectively half as fast
i think it's worth a significant amount of complexity to get your display system to suck up 30% of your computer instead of 50%, even when you're doing something that updates the full screen every frame, such as smooth scrolling, a 3d fps, or watching a movie, and that's why I think it's important to try to minimize copies in the pixel paper path
also keep in mind that many people dual-wield monitors these days, and 3840x2160, though common, is not as big as they get
In Linux, you often use sockets to communicate over the network, perhaps with sendmsg[1] to send data.
On Inferno (as I understand it), everything is done via the standard filesystem operations of open/read/write/close. And that includes everything on the system - printers, mouse+keyboard, even the entire windowing system.
/dev/eth0 ?
$ ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:15:5d:23:d8:89 brd ff:ff:ff:ff:ff:ff
inet 172.27.50.201/20 brd 172.27.63.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::215:5dff:fe23:d889/64 scope link
valid_lft forever preferred_lft forever
$ ls /dev/eth0
ls: cannot access '/dev/eth0': No such file or directory
https://web.archive.org/web/20070311234222/http://tuxscreen....
That didn't really age to well for Unix. I wonder if it is different for a distributed os.
Similar to how HTTP is just a transport protocol, it's what you transport and how that is interpreted that matters. By restricting the number and kind of endpoints that you allow a driver or virtual device to have you ensure that all tooling is instantly composable, which is a much bigger advantage than the ones that you get from 'more effective APIs', which always turn into a giant salad of RPC. Think protocols, not functions and you're well under way to seeing why there is a lot of power to be found there that we've thrown out in the name of a couple of percent of efficiency.
Plan9 / Inferno were also design descendants of System V Streams.
My two cents is that composing applications at the FS level is more expressive than Files/Sockets/HTTP requests.
Gluing up different of pieces of software becomes easier as most of the time it becomes just a matter of mounting a filesystem and you don’t need to worry about it being provided by a local or remote process.
> “People often ask where the names Plan 9, Inferno, and Vita Nuova originated. Allegedly, Rob Pike was reading Dante’s Divine Comedy when the Computing Science Research Group at Bell Labs was working on Inferno. Inferno is named after the first book of the Divine Comedy, as are many of its components, including Dis, Styx and Limbo. The company name Vita Nuova continues the association with Dante: his first work, a book of poetry about his childhood sweetheart Beatrice, was called La Vita Nuova. The literal translation of Vita Nuova is ‘New Life,’ which in the circumstances is surprisingly prophetic. Plan 9 is named after the famous Ed Wood movie Plan 9 from Outer Space. There are no other connections except that the striking artwork for the products is a retro, 60s SciFi image modeled on the Plan 9 movie poster.’
https://dantetoday.krieger.jhu.edu/2009/03/08/vita-nuova-and...
One can build a tiny grid with encrypted communications to share resources using Inferno operations to glue it all together.
Nowadays 9p, (known as Styx on Inferno), is used in the Linux kernel, as part of WSL, QEMU, and various other places to share resources and files on a network.
It's a really simple protocol. I started working on an implementation in Swift, and should probably finish it someday.