So: why Mezzano?
You don't have to reboot a machine just because you make a patch to the kernel, and at the same time, you can patch the system using the same high level development tools as you would use to develop an application.
The thing is that there's what we call an impedence mismatch between a unix kernel and applications running on it, in that the data types processed by applications which are of higher level, don't match the data types processed by the unix kernels. For example, a Ruby Integer is actually a bignum, and you cannot pass a bignum to the unix kernel: you have to provide a bit field with 32-bit or 64-bit. This is work, and this is pain. It is already pain when you write your applications in C or C++ where assumedly you already have bit field types, because your application could be compiled to run on kernels and processors using different word width! Imagine the pain it is when you write your applications in Common Lisp with very high level data types (ratios, closures, pathnames!), and when you have to map those data objects onto the lame bits accepted by unix syscalls. Consider all the literature written about logging, the byte vector logging provided by unix syslog vs. high level object logging provided by higher level libraries or other systems.
And of course, this is not limited to the interface between applications and systems, but goes beyond to the interface between applications. When the system provides a pipe abstraction where all you can pass between applications are streams of byte, this leads to a lot of suffering. You have to serialize and deserialize your data, you have to consider formats (a java float doesn't have the same syntax as a Common Lisp float!), encoding (utf-8? iso-8859-1, -15?). Have you ever been advised to never parse the output of /bin/ls? For good reasons!
Instead, if your system is written in Lisp, then you can directly pass lisp objects from one lisp application to another lisp application, and there's no need to serialize/deserialize, to parse or otherwise mangle the data: you just have lisp objects and you can use them directly. You can pass closures (which enclose the lisp object data along with the lisp functions needed to process them).
So, if I understand correctly, the memory block in which a particular object is stored doesn't belong a priori to this or that process, but rather you can simply hand down objects to other processes? This is very cool! I've been thinking for quite a while that the very idea of memory protection is just a lame workaround to deal with the fact C is memory-unsafe.
Whilst convenient, it isn't an unqualified good. Bugs in any part of the system have the potential to catastrophically break everything. It's also terrible for security. |Without the notion of a distinct kernel - or "unsafe" code that only privileged users can compile or load - there's nothing to stop user code reading or writing directly to/from IO ports or arbitrary memory locations.
Being able to pass closures isn't something that's particular to Lisp, nor does it require a lack of memory protection. It just requires that the calling convention can denote a closure as such, and can arrange for it to be invoked in the right security context - i.e. the one in which it was created.
Another issue with having a single global environment backing everything is that you can't easily experiment with changing built-in Lisp functionality without immediately crashing the system.
This sort of thing is probably perfectly fine for embedded systems, and "micro-service" VMs, though. The benefits may well outweigh the risks then. And of course it's great as a starting point for experimentation.
I'd rather there were no distinction between processes and threads. It's artificial. I want to live in a world where processes cooperate to produce useful results for the user, rather than assume other processes are out there to corrupt their data. If there's any memory protection, I'd rather it be a compile-time, rather than runtime check.
> Bugs in any part of the system have the potential to catastrophically break everything. It's also terrible for security.
Proof-carrying code seems like a better solution than memory protection.
> Another issue with having a single global environment backing everything is that you can't easily experiment with changing built-in Lisp functionality without immediately crashing the system.
To be honest, I'm not so interested in “experimentation”. (I'm culturally not a Lisper. I like thinking and getting things right before I write code.) I just think concurrency would be a lot simpler if you could pass objects directly between programs written by different people.
I think the idea of compile-time safety proofs is interesting. The number of runtime cycles spent on policy enforcement could be greatly reduced. Would it ever be possible, though, to derive such proofs for programs written in "unsafe" languages, or for arbitrary binaries?
An OS which only supports one language would be unlikely to be generally useful, although it might provide sufficiently compelling benefits for specialized use cases. I believe MirageOS represents an OCaml implementation of this idea, although it may in fact support other languages.
The security angle is interesting.
The thing about independent processes is that we're relying on the OS to provide a wall around each process to limit it's impact by default. Maybe a better approach would to be include something like chroot/containerization for execution of closures within the language? Ie, if program 'A' receives data from program 'B', it can evaluate that data in a sandbox with limited permissions (IO limitations like no networking, or CPU resource limitations and time limits) and receive exceptions if the closure tries to exceed it's permissions. That could fit in pretty cleanly with Common Lisp's condition system.
A con of this though: you could argue that it really breaks encapsulation; it's bad enough when different parts of a large program become too closely coupled - allowing different applications to depend on the internal state of each other could be a nightmare.
On some hypothetical Lisp/high-level-language processor, that needn't be the case, of course, but I honestly don't think we'll ever see something like that again.
The damage this can potentially cause can be prevented using proof-carrying code.
Store applications as byte code or source code and use a trusted JIT to generate machine code. With a system-wide garbage collector and bounds checking you would have a memory safe single address space operating system.
Why?
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.4...
http://mumble.net/~jar/pubs/secureos/
That was the Scheme mathematically-verified in VLISP project:
https://en.wikipedia.org/wiki/PreScheme
There was also a mathematically-verified, Scheme CPU:
http://www.cs.indiana.edu/pub/techreports/TR544.pdf
I imagine combining even these three components into an integrated systems would provide a lot more security and availability than the average desktop. The simplicity of the CPU might also aid in producing a NonStop-like solution for five 9's availability.
I grew up with a machine which booted into a BASIC interpreter. It was relatively primitive, but fun. How would it look & feel with a more powerful programming language?
This may all have its limits, but it looks like an interesting experiment. At some point in time it might even useful...
ZX Spectrum by any chance?
My first game was a horse racing / betting game, reflecting my Mom and Dad's penchant for the ponies (OTB, you owe me big time ;) IIRC, I could only use graphics characters if all the letters were uppercase. My game had 3 horses to a race that ran from left to right based on a random number of which horse and how many spaces it moved.
I generated a random number of 0 to 3 for each of the horses with:
100 FOR R = 1 TO 3 110 X = INT(4 * RND(1)) 120 NEXT R 130 SPC(A)
You could bet based on fixed odds, and the payoff would show at the end of the race (bet * odds). My memorable joy was watching my Mom and Dad rooting at the 9 inch monochrome green screen! I was hooked on coding, but only at home. I rarely worked coding for a living.
It booted up into PET BASIC, and aside from some PEEK/POKE limitations, you could access all of it. People hooked up joysticks later to the user port, and hacked speakers or buzzers for sound. I loved the Datasette (cassette tape drive) for storage! You had to put the tape in the drive, instruct BASIC to LOAD "PROG", and then it would prompt you to hit 'PLAY' on the Datasette. I think you then typed RUN "PROG" when if finished loading.
I would go to the store where I bought it in NYC, and they had like 4 or 6 plastic bags with cassette tapes in them and a one sheet or a few sheets of instructions. I wanted FORTRAN or APL, but APL was not available on my PET.
I would love a real LISP Machine, even an historical one for the pleasure of it really being 'turtles all the way down'! I love Lisp more than BASIC, but PET BASIC will always have a special place in my heart, and in the cobwebs of my mind.
[Edit] It would boot up in about 4 seconds or so!
When I look at the video of Kalman Reti running Symbolics Lisp Machine in an emulator [1], I am still blown away by how much more sophisticated and aesthetically pleasing it was compared to my PET or the Apple ii. The difference seems almost asynchronous, like time-traveler tech.
How you could just drop in to any system library, or even the kernel, and use the same language, Lisp, to modify anything live is astounding. A lot of people have dismissed any talk of how great LISP Machines were as a bunch of nostalgia, but I don't think any of them have watched 15 minutes of somebody operating in that environment. You can't look at it, and keep a straight face when talking about how great the Apple ii was or the Lisa for that matter later on.
Example: https://youtu.be/T0PFUC4Nuuk
same here (bbc b). i still miss that experience sometimes; a bash prompt is not quite the same thing.
One way to think of it is as a Lisp Machine [1] using an x86 CPU.
http://www.symbolics-dks.com/Genera-why-1.htm
...still aren't available in mainstream OS's despite being totally awesome. I particularly like how they came with integrated editor and source so an OS-related problem in app makes editor show you the problem (data or whatever), the source code involved, and a REPL letting you live-update it. Academics keep building prototypes that approximate any one of these steps to some degree for Linux or BSD. Not the whole thing, consistency, and production grade. :)
What is the Right Way? I couldn't find their take on that in 5 minutes on their website; this seems closest: https://doc.redox-os.org/book/introduction/why_redox.html
Just curious, how much of the Redox code is wrapped around unsafe?
Rust is built around wrapping unsafe things safely - i.e. it's possible in nearly every case to develop a performant API that doesn't let you do anything to break Rust's invariant of memory safety. Usually, these are small, meaning they're testable, which is likely good enough for almost everyone.
The next step up in safety is proving a program against a model of a system, which is a heck of a lot of hard work - you have to create a formal model of every single piece of hardware you might ever want to use. The next step after that would be proving a system actually matches the model.