Harvey OS – A Fresh Take on Plan 9
harvey-os.org
harvey-os.org
I suspect there's an opportunity for a lot of alternative systems to make inroads in that space. But, then again, if the OS doesn't matter...it may be that we all just end up using the path of least resistance, which is almost always a small Linux distro with all the stuff we're used to. But, for someone that loves Plan 9 or Solaris or OpenBSD and wants to deliver service containers, they can probably get away with deploying containers using those systems without people balking at the idea.
That, of course, doesn't mean that only microkernels are used in embedded and real-time applications. It's not uncommon to see, for example, a stripped-down Linux or even Windows in some applications. In some cases, even MS-DOS and its kin are still used. But by and large, the most of the kernels you're looking at in this space are indeed microkernels.
The microkernel architecture also offers some advantages in the high-assurance space. See seL4, for example. This doesn't necessarily have anything to do with any technological superiority of microkernels, per se, but they tend to be far smaller than monolithic kernels, which in turn makes them much easier to audit and verify. The Minix 3 kernel, for example, is something like 6,000 lines of C -- roughly 2/3 the size of GNU grep -- which makes it comparatively easy to audit, simply because there's less code to audit.
Yes, but the kernel services are the part that's incompatible between systems. Containers do not run their own kernel, but instead run on their host OS kernel.
(Is there a good generic word for Oracle Solaris + SmartOS + Illumos + etc.?)
When Oracle took the source closed again, they essentially forked their own private product. They have diverged significantly enough from everybody else that it's not really helpful to include them in the same family anymore.
tl,dr: we just call it all "illumos" now.
I knew it had fans but I assumed people moved on after Oracle closed the open source development.
But where I was it seemed like it was open sourced there was a lot of excitement about its future, and after being bought by Oracle and closed source, not so much.
This idea would mean you're not running the OS's kernel, and you are only running the userspace components that were written against Linux emulation, which are often just Linux binaries copied from some Linux distribution.
We have that. It's called Linux.
Containers don't actually contain very well on Linux (yet, though some nice things are happening on that front, and you can already build very solid things with SELinux, but I haven't seen anybody actually doing so; except probably the Project Atomic folks).
But, yes, in theory, a container-based deployment is hella tight on the security front. One service being compromised is no big deal: Kill it and instantly spin up a new one on a new IP (hopefully with a fix for the security issue). If your visibility into the system is high enough, you may even be able to architect detection of compromises; e.g. if your container for database service starts sending email or opens an IRC port, you know it's fucked, and you kill it with prejudice. In a system built to think of containers as black boxes with APIs, you don't need to keep any particular one around if you have reason to tear it down.
Note that I haven't actually seen people doing any of this with containers. But, it's a thing that one could do that cannot be done when all of your services are applications running on a single big machine. The narrower the purpose of your container, the more secure it can become. You still have lots of moving parts in your total infrastructure (more, actually, since the container management has a cost, too), but each moving part has very well-defined boundaries, and misbehavior is easier to spot and easier to shut down quickly and via automated means.
Operating system upgrades.
Configuration management.
Startup/teardown of multiple instances.
Symbiotic or dependent processes/programs.
Programs not specifically built with isolation in mind.
Inter-application security.
For example, running a tiny jobs server:
* Scheduler in one process which spawns
* Application code running in many subprocesses
The scheduler might be in Ruby and the jobs might be in Ruby; or the scheduler might be Cron and the jobs in shell. In either case, if a job crashes the other jobs and the scheduler are very likely to carry on their work.
It is also nice that you can bring together tools from different communities into a single application. A "Rails" container might utilize Nginx (C), Rails/Unicorn (Ruby) and Node (JS/C++) together. A single statically linked application would, barring some really amazing innovations in something like Clang modules, have to be in one language therefore from one community.
Erlang solved a lot of problems of scale in very effective ways. These new architectures are solving the same problems in ways that look similar from a distance. This new(ish) approach just means we can use existing software mostly unchanged, while still getting a lot of the resiliency benefits of something like Erlang; doing it with Erlang means everything has to be written in Erlang. That wouldn't be a crazy idea for some deployments. But, for others, it's not tenable.
Not necessarily. If they all can compile to native machine code with the same C-like semantics, they can (in theory) be combined in that sort of manner. This is true of both Rust and Go, last I checked, as it is with pretty much any compiled language that has a "convert to C" step (Chicken Scheme is an example that I'm particularly familiar with) or perhaps even a "convert to LLVM IR" step.
Not that this is especially feasible for the majority of languages being used in web development (at least by those with a lick of sanity; Go and Scheme are the only exceptions that I'm aware of off the top of my head), but it's still worth considering.
I'm sure the Plan9 ABI variant used by Go makes more sense than the platform ABI, but supporting the platform ABI for FFI doesn't add much complexity at all to the compiler.
I'm 91% sure my point still stands with Rust, though, at the very least if everything's compiled through LLVM (and uses the same calling conventions).
Isn't that the whole idea of unikernels?
And the answer (IMO) is that in the end your unikernels are just fat applications and your hypervisor is just an emaciated OS. The abstractions are at the wrong places and you can't properly optimize resource usage because the hypervisor has to treat the guest OS as a black box. Otherwise it's eventually going to cease being a hypervisor and be a cruft-laden kernel.
If our only concern was to get the most performance out of a single piece of hardware, that would be the end of the discussion. But, that's not the problem this kind of architecture is setting out to solve. We've already reached the point where raw performance on a single box is less important than being able to virtualize. EC2 proved that a decade ago; we're willing to trade raw performance for the flexibility of virtualized infrastructure in some cases. Hell, EC2 is about twice as expensive as similar physical hardware (even today); but subtracting the cost of managing the hardware and adding the ability to spin them up and down makes them more cost-effective for some deployments.
So, sure, people are trying to make it efficient. And virtualized and container based hardware has improved remarkably in that time (virtual disk and network I/O is now very close to bare hardware performance, whereas it was very far from it in the early days). But, even when they fail to achieve near peak performance there's still a compelling benefit for many uses.
The idea is to have each app in a "ready-to-run" state, so that it can be popped into existence anywhere on a pile of infrastructure and shutdown when it's not needed anymore. A container+app is a self-contained unit. If you're running several apps on an OS, and you update the OS, you now have to test all of the apps to be sure the update doesn't break things. If a security vulnerability happens for any of those services, you may no longer be able to trust the entire system. Apps that misbehave might impact all of the other services on the system. Containers are not a panacea, mind you...they introduce new headaches of their own, and the way a lot of folks are building and deploying containers is a joke (it's been a great excuse for people to start pushing out big binary blobs of crap again, which was a thing I'd hoped was behind us years ago).
But, the principle benefit is that a container provides a level of abstraction at the service level rather than the application level. This "thing" (which is a container that has an app in it) can be spun up anywhere and do its job. It can be upgraded or modified (including libraries and other packages) without impacting any other thing in your stack. Part of a self-healing infrastructure is being able to think of any service as a cog in a greater machine, rather than some individual piece of complexity you have to interact with to keep it running. And, ideally, we'll be able to outsource maintenance of those cogs to someone else, the way we outsource packaging binaries to OS vendors, today. Right now, that's a mess, and most of the people building containers have no clue what they're doing; so it's a minefield. But, it won't always be so.
It's just another abstraction for us to work with, is what I'm trying to say. A container can be a black box that we only need to open up occasionally (or never, if we trust our vendor to build it right).
"If it is something better, then why don't we write applications to be that way to begin with? Serious question."
We probably will, eventually. At least, it'll be closer to that than big full OS + a bunch of apps running on it.
The benefits are many; the costs and tooling are still high and fragile. But, it'll get better, and we'll get more fine-grained in how we're deploying these pieces. Most people, including myself, still think in terms of "I have a server, now I put things on it, and they combine to make a website/mail server/whatever". That's probably the wrong way to approach big web scale problems (though the number of people actually managing web scale systems vs. server-scale systems is probably small; most web sites and mail servers will never need to expand beyond a single server, and it's probably a stupid waste of time and resources to do otherwise).
The future of big web service looks like Kubernetes or something along those lines (one-app per container, very small and focused OS images, somewhat ephemeral in nature...they will come and go). It does not look like one big server with a bunch of apps running on it.
https://en.wikipedia.org/wiki/Tanenbaum–Torvalds_debate
My take on that debate: They were both right, but they were optimizing for different things.
Tenenbaum was right in a theoretical sense and an eventual sense. Microkernels are better. Separation of concerns is better.
Torvalds was right in the early-mid 1990s and right in a pragmatic sense. More monolithic (or vertically integrated) kernels are easier to build and ship and are faster on smaller machines. A microkernel made little pragmatic sense on a early-mid 90s box. But keep in mind that these machines were far less powerful than a Raspberry Pi. My first Linux machine was a 66mhz 80586 clone with 32MB of RAM.
Once you have machines with dozens or even hundreds of cores, NUMA, and that host massively diverse multi-tenant work loads, microkernels start making a whole lot of sense. Once you have to containerize apps for security, they make sense too. They also make sense if we want to eliminate reboots, etc. They make much more pragmatic sense now than they did in the 90s.
Ultimately microkernels or microkernel-like systems will win.
If we're talking about BSD jails, Solaris/Illumos zones, and Linux cgroups + namespaces, there are basically two scenarios: Either multi-tenancy is handled through hardware virtualization (mainstream cloud providers), or all of the containers are run on a shared monolithic kernel (Joyent's Triton, running Linux binaries in containers via LX-branded zones on Illumos). I suppose the former scenario could be seen as a vindication of microkernels, if you consider the Xen hypervisor a microkernel. But that has nothing to do with containers.
Xen is becoming a major OS, and it is a microkernel OS.
Except now it's all a lot cheaper.
ESPOL, NEWP, Algol-68RS are a few examples.
That is going to take a while, maybe a few developer generations still.
Apparently Genode OS has now integrated Rust, but they still aren't using it.
It would be nice if the Android team recognised the work being done by the Go team and integrated Go into its OS at very least on the NDK, but I don't see it happen. Specially after seeing their reaction about Java alternatives at this Google IO. The attitude "nothing but Java" is still the official one.
On Windows I would like to see .NET Native become more relevant and bring System C# goodies to mainstream Windows.
I'm not sure why Java should be excluded if we're talking about systems languages other than C/C++ that are safer.
Also Google is pushing Android Java, not Java, with its partial support of standard Java features.
If you see their Google IO talks, they really mean Java and not other languages that happen to target the JVM.
That is what most cars, factory control units, avionics and many other embedded real-time OSes are.
Every phone also has a tiny OS running their radio band unit, which happens to be OKL4.
Just the Linux crowd seems to be unaware of them.
I also started to see POSIX as the extended C runtime that didn't make it into the ANSI C standard.
C compiler: yes, years ago, Ken C was faster than alternatives like gcc. To my surprise, that's no longer true. A full build of all the plan 9 libraries, tools, and kernels takes about 2-3 minutes with ken c, gcc, and clang. It's why on every push to github we can do a full build with both gcc and clang using travis. There's no longer a good reason to stick with the Plan 9 C compiler and, once the Go team dropped it too, there were lots of good reasons to move away from it. So we did. As the original paper from Ken pointed out, it only ever produced medium quality code. It also had several bugs with 64-bit scalars that people had to work around in their C code. It was fast but in the end not a very good compiler.
Unix wrinkles: there are so many, but just consider the sockets interface. Sockets completely break the Unix model of named entities that you can open and close and read and write. Sockets don't have names and are not visible in the file system. It's a long list of what they got wrong and that's why, in 1978, other mechanisms were proposed (if you're willing to dig you can find the RFC and you'll see ideas like /dev/tcp/hostname and so on). But to our dismay (I was there at the time) the current BSD socket interface won. Many of us were disappointed by that decision but it was the fastest way to get network support into Unix.
On the acceptance of Harvey: Presotto and Hume showed up for our BOF at the 2015 Usenix in Santa Clara and were very supportive of what we are doing with Harvey. Other former members of the Bell Labs community have been similarly supportive.
On the logo: I'm glad people like it.
On the way forward: we welcome new contributors, there's lots of work to do.
9front is alive, sure, but bootstrapping things atop a modern compiler and a (at least partially) Linux compatible ABI makes a lot of sense.
Meaning, most software won't run on Plan9 without major modifications, so bringing a modern GCC/CLANG would give access to better compilation, but really there needs to be more work on APE to get more software over (lots of low hanging fruit there from what I've seen).
The Go folks have adopted some of the same ideas (minimal optimization and no nested include files), and they build on the experience of the Plan 9 compiler to better divide the work between the "compiler" and "linker", so we gain more from parallel make.
The build speed doesn't matter much if you don't like the runtime speed.
Do you have any sources on this? Can you perhaps link me to something that explains the differences between the ARM and the P2 that lead to such a difference in performance despite the clock speeds?
I did this partly because plan9 had published some source code and times on a system with exactly the same hardware as mine - AMD k6/350.
I suspect the plan9 compiler hasn't had nearly as many changes since then so the gap would be much wider.
I acknowledge compile time is important also - and this is where the plan9/golang people have made a reasonable trade-off.
http://vignette3.wikia.nocookie.net/farscape/images/b/bd/Har...
If I'm just using a library operating system that is linked directly into a single unikernel that merges the application code and system code, then I don't really care if I'm running on a hypervisor, on bare metal, or a host operating system. I'm only using a few system capabilities, and I'm not really taking advantage of other services running on the OS.
So I don't really care if the host OS is Linux, or Plan 9, or BSD. It just has to be UNIX-y enough to host a VM for my unikernel.
What would those be?