HermiTux – A Linux binary-compatible unikernel

> transform syscalls to function calls

I thought that was kind of the point of a unikernel. Or am I mistaken? Does the other one from the other day not do that?

https://github.com/cloudius-systems/osv is an offshot of a project from ScyllaDB.

That video is great! thanks

Well, there's two things here. One, typically executable code is packaged in a file along with other content, otherwise, the OS doesn't know that the contents are intended to be executed.

The broader problem is, if you could somehow work around that... what are you going to _do_? You need OS-specific code to read and write files, to display things, to get input from the keyboard/mouse, to talk to the network. It's not like Ye Olden Days where you can just write directly to the display memory and have things show up, the OS controls your access to all of the hardware. An OS-agnostic executable can't accomplish much on a modern PC.

There are a few roadblocks that probably will not be overcome, at least not without introducing more of those abstraction layers.

The binary format itself is a roadblock, because the loader of your OS needs to know that in order to load and start your program. To be compatible e.g. for PE and ELF, you would either need another loader or binaries that look like both formats at the same time. The latter isn't really possible (except for some interesting puzzles) the former is what binfmt support in Linux does. On other OSes, you do that manually like in "java foobar.class" or "wine something.exe".

The syscall convention is another problem, every OS does them somewhat differently, and even the same OS on the same architecture can have different conventions one may use: int 0x80 (or whatever number one likes), call gates, SYSENTER, traps or trampolines. I certainly forgot a few. Within those there is another variability around parameter passing and return values (register, stack, magic location, all of the above), buffer handling for larger parameters and preserved registers. Syscall names, numbers and semantics are all totally different. Then there are mechanisms like unix signals and callbacks from the OS, futures and zero-copy-IO that are very very OS specific. There also is absolutely no smallest common denominator.

Memory layout has peculiarities, like randomisation, different sections, who sets up the stack, that kind of stuff. Mostly the linker and loader will do things here, but again, it depends.

And then there are libraries. Even if you use a static binary (which you probably would have to), modern OSes still expect you to dynamically link some libraries like all syscall entry points or stubs, things like libnss, etc.

All in all quite impossible without at least another layer like wine or the java vm, some big loader switch/case and quite a bit of fiddling to adapt to some hard-to-adapt peculiarities. E.g. the Linux kernel has quite a few things built specially for wine to work.

CloudABI (https://github.com/NuxiNL/cloudabi) was an interesting attempt at doing just that. However it seems it didn't gain enough traction and people involved veered towards wasm/wasi which also abstracts over the instruction set (at the expense of performance, unless I'm missing some development on the AOT front)

I think the core problem is that actually compiling for different operating systems is essentially none of the work needed for compatability and provides a stage that lets you remove stuff that isn't needed for that OS, while using bytecode makes it possible to support different instruction sets and can be compiled for the platform it is used on as needed or ahead of time.

Going the other way, if an application is distributed as a VM or other method of minimal reliance on the host OS it has all the issues of static binaries (heavier resource use, can't benefit from general updates) plus it won't have the look and feel of different operating systems unless it includes all that complexity on every OS. There are a number of other compromises that could be made vs. the most commonly used ones, but it is all a matter of tradeoffs. E.g. Flatpack has a set of base images as well as the application image, but it seems to have issues with the availability of dependencies.

The closer operating systems are the easier it is for a single executable to work. The BSDs have have been able to run binaries from various other operating systems, however as the other operating systems change it a lot of work to keep up and some OS APIs are just bad ideas that would compromise the operating system if they are adopted (in some cases a better way to allow compatibility could be created with a lot more work).

One option that I am interested in that might gain some popularity at some point would be to run a much simpler OS in a VM that can do some of the useful things you want a computer to do while not trying to do all of them but without trying to make individual applications fit in with the host OS. This would require users to learn two operating systems but would make the small OS easier to understand in detail and would have some significant security advantages due to being able to impose restrictions that would excessively restrict a general purpose OS. However, there will always be pressure to integrate more into the host OS. A major issue limiting this possibility right now is the situtation with GPU access from a VM. Also, commercial OS vendors are more than willing to make gratuitous incompatabilities to limit any alternative that starts to grow in popularity.

Everything that needs more than a single process won't run, either because it needs a system service or because it does multi-process-sandboxing. Single-address-space is extremely limiting for a lot of stuff, so most webservers are out. Not sure if threads would even work. Any desktop software is out anyways. Basically you will have to write your own software for this model of operation, or pick stuff that is accidentially suited, which won't be a lot.