Linux compatibility in IncludeOS | Hacker News Reader

Linux compatibility in IncludeOS

includeos.org

85 pointsingve34 comments

Claiming that they've "implemented" 42 syscalls seems like an overstatement: almost everything is a stub that does nothing, and the bits that are implemented are not even close to being half-baked. Even relatively easy stuff like dup is missing.

The number of system calls says close to nothing of the actual API surface present here. It's a poor measure. What actually matters is: what can I run?

That said, 42 is not enough to run a lot of applications.

It'll be interesting to get to the point where they can run a language's runtime or unit tests.

Take it from someone who worked on gVisor: this is gonna take a long time, and eventually you'll end up with a system that's just Linux. You'll end up either being bug-for-bug compatible, which means bringing in loads of compatibility code and architecting your kernel just like Linux, or you'll end up having to change applications (and then you're not all that compatible anymore).

Neither are fun or sustainable compatibility stories. You'd probably actually get Linux apps to work on a random OS faster & nicer by porting gVisor.

> Moreover, since implementing a system call is typically pretty simple [...]

You're in for a fun ride, folks.

Given what I just read about IncludeOS (I'm not really familiar), it feels like implementing as much of musl as possible is probably the best you can do compatibility wise. There are going to be a lot of Linux features simply incompatible with includeOS, so claiming that 400 syscalls will be implemented is unlikely.

That said - why do you have to get musl compatibility by implementing the Linux syscalls? Is contributing to musl an option?

Your assumption is correct. IncludeOS will never be fully compatible with Linux. Our goal is to provide "just enough POSIX" so that the applications that are relevant for the OS will compile and run. Mostly we're concerned with various "gateway-like" applications - firewalls, load balancers and down the line also IoT-gateways.

Wrt. to why we have to implement Linux-like syscalls; Musl is written specifically for Linux so in order to "unlock" the relevant POSIX-bits that Musl can provide we have to provide the underpinnings that Musl requires.

I'm hoping we can upstream some of our non-intrusive changes to Musl. However since Musl has a very specific purpose (a Linux-specific C and POSIX library) I'm not certain that our changes are within the scope of the project. To date we've only concerned ourselves with a small subset of Musl and only on amd64.

(I work for Includeos)

I don't think we need to run everything. I think we'll be happy if we can take some common libraries and compile them and they work. That is currently our goal. Down the road, depending on interest, we can perhaps expand a bit.

>> Moreover, since implementing a system call is typically pretty simple [...]

> You're in for a fun ride, folks.

Thanks for the heads up. It was meant as opposed to implementing the syscall and layers on top of it.

That sounds much more attainable than what the blog post suggested - sorry if I took that the wrong way!

No worries. I never meant for the blogpost to reach an audience outside of our pretty narrow knitting circle so I didn't pay much attention to detail. I very much appreciate your insights here.

Do you recommend gvisor over dune or kata containers?

If you're into unikernels, rumpkernels[0] are the least-modification-required approach I've seen.

Basically, if you compile your program with a minimally (as minimal as is possible) modified compiler, you get "rumprunnable" output. there's a list of popular packages built to run on rump kernels[1] (Rust is supported[2]). I haven't tried to use the toolchain yet, but it's pretty appealing to me.

The biggest issue is that rumpkernels seem to not be in developed anymore. I can't tell if the entire is just off somewhere being productive with their rumpkernels and are happy or if no one is around at all. The Github organization[3]'s repos all seem to not have been changed in a while either.

Outside of rumpkernels, Nabla containers[4] also look interesting to me. They're built on Solo5[5], but the only microkernels on there are currently MirageOS (for OCaml) and IncludeOS (C++?). Nabla looks amazing (containers look like a great fit and they seemed to have improved ergonomics massively), but the limitations[6] are intense.

[0]: https://github.com/rumpkernel/wiki/wiki/Info:-FAQ

[1]: https://github.com/rumpkernel/rumprun-packages

[2]: https://github.com/rumpkernel/rumprun-packages/tree/master/r...

[3]: https://github.com/rumpkernel

[4]: https://nabla-containers.github.io/

[5]: https://github.com/solo5/solo5

[6]: https://github.com/nabla-containers/runnc#limitations

I'm hoping we're be able to port libuv to IncludeOS (which should be doable, much like libuv IncludeOS is eventdriven). Then we should be able to get Node up and running. Running Node.js through IncludeOS on Nabla could be interesting in a "FaaS" context.

Last I heard the author of Rumprun has picked up beer brewing full time and isn't interested in maintaining the project.

It would be great if someone where to pick it up.

OSv is also very relevant. They provide a much richer and more compatible runtime environment compared to IncludeOS (I think they even have zfs). And OSv is pretty active these days, so if you looking for a way to run some high performance stuff OSv is worth looking into.

Thanks for shedding some light on the possible status of the Rumprun maintainer! Hope he's happy brewing beers :)

Thanks for mentioning OSv[0], I'd never heard about it before.

I'm excited to see what happens with Nabla, I read the accompanying paper and was pretty excited about the possibilities. Nabla's greatest strength is it's ergonomics, IMO, though rumprun comes close (someone took some time to make dockerized containers for it).

[EDIT] - looking at OSv's (primary?) tool capstan[1], I'm not sure it's quite the same, it seems more like linuxkit[2] than rumprun. Maybe my intuition is wrong, but it seems much more like you're bringing along a stripped down OS than you're writing a program meant to interact with a shared unikernel. The distinction is so fuzzy in my head I'm not sure there is one, but it feels different.

[0]: http://osv.io/

[1]: https://github.com/cloudius-systems/capstan

[2]: https://github.com/linuxkit/linuxkit

Nabla containers actually use rumprun under the covers. Here is the rumprun fork that supports a solo5 backend: https://github.com/nabla-containers/rumprun

The article is "broken", it talks about Musl and then says the found Musl.

There’s a lot to like here.

1. Focusing on compatibility is smart because it’s the weak point of IncludeOS and other unikernel projects.

2. Musl is an incredibly underrated project and I hope one day it becomes the new standard libc.

This is just a first stab, but it could turn into something great.

I wonder if gvisor and includeos could be combined in interesting ways, or at least steal from each other’s code and ideas?

This is an exciting development that I am watching closely.

It should be noted that whatever the limits of the 42 system calls, network operations are very well supported. Many useful programs hardly need anything else.

Would it be possible to run Linux as a unikernel and port your application as a kernel module? How a leaned down Linux kernel would compare to includeOS?

There is a project that tries to turn Linux into a library which could lead to a unikernel approach with the Linux. If you look at https://github.com/lkl you can see where they are.

A leaned down Linux-kernel with the application linked into kernel space would be pretty comparable. But it would be pretty enormous beast.

You can compile the Linux kernel with only the modules necessary for running it in a virtualized environment (ie, only virtio modules) and simply include your application in the initramfs as init. If you include nothing else, that's about 6MB + your application.

I'm pretty ignorant on Unikernels so maybe there's another solution, but not being able to `fork()` is a significant problem for many applications.

My understanding is that unikernels are single-application by design. I agree that a mechanism to spawn other apps is useful, but in this case it won't be a simple fork but the creation of a complete new os instance.

I suspect GP is referring to single applications that aren't single process. Postgres is one example. Vacuum and checkpoint are separate processes.

Applications also sometimes have separate processes for sentinal functions, logging, etc. Or default configurations that are multi process, like many load balancers, web servers, API gateways, etc.

There are plenty of applications that used multiple processes.

Absolutely. And most of those are deemed out of scope for Unikernels.

On Unix you typically fork as a part of initializing your application in order to demonize. Or when you are executing another program so a fork/exec.

The first one isn't needed in a unikernel. You are already demonized when you boot up.

The second use is not available to unikernel applications. The limitation of a single application per vm applies.

Looks like Linux syscalls are becoming some kind of de-facto standard. They are already implemented by Linux, FreeBSD, Windows and now IncludeOS.

The mind reels.

Not because Linux is a bad kernel, mind you, but because UNIX itself reeks of ideas that were ok when it was originally developed and clunky at best today. Even Microsoft seems to have given up. The future of computing looks pretty grim to me.

Could you elaborate?

On what? Reasons UNIX isn't so great? There was an entire handbook written about it decades ago, much of which is still relevant. Text isn't a universal interface, files are not a good abstraction for everything (not the way UNIX does it anyway), the permissions system is extremely limited and quite backward for modern problems, the way the file hierarchy is organized hasn't made sense for about 30 years, POSIX has a lot of well known issues, etc.

Two specific errors that are very well-known:

* Filesystem semantics. What POSIX actually says and what most people think it says in this regard is very different. And both groups acknowledge that the semantics are not very useful for things like high-performance distributed filesystems in HPC.

* Signals. Everything about signals is pretty annoying. You can't install per-thread signal handlers, which means you can't do things like "catch" segmentation faults. And signals aren't files, but you can only poll for file things to occur, so you have to reexpose signals as files via signalfd...

What does fork() even mean in a modern multi-threaded process?

A lot of new kernel features have just given up to the point where current advice is to never call it unless you plan to exec immediately after. But the technical debt within the kernel for this misfeature is massive.

Can someone point me to a more detailed overview of IncludeOS than the brief summary they have on their home page?

For a technical, in-depth overview of IncludeOS and how it was made I find the main authors presentation at Cppcon 2017 is a pretty good introduction:

https://www.youtube.com/watch?v=h7D88U-5pKc (one hour, quite entertaining).

There is also the original paper which was presented at the 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom).

https://ieeexplore.ieee.org/abstract/document/7396164

So, what's wrong with newlib?

Nothing; it is just much smaller by not providing POSIX and Linux like interfaces.

Alpine proves full musl can already run a full distro's worth of programs and is very easy to port anything using glibc.

Newlib tends to be used in projects like Cygwin as a small C standard library, where the rest of the POSIX API is implemented separately (by proxying to Win32 system calls for Cygwin).

Reply on news.ycombinator.com