OSv – Linux binary compatible unikernel for virtualized environments

The biggest problem is that context switching overhead becomes the main "work" being done, rather than the actual work of the threads, for some core count n (which depends on workload).

You can already saturate an entire core just copying data from a NIC. Of course, enter BPF and all that jazz. Plus it doesn't play as nice with BIG.little multiprocessing (AMP - asymmetric multiprocessing).

edit: I should note I am not advocating getting rid of preemption entirely, because it is needed for at least one processor, but having a fixed quantum causing a context switch on application CPUs is highly disruptive, both from the operation itself, and from blowing the cache all the time. Incidentally, one of the main reason for green threads.

One modern systems, preemptive multitasking has a surprisingly high overhead. Much of your CPU time can be spent on context switching and the associated cache thrashing, which isn't productive. Furthermore, many kinds of macro-optimizations based on temporal locality stop working because preemption can randomly inject badly chosen computation into your carefully crafted computational sequencing. Preemptive multitasking makes it more difficult to ensure robustness under load generally because the default behavior of the system is that it will happily do the wrong thing at the wrong time under stress, because the OS has no idea what the right thing is, which can aggravate the situation. Obviously, none of this applies to systems that are typically under light-to-modest load.

For servers with hardware capable of very high bandwidth I/O, preemptive multitasking is a major drag on performance and scalability. Hence why "thread-per-core" software architectures are idiomatic for high-performance data infrastructure software these days. You don't just gain significant absolute performance, the behavior can also degrade more gracefully under extreme load.

Like with all things, it depends on what you are doing. I've designed database engines both ways and Linux is not opinionated about such things.

You use the native cloud facilities available by spinning up a new instance per deploy. Whether you need a simple shared thread (smallest instances available) or you want to tackle the 368 thread monsters available on something like GCloud both are fine.

It's worth pointing out that if you run containers on public cloud you are inherently virtualized already - this actually removes layers instead of adding layers on.

Docker is dead. (OK, to be fair, it's still in the process of dying).

As Docker don't solve isolation you need to run Docker in VMs anyway, so actually people start to think how to get rid of this unnecessary overhead. The foreseeable future are microVMs (the hype is just heating, projects show up "everywhere"). And in the long run unikernels could become the "packaging format" for those microVMs (but they have to mature first as ease of use, und performance / stability are not there right now).

"reduces efficiency and increases complexity". Sorry there.

You've got it backwards. Containers strenghen security. Container mgmt runtimes executing as root causes security problems, like the setuid programs of old.

Note that the first reply to the HN comment you linked to points out that the poster misread the graph.

very cool stuff, thanks for sharing

very interesting stuff, thanks for sharing

That's pretty awesome! Did you write it up anywhere?

How do they do fork()?

Any particular reason why they benchmark against Linux 3.13 a 7 years old kernel?

Here's a post from one of the OSv developers on performance compared to Linux in 2020 [1]. It paints quite a different picture compared to those benchmarks from 2013.

>While ideally a unikernel like OSv could provide better performance than traditional kernels because of things like lower system call and context switch overhead, less locking, and other things, there are many things working against this ideal, and resulting in disappointing performance comparisons:

1. Most modern high-performance software has evolved on Linux, and evolved with its limitations in mind. So for example, if Linux's context switches are slow, application developers start writing software which lowers the number of context switches - even to the point of just one thread per core. If Linux's system calls are slow, application developers start to batch many operations in one system call, starting with epoll() some 20 years ago, and culminating with io_uring recently introduced to Linux. With these applications, it is pointless to speed up system calls or context switches, because these take a tiny percentage of the runtime.

2. They say a chain is as weak as its weakest link. This is even more true in many-core performance (due to Amdahl's law). Complex software uses many many OS features. If OSv speeds up context switches and system calls and networking (say) by 10%, but then some other thing the software does is 2 times slower than in Linux, it is very possible that OSv's overall performance will be lower than Linux. Unfortunately, this is exactly what we saw a few years ago when ScyllaDB (then "Cloudius Systems") was actively benchmarking and developing OSv: Many benchmarks we tried were initially slower in OSv than in Linux. When we profiled what happened, we discovered that although many things in OSv were better than Linux, one (or a few) specific things in OSv which were significantly efficient in OSv than in Linux. This could be some silly filesystem feature we never thought was very important but was very frequently used in this application, it could be that OSv's scheduler wasn't as clever as Linux's to handle this specific use case. It could be that some specific algorithm was lock-free in Linux but uses locks in OSv, so becomes increasingly worse on OSv with the more CPUs you have. The main point is that if an application uses 100 different OS features - Linux had hundreds of developers optimizing each of these 100 features for years. The handful OSv developers focused on specific features and made clever improvements to them - but the rest are probably less optimized than Linux.

3. Many-core development is hot When the OSv project started 7 years ago, it was already becoming clear that many-core machines were the future, but it wasn't as obvious as it is today. So OSv could get some early wins by developing some clever lock-reducing improvements to its networking stack and other places. But the Linux developers are not idiots, and spent the last 7 years improving Linux's scalability on many-core systems. And they went further than OSv ever got - they support NUMA configurations, multi-queue network cards, and a plethora of new ideas for improving scalability on many core systems. On modern many-core, multi-socket, multi-queue-network-card systems, there is a high chance that Linux will be faster than OSv.

4. Posix API is slow This is related to the first issue (of software having evolved on Linux), but this time for more "traditional" software and not really state-of-the-art applications using the latest fads like io_uring. This traditional software is using the Posix API - filesystem, networking, memory handling etc. that was designed decades ago, and make various problematic guarantees. Just as one example, the possibility to poll the same file descriptor from many threads requires a lock every time this file descriptor is used. This slows down both Linux and OSv, but not giving OSv any advantage over Linux, because both need to correctly support the same slow API. Or even the contrary - Linux and its hundreds of developers continue to come up with clever tricks for each of these details (e.g., use RCU instead of locks for file descriptors) while OSv's few (today, very few) developers only had time to optimize a few specific cases.

So it's no longer clear that if raw performance is your goal, OSv is the right direction. OSv can still be valuable for other reasons - smaller self-contained images, smaller code base, faster boot, etc. For raw performance, our company (ScyllaDB) went on a different direction: The Seastar library (http://nadav.harel.org.il/seastar/, https://github.com/scylladb/seastar) allows writing high-performance applications on regular Linux, by avoiding or minimizing all the features that makes Linux slow (like context switches) or cause scalability problems on modern many-core machines. Initially Seastar ran on both Linux and OSv (with identical performance, because it avoided the heavy parts of both), but unfortunately today it is using too many new Linux features which don't work on OSv - so it no longer runs on OSv.

[1] https://groups.google.com/g/osv-dev/c/0WveOhAsZEU/m/jqHyNs9l...

I'm just curious. I like the idea to use software for things they were not originally intended for.