A Go unikernel running on x86 bare metal | Hacker News Reader

A Go unikernel running on x86 bare metal

github.com

177 pointsLealen42 comments

Honest question: why use a unikernel on bare metal?

I can see the lure on top of an hypervisor. You already have an actual kernel running which you can use it to administer the machine. Why pay the performance price of a full kernel in your vm if you are only going to run one application ?

On bare metal however, you will probably need to bundle a ton of things if you don't want to end up with an unmanageable device. That leaves the performance gain of compiling everything together or actually disposable devices (maybe for embedded?) as potential use cases I can envision. Is there more ?

Why pay the performance price of a full kernel in your machine if you are only going to run one application?

I'm not the author, but running on bare metal is more fun than running in a hypervisor. And removing layers from your stack lets you see what the cost and value of the layers are. I've only done bootstrapping on x86 and 8-bit processors (which is just set your reset vector), but x86 has a lot of isoterric setup stuff, and learning about the magic is interesting, if not totally useful.

Also, some runtime environments have a lot of manageability within; probably less than an OS, but maybe enough. Not sure about Go.

One reason to avoid bare metal is drivers. There is a lot of hardware around. New hardware is released every day. And you need to write and maintain drivers for that hardware. That's a tremendous effort and hardware manufacturers usually write drivers for a very few operating systems and Go unikernel certainly won't be one of them.

Using virtualization provides a simple common layer, so host operating system can deal with drivers and guest operating system can deal with more interesting things.

Of course you'll need to write a OS specific driver for critical hardware like interrupt controllers and system timers, but drivers for other things like network cards and disk controllers can be reused if you adopt the other OS's interfaces. Things like ndiswrapper exist to use windows drivers on Linux or FreeBSD; Linux, FreeBSD, and NetBSD also have a large pool of drivers, which can be used depending on which system's driver model you like best.

Let's say I grabbed the e1000 driver from Linux, how much of the Linux kernel would I end up re-writing (badly)?

I haven't done it, and the e1000 driver is pretty big, so it's hard to scan quickly, but the general shape is you'd need to support the kernel functions the driver calls, and setup compatible kernel structures for the driver. At least the parts of the driver you call.

Things like kernel malloc to setup pages and what not, mapping the device address, allocating interrupts, DMA (maybe), PCI, logging, maybe locking, basic c library stuff (memcpy etc). Some of those, you can probably just stub out, but some if it you have to do (badly or not). Some of that you probably need or want to build anyway.

It's a fair bit of work, but if you want to support a lot of hardware, it's probably less work than porting a bunch of drivers.

The e1000 came to mind because it's a commonly emulated network card. I wonder now what is more common in real servers.

e1000 is pretty common in servers, too. It's got a lot of variations though, which makes the full driver pretty big. virtio-net is a lot easier to write a driver for, of course. :)

Why not just grab some other parts of Linux? Or, better yet, link your application right into it?

How much benefit would it be over stripping an existing kernel, e.g. Linux/BSD, remove all modules that you don't need, tune schedulers for one application etc?

Hard to say, you'd really need to do both well to compare.

Depending on the unikernel design though, you can significantly reduce user/kernel context switches beyond what you can with a tuned general purpose OS, potentially to the point of always sitting in ring 0. How much difference that makes, of course depends on your application and the quality of your various kernels.

Ideally, running on a hypervisor is absolutely identical to running on bare metal, and any difference the guest can detect is a bug in the hypervisor and/or the hardware. That was the value proposition of CP-40 back in the 1960s, right down to being able to run newer versions of CP (later called VM) as a guest on older versions, in order to develop and debug the next generation of hypervisor in a known-good environment which is, again, utterly indistinguishable from running on bare metal.

https://en.wikipedia.org/wiki/IBM_CP-40

I wonder how long it'll take the rest of the world to achieve that.

Embedded definitely uses unikernels a lot (though often with the term "Library OS" for any OS that's statically linked to the single application it runs). Hypervisors add overhead that you often can't afford on a microcontroller.

Embedded systems are bare metal. There are tons of usage. Bare metal can be anything from a 8bit microcontroller to multicore Ghz system. Every modern RF modem has a cpu, and uses some kind of unikernel or rtos, if you have smartphone You have at least one such a system running.

We fully expect and are planning for multi-tenant applications on devices similar to a rpi4 looking at the future so I firmly agree that flashing the device isn't really desirable.

As an example if you have image inferencing project you might have a compute stick co-processor plugged in one slot and a camera in another. You might have a webapp as one application and ffmpeg in another. No reason not to isolate them as individual unikernels. Also, there's the case that one software team might have written one of those apps and a different team wrote the other.

What are the differences between Unikernel and RTOS? Both are lean. Except for the fact that RTOSes are usually coded in C/C++ mostly. And Unikernel implementations are present in various languages. Other than that what is the difference? Moreover, RTOSes provide all those Real-Time guarantees.

This is fascinating to me as I reorient https://go-micro.dev towards something of an embeddable distributed systems framework. Really cool work, thanks for sharing.

Hi Asim, your project is really interesting to me. Just curious, do you have go docs or tutorials hosted somewhere?

Not at this time. Google's pkg.go.dev only renders a subset of licenses and I now use polyform noncommercial. There is a repo of examples github.com/asim/examples for v2. With v3 I'll look to create some new examples and docs.

The README is concise and well written in my opinion, I was wondering how useful it would be without standard libraries before I clicked the link, the first paragraph says it has support for network libraries and went on to show Mario running in this!

This would be interesting if it could run on a raspberry pi.

I see it is only x86 right now.

If this could run on ARMhf/ARM64 it would be the holy grail of appliance-style hardware.

I suspect this would require incorporation of some rPi-specific drivers, so it's a hard sell. Nevertheless this is perhaps more possible for rPi than for any other type of device because it's a well defined hardware target with a fixed BoM with no major part changes through the supported lifetime. I would support a kickstarter aiming to produce this for Pi.

The RPi SoC also has nearly no real documentation, unless you are Broadcom or NDA'd by them.

Nanos is specifically targeting ARM64 for the raspberry pi 4 and expect that support to land formally very soon. (I'm with the project).

Anyone's using this in production? Or any other unikernels?

tl;dr: a unikernel means there's no standard OS other than the runtime execution environment specifically just for Golang in this instance; which means it should be much faster and much more lean (at the price of not having standard tools like SSH and all)

Not yet but planning and executing towards it.

To add to others points and your own which may not be as obvious to some but my background stems from the financial world and security has always been a problem, breach du jour. The way things are done now are nothing short of broken, PCI and more, and all the glue holding the shattered "window" panes together certainly doesn't lead to anything secure or even a "window" you can see through. I'm certain there are millions that will disagree, job security of course, but when one has maintained critical uptime services on M$ as long as I have one begins to question there must be a better way. Solve your own problems first I always say as I am currently architecting that path, the sixth time of my career actually but this occurrence is different entirely.

Very curious. Can you elaborate more? Do you use Go in the finance world, and hope to use the unikernel to lower your attach surface?

I didn't realize Go was a very common language in finance.

I am building an new way to conduct a technology business end to end in entirety, hardware through software, THE BIG PICTURE. Obviously these are my words and likely further explanation could be redefined in other terms based on ones understanding however I have done this several times before with extreme security and efficiency in mind at every decision made, AUTOMATION because PEOPLE ARE THE PROBLEM. My background is payments, finance and banking which I need not elaborate the security that has been in this arena alone for quite some time but anyone nowadays, even a 7 year old, can use any search engine and find a recent breach referencing lost financial information and EVERYONE reading this has likely been issued multiple new credit cards or worse. There are several people and companies working towards unikernels and the like but my jam, and always has been, is to apply the new way of thinking to a business, a serial puzzle master if you will.

In short you are correct in using ONLY GO for absolutely EVERYTHING to reduce technical debt, ZERO TRUST architecture, automated everything and much much more. Might sound like hogwash to some but once more, if my past is an indication of my future as I love a challenge.

It is what you cannot see that matters most.

I would suggest looking into OCaml unikernels if you work in finance.

At least one or two order of magnitude more systems are using unikernels or rtos rather than a full-blown os. You can write a small scheduler even for a small 8bit microcontrollers (avr). The main purpose of the unikernels or small rtos are handling multiple running contexts (aka threads or tasks). I wrote several smaller kernels for embedded sytems, before starting threos.io . It's much easier to handle multiple tasks. ex. one of our project was a wireless voting terminal, one task handles the communication (RF) interface (protocol, error correction etc.), the other handles the user interface (buttons leds etc), and one task is handling the vote application program or logic. The tasks are communication over message passing. All of this running in a cortex-M3 with 8k-32kb ram. A more common example are wireless RF modems (WIFI, GPRS, GPS, Bluetooth etc.), nearly every RF modem uses some kind of unikernel or rtos.

FreeRTOS is a unikernel by that definition (by every definition I've seen, though they still use the term "Library OS" which seems essentially synonymous). It's quite common in embedded devices.

By that definition, older unixes were unikernels for the C runtime.

Well, the usual definition I see for unikernels also stipulates that there's only a single process. So FreeRTOS is still a unikernel, but early UNIX wouldn't be.

Though the distinction between "process" and "thread" gets very fuzzy when you have only a single address space.

Well, that's not entirely wrong. After all, the libc is pretty much a part of Unix API today.

Actually is the other way around, libc is the part of UNIX API ISO was willing to put into the language standard, while the remaining part of the C UNIX runtime ended up as POSIX.

u/eyberg's https://nanovms.com/ look promising but they aren't self-service right now.

I work a little on nanos..it clearly needs alot of usability work, stability, and feature work. but could you say explicitly what you mean by 'self service'?

If I've got some code I want deployed to nanovms cloud, say, there doesn't seem to be a way to do so. All I see on the website is "schedule demo".

We don't actually do any hosting. This is a very common misconception.

OPS - https://ops.city can deploy to any public cloud and can run on openstack/kvm/vsphere as well.

Yes I'd very much like to know if there's a JDK or rust unikernel someone has used in production - commercial or open source.

https://github.com/nanovms/ops-examples/tree/master/rust/01-...

https://github.com/nanovms/ops-examples/tree/master/java

to get you started - i don't know any of our users that are using those in prod though atm

These guys do all sorts of embedded deployments for Java, bare metal, real time OS or just plain Linux.

https://www.aicas.com/wp/

https://www.ptc.com/en/products/developer-tools/perc

http://osv.io/

Reply on news.ycombinator.com