Nanos – A Unikernel
nanos.org
nanos.org
> Yes, but we caution users to evaluate if you really need kubernetes. Chances are you don't and you will experience severe performance and security problems if you choose to run under k8s. If you still find you must here are instructions for running Nanos under k8s.
Pretty interesting FAQ! I hope to see more HN discussion about this page!
I'm here wondering, what security problems could this have? https://docs.ops.city/ops/k8s doesn't elaborate and just says
> Security Warning
> Running unikernels under kubernetes diminishes some of their security benefits.
That's not the case when you are running in k8s and the last container breakout was just announced ~1 month ago: https://github.com/opencontainers/runc/security/advisories/G... .
At the end of the day it is simply not a security boundary. It can solve other problems but not security ones.
That's an odd, perhaps presumptuous, claim to make considering this isn't even in the same orchestration/scheduling space as Kubernetes. Especially without mentioning alternatives.
From later in the FAQ:
> The complexity that comes with kubernetes is that it requires you to re-invent all the layers of a cloud platform that already exists. If you run a vanilla linux instance on AWS you get out of the box: networking, storage, security, routing, etc all for free.
They might not like the abstractions, but to say you have to "re-invent" all those layers is reaching.
It’s currently being implemented as an abstraction on top of existing systems, because cloud providers have to leverage what they have and because its API still lacks all the functionality. But gradually it became a managed service that can provision databases, etc, and in another 5-10 years most cloud interactions will happen through it
I wish to encounter some unikernels in production sometime.
Now, in cloud native we have decomposed applications - logging is now separate, so is storage, etc.
I should be possible to change the stack underneath without anyone noticing, or at least with minimal disruption to operations.
its a shipping container, an adaption jig, an invitation to rethink the world underneath without insisting that anyone rewrite anything.
also devs: let's add just one more layer on top of linux -> docker -> k8s
godspeed to the nanos team for trying to simplify the stack
I was running a web application written in ruby, distributed in a container, running in docker on linux in a VM. That could become a unikernel running directly instead of the VM. Saves quite some layers i'd say :)
Nanos is a single process operating system designed to run as a virtual machine and has no support to run on hardware.
right now we don't have any plans to support bare metal
installs like this as that would imply a bunch of other
mgmt related tooling that would not be present
(eg: start/stop the server, configure networking,
deploy a new one, access rights, etc.) it also breaks
the assumptions we have that it is only being deployed
as a vm which means having to support a ton of random
hardware drivers, nanos is intended to always be ran on
top of a hypervisor of some kind - whether it's public
cloud or something under your own control
(eg: proxmox/vsphere/etc.)
___
It minimizes the software stack (and with that: attack surface) that application sits on, inside a VM.
It does not (nor is it expected to) help to minimize said application.
And it does not minimize the software stack that runs the VM.
There's a lot that an operating system brings to the table, that casual users may not be aware of. They tend to have a bunch in them not because "why not" but because it's providing value.
I mean -- isn't that all the good stuff?
Like -- I boot this thing and I don't really have a filesystem in the way we know it. First of all, that's confusing (I say this as a ZFS devotee). Second, I have no idea why this is acting crazy in production, or in testing, compared to the Linux version? Etc., etc.
This is a rather common misconception. Nanos (and many other unikernel projects) have filesystems. Most of the applications we target are webapp servers, databases, etc. They all want to, at the very least, write to a tmp file and more commonly want to load lots of files. What they don't have is an interactive userland.
Logging and observability are valuable. But running a full multiuser OS with kernel and userland for your one process, adding extra context switches and what have you to everything you do, just for the rare occasions where you log in and run a few commands, seems crazy to me. As long as it can output logs/traces/etc. to a collector (which is what I'd want to do anyway, no-one wants to have to SSH to each separate instance to look at log files on the local filesystem) and there's a way to attach a debugger (e.g. the Java style where your debugger connects to it on a given port), I don't see any advantage from having e.g. a filesystem per se. Likewise being able to run it locally the same way as production is important, but that doesn't have to mean running it as a Linux process - people are happy running Docker images for local dev, running a unikernel in a VM isn't a lot different from that.
If I want to have a bunch of instances scaled up and wired together network wise and defined in a nice yaml file, kubernetes makes this easy and replicable and I just have to push a container image somewhere like GHCR and have it all rolled out with CI.
You can deploy almost the same system on top of the managed kubernetes offerings from any of the big cloud providers, so other than working out a few differences generally your config will work on anything.
If I want to do this with unikernels I'm lost, would I do it with AWS services like EC2 and auto scaling groups or something? How do I get it all wired together and configured? Perhaps terraform but I'm not sure.
It's all about the maturity of the glue and the architectures, not to mention training and familiarity. Does (mystery unikernels solution) have the equivalent of custom controllers somewhere in that stack?
K8s provides an abstraction layer over resources and lets you define your system holistically.
If this exists in a mature and flexible state for unikernels I'm all ears.
What real advantage is there to using unikernels over alpine linux + k8s (or whatever) orchestration?
This is a key point that many many people struggle with until they actually push their unikernels out to the cloud. It is one thing to run ops locally but try doing a 'ops image create -t aws' followed by 'ops instance create -t aws' and then you'll understand how we manage to push a lot of the scheduling work back onto the cloud provider itself.
It would be cool to write some sort of book like, ops for k8s people, where it takes a lot of the common concepts in k8s and explains how you'd do them with this system.
ops seems like decent a CLI to get things done but as far as I can see not really an orchestration system in the same way as k8s? Perhaps I misunderstood.
There is a terraform provider that we ended up writing for one company but only because they were a (large) hashicorp shop and nothing went to prod unless it was in tf: https://docs.ops.city/ops/terraform This doesn't really do much though. It's only there to check a box and fit into an existing paradigm.
A lot of the orchestration bits that k8s provides is because when it was initially adopted there wasn't a great way to attach networks/storage/etc. to various containers like people had for on the clouds with vms. OPS just re-uses the same primitives you'l find on every single cloud out there. If you need more advanced functionality like auto-scaling that also can utilize the clouds ASGs.
I guess this means your environment is very "native" to a specific cloud however and not as easily transferrable.
One thing I did like about k8s is that it's easy to self-host with microk8s but tbh, I think in production you'd be using a cloud provider's managed k8s (e.g. AKS) anyway.
Looking at how to attach/configure a ElastiCache, RDS, or some EFS to a project also.
What's the best way to runtime configure an image when you create it as an instance? I couldn't see any options on ops instance create for environment variables or anything like that but this also seems like, weird - IE you wouldn't give a VM environment variables, but with a normal EC2 instance there's an agent that handles this sort of thing. Would you have to bake the configuration into the image? Also how does one output the ELF for the image? Is it just the file of [image name] in ~/.ops/images ?
Or is it the case that terraform should build a correctly pre-configured image as part of the apply process?w
Always good to have other options, of course! I do appreciate the minimalism. This is exciting :)
The terraform docs example uses GCP, perhaps if I figure out how to do it on AWS I could PR the docs with an example.
At that stage where my questions lead to more questions.
I wonder why Alpine linux won over this though?
[1] https://bcantrill.dtrace.org/2016/01/22/unikernels-are-unfit... [2] https://docs.ops.city/ops [3] https://oxide.computer/podcasts/oxide-and-friends/1184552
The closest I've found was a Reddit article titled "Stardust Oxide: I wrote a unikernel in Rust for my bachelors dissertation" with this comment "My impression, after investigating Singularity for my own undergrad: No. They did bytecode varification against a manifest after codegen, in their own intermediate language that all installed programs had to use. For the Rust compiler I imagine you're thinking of doing certification against the module sources? That's not going to provide the same level of verification."
"Possible to formally verify critical components" https://www.cs.williams.edu/~cs432/osco/01-whit.pdf
"As part of our mission to build robust and secure systems, we strive to support technology that roots its design in the field of formal methods" has a TCP stack manually derived from a formal model https://tarides.com/blog/2024-01-24-mirageos-designing-a-mor...
* by learning system I include any system that learns from experience
* https://nanos.org/ The core technology?
* https://nanovms.com/ A company providing services and offerings around said technology?
* https://ops.city/ The orchestration?
And I am not sure what "thing" I am using. Is there some disambiguation? I know is OPS is the orchestration CLI, but I am confused at the difference between Nanos and NanoVMs. What should I call the section of my README that deals with this tech? Currently gone with Nanos/OPS but I am confused.
Great idea, and love the security context, but I feel like you loose so much.
Logging for example, is it all syslog? How do you manage them? What if you need them in splunk, how does that work?
I love the idea, but can’t figure out how to use them in practice.
Unlike Kubernetes, unikernel images are way slimmer and faster.
There is something nice about them as your mental map is reduced to the minimum amount of components, your app - amalgamated with the unikernel - and the virtualization platform of your choice.
I need to test support for volumes, and if there is interest I might do a write-up about it.
How do I package my own applications.
Say I have a binary “helloctl” that I want to package, where do I read about how to do this?
Took a look at the docs and it is neat with details about nanos but no info about how to use my own binaries, except install ops and run the sample nodejs app.
I could of course install ops and try to look around for other pkg commands but a basic “get started” should show me how to package my own stuff. Or may be I am just too lazy :)
https://github.com/nanovms/ops-examples
If you're looking for a particular piece of software search on the repo first:
If you don't find it and need help creating one ping me or open an issue in ops.
Online judge platform do a very small subset of problem. You can sandbox it to no network or filesystem accesses, and no syscalls except the few like read/write/select.
Sidenote: I'm not really looking to replace it, I was just asking out of curiosity since this is my first time hearing of unikernels
Yes. This is the fastest you can get.
If you want safer, add pr_set_seccomp _in addition_ to it. but that would be a custom solution.
What about compared to low-latency and real-time linux kernels?
Having said that with enough interest I think we could look at project such as https://projectacrn.org/ but it's not really a focus at the moment and would probably become a 'flavor'.
As for scheduling itself though, we recently added support for UMCG: https://nanovms.com/dev/tutorials/user-mode-concurrency-grou... .
- pedagogy
- systems language research
- CPU architecture research
- embedded systems development where a linux process model is useful
- OS research (assuming anyone still does that)
- systems with high performance or low variance needs
- HPCOutside of gh discussions there is also https://forums.nanovms.com/. We made a decision a while ago to follow Zig's lead here and have no 'official' community space (https://github.com/ziglang/zig?tab=readme-ov-file#community) instead letting people form their own spaces.
I can understand why project maintainers don't want to carry the burden of gardening a space, but Zig is not doing itself any favors here. Instead, I would suggest maintainers to open a "vacancy" for an accessible and privacy friendly space, and if it fits the criteria, bless it. Or just start one yourself and hand it over.
Why? From a quick glance, most of the Zig communities are on discord. That means all folks that value privacy would not participate. And even if you are ok to lose out on that demographic, you still have a highly fractured information space. Although in the case of discord, "information space" is too generous. "Information black hole" is a more apt term, because none of it gets indexed by search engines. One cannot do without a bit of leadership to make a project successful.
... and props for your project!
How much privacy do you need to talk about a programming language?
Official Discords are great!
Is there a way to inherit from config files?
You can then deploy this as-is with a simple config.json or you can create your own new package (for instance if it's something like node and you want to share it with your team).
Well chuffed. I couldn't get ops pkg push to work at all, and instead had to manually upload a tar.gz as no matter what I did, it wouldn't find my local image (though it would find it if I did ops pkg load -l nano-web_0.0.1).
Is there some sort of Discord or IRC to ask dumb questions instead of spamming up HN?
Either way I've made an issue:
There is still quite a lot of code in your average linux kernel that is not easy to ifdef out. For instance the capability of running many different programs by many different users means you have a scheduler that has to support that. You have to protect those different programs from being screwed with by other 'users' or other programs (eg: just cause I'm able to own one program doesn't give me access immediately to someone else's on the same machine if owned by a different user) even though a lot of companies have largely walked away from that concept.
It goes a lot deeper than that though. Shared memory, IPC amongst processes, semaphores all touch that. Then we start talking about users and their ptys and managing that entire environment. Then you have to start looking at /sys and /proc and /dev and all the other places that get stuffed with all sorts of things.
It really truly is very different because it started with a completely different architecture and deployment model in design.
Closes tab
Distributing unsigned software, then asking users to blindly execute it, is simply irresponsible.
In that case, you can simply stick to the commit hash you read the source code from.
https://raw.githubusercontent.com/nanovms/ops/0b7e8bb9e56767...
Download first, review, then execute.
Or even better, download, verify signatures from multiple reputable people, optionally review, then execute.
Isn't the commit and its hash immutable?
2. None of your artifacts or build script are signed by you (third party signs by apple are pointless)
3. Builds have no evidence of being reproducible, or anyone having reproduced and counter signed them.
Compare to say the Bitcoin or Monero processes that have multiple people build and sign every release so it is easy to trust there were no SPOFs.
See Arch Linux, Debian, Guix, Stagex
1) In many interpreted languages it is common to have convenience commands that shell out to call a script which shells out to call another script and 8 layers later you get to the actual real command. When I'm creating a package for an application that does this I have to usually figure out what env vars are being set, what paths are being changed and so forth. This is probably a super easy thing for whoever made the software to begin with but not so easy for someone that wants to just use it. So the solution is to make the original author aware that it might go into an unikernel environment or, far less probable, convince them that a better method would be to not do this practice to begin with as it.
2) In older software (specifically I'm looking at mid-to-late 90s), in a time before threads and commodity SMP machines and the cloud it was pretty common to write software that used multiple processes to do many things. Postgres is the most common example I use here (keep in mind postgres is descended from Ingres from the early 80s and Dr. Stonebraker is now on his tenth? twentieth? database venture DBOS (https://www.dbos.dev/) - which definitely has ideas that we are very keen about.
Anyways, that's not really the case today with go, rust, java, etc. For apps like this we will, from time to time port them. That's exactly what we did with postgres too to make it multi-threaded instead: https://repo.ops.city/v2/packages/francescolavra/postgres/16... .
I think there is a lot of opportunity out there for individuals to come in and create newer versions of software like this and get some really awesome benefits while maintaining more or less the same parity.
presumably Rust apps will work on this too then
Partly because it is the "carbon fiber" of programming languages. Just its use is something you can use to market to other engineers.
Partly because it's very ergonomic and I enjoy working with it. I'd much rather dive into esoteric Rust code than esoteric C code.
Partly because code written in of Rust will have less errors than code written inside of C. It isn't just memory safety - it's all of the type theory/best practices we figured out in the last 50 years.
I understand why it was written in C. It's an easier language to work with and they probably needed something out quickly. But coding in Rust is a sign (not proof, but a sign) that a program is of quality make.