Clive OS: Go applications that can run without an operating system (2016) | Hacker News Reader

Clive OS: Go applications that can run without an operating system (2016)

lsub.org

162 pointsmerqurio30 comments

It would have helped to state their mission goals. This is basically a page that says "look we made this, no underlying stack and lots of CSP", without stating what they hope to achieve or improve versus the status quo.

Are there security benefits? Probably. Better performance? Could be. But why did they come up with their solution? What about differences with other Unikernel approaches? I mean its a research lab, not a hobby project.

This submission is here because of the link to the intro to OSes book[1] by one of the Clive authors that was posted yesterday. It's worth taking a look at the other stuff referenced on this page page, as well as the author's other links.[2]

To answer your question:

> Better performance? Could be.

That's one aspect. If you follow the link from the Clive page to its predecessor Nix[3], its tagline is "high performance cloud computing is nix". A current (2018) justification for that project can be found in the form of an abstract for a paper with the same name[4].

But then the other part is given away in the abstract for Clive itself: "Clive introduces new file system technology and novel interfaces for the construction of networked system"[5], which is another way of saying that this is blue sky research, which is itself a codeword for we decided to try out a bunch of things purely _because_ they're different and want to see if anything will come out of it. Notice how the screenshot here shows the OS shell is some sort of acme-like[6] made to run in a web browser, for example.

To repeat my earlier remark, the author's other pages are well worth a look. For example, don't miss this[7] page chock full of links to research OSes (although many of them are dead). Or there's Picky[8], which looks like what you'd get if you took Pascal/Oberon and more or less did a transliteration so that the grammar gives you a curly brace language instead of a wordy, shouty one with BEGIN, END, etc.

1. https://news.ycombinator.com/item?id=16253193

2. http://lsub.org/ls/nix.html

3. http://lsub.org/ls/nix.html

4. http://lsub.org/ls/nixbltjabs.pdf

5. http://lsub.org/export/clivesys.pdf

6. http://syssoftware.blogspot.com/2016/02/clives-ink-screensho...

7. https://lsub.org/who/nemo/os.html

8. http://lsub.org/ls/picky.html

There is a new paper[1] from F.J. Balleteros published on 2017 that serves as a great justification too.

- [1] https://jisajournal.springeropen.com/articles/10.1186/s13174...

for anyone interested in running Go apps as Unikernels, I suggest checking out https://github.com/solo-io/unik

disclosure: I'm one of the authors

Java, Node.js, Python 2/3, C/C++, and Go! Bravo!

The naming / logo / design is an obvious homage to Clive Sinclair :)

Thank you for pointing this out. I spent so much time with the old ZX Spectrum, but I still totally missed the reference.

For context, Clive Sinclair is most famous for the ZX Spectrum computer https://en.wikipedia.org/wiki/ZX_Spectrum

My very first thought as well, though if it is so, then I wish they should have mentioned that up front. Maybe they don't because of legal reasons? Bleh

For anyone interested in unikernels for other languages than Go, there's a list over on Wikipedia: https://en.wikipedia.org/wiki/Unikernel#Examples

Why make bootable images, as opposed to docker images? What sorts of applications do you see using this?

in one case I boot a blob and it comes up with everything it needs and starts functioning immediately. there is no ambiguity in which code is executing since its exactly those bits that I constructed it with, and no external service dependencies.

if you work in a model where you never upgrade servers, just bring up new ones and turn down old ones, doesn't it seem cleaner? obviously they both work.

Hardware level isolation. Or: if you run your docker container as a vm instance; reduced overhead with only the kernel parts you need.

Typical docker is just a glorified chroot under a shared Linux kernel. With namespaces it asymptotically approach a bsd jail.

But the "easy" way to get some real isolation, is to use hw support for virtualization - but now you need a kernel (software that enable programs to talk via a fixed abstraction to hardware; drivers for real and emulated hw).

And you've reinvented 5% of the magic dust that IBM put into mainframes.

Does anyone know of any Unikernerls running out there in the wild? This seems like a very efficient way to run heavy workloads. I wonder what the catch is, and if the benefits are outweighed by what you miss without an OS.

This is all theoretical right now, but I'm more excited about a world where VMs are treated like containers. Application orchestration at the VM level instead of at the container level. Reduced attack surface (no unnecessary OS components). Fewer moving parts = easier to reason about.

There are still a lot of kinks to be worked out. For one, what is the debugging story when you can't SSH onto a VM? Or do you compile into your application something analogous to SSH? If so, does that mean you're bringing in much of the "unnecessary baggage" that makes unikernels so compelling in theory?

I'm excited, but cautiously so.

It's certainty an exciting area to keep an eye on. I have not looked at containers too much, I understand the basic theory but I've not had a use case myself to deploy them in production. I am much more familiar and in love with virtual machines however. If applications could be built as a self contained tiny VM's I can see myself using these quite heavily.

I think the most exciting thing about containers is actually the infrastructure and tooling around them. The technology for managing containers and container images is a lot better than that for managing VMs and VM images. As far as I know, there isn't anything comparable to Docker, Docker registry, Docker swarm, etc for VMs, or at least they are outclassed in terms of power and usability.

If we could bring all of the benefits of containers to the VM world (including subsecond boot times), I think that could revolutionize at least cloud computing (if not other forms).

Does CloudStack/OpenStack not allow for container lime management? Again, without much knowledge of containers I am not sure of the comparison.

I do know I sometimes get annoyed with something like a web app or service needing a whole OS, to largely do nothing except use resources unnecessarily. Something that can see storage and that has an IP Stack would free up a lot of resources.

I don't know about CloudStack or OpenStack.

I don't think the OS is consuming a whole lot of resources on the scale of a web service. You might get some performance improvements from static linkage vs syscalls, but I'm guessing the "performance improvement" story is mostly one of latency (startup times) and not throughput.

Mostly the nice bit of getting rid of the OS is improved startup times, reduced attack surface, and an easier-to-reason-about system.

This probably impacts tooling as well--you can actually compile your entire image from source code instead of needing to fire up an instance, script changes to it, shut it down, then copy it.

I'd not considered the reduced attack surface, I'd mostly been thinking of the efficiency gains. It's a very good point.

The closest match I know of is containers:Docker::VMs:Packer https://www.packer.io

well containers, self contained tiny vms, etc. have one single downside. shared libraries do not work well with them, which means updating the host os does not fix anything inside the container. something that helps against that is actually running something like distroless, however that still won't make it, the best thing is probably update all the stuff every day automatically. (not a hard thing on something like k8s, etc..)

Indeed, it's a very interesting space to explore.

You get proper isolation at the "hardware" level, with performance and efficiency probably not very far from containers, or at least good enough for most use cases (with VT-x, etc.).

Some of the issues of containers disappear (cross-platform support, for one), but we gain others instead, which prohibits me from attempting to use this in production for the immediate future. (Side note: is anyone actually using unikernels in production today? Please share your experience.)

It took us quite a while to get the orchestration part right for containers, and even today, a lot of companies struggle with it. Not to mention writing tools for storage management, debugging, etc.

We need similar tooling for unikernels+VMs, and I haven't seen such initiatives yet. That's OK, it's still a young platform, but I'll definitely be keeping my eye on it.

Galois uses their HaLVM on Cyberchaff.

https://galois.com/project/cyberchaff/

The TCP/IP stack from Docker is based on MirageOS code.

https://blog.docker.com/2016/05/docker-unikernels-open-sourc...

Relevant: Programming Languages as Operating Systems

https://www2.ccs.neu.edu/racket/pubs/icfp99-ffkf.pdf

Imagine if every language was it's own Interactive Shell with no kernel. Scripting and functional languages already have this, as do C and C++[1].

You would lose language interchangeability on the system until someone wrote compilers for the other language software that they want to support.

And who doesn't like to write compilers in their favorite language.

[1] https://github.com/root-project/cling

No need to imagine, Oberon was quite a nice experience. :)

For Modula II.