Ignite – Use Firecracker VMs with Docker images

Just like VM repos used to be used....

So, wait. You’re saying the people who developed that squeeze juice thing or the drop of blood thing shouldn’t be criticized because it is obviously a stupid idea from the get-go? C’mon man, some of us have critical thinking skills and some of us will tell you when an idea is stupid or over-engineered.

Containers as VMs is usually peak over-engineering. It’s unnecessary most of the time. In fact, there’s only a few use-cases where it’s desired to have that isolation:

- different clock rates in the container than outside (deliberate skew)

- high security applications

- require certain kernel modules that you don’t want installed in all containers

Isolation can already be guaranteed by the kernel (excepting some bugs), so I don’t think that is a valid use-case, but I could be wrong.

OTOH, smart people seem to constantly try to reinvent existing products or over-engineer tools to set themselves apart and create a new business. See the endless stream of new JS frameworks popping up every day. Or the myriad ways that people try to create clouds on top of clouds (Yo dawg! I heard you like clouds!). Or new databases (although I confess I follow these). Or reinventing static HTML page delivery. ... and it goes on and on. All of these are smart people investing time and energy into a "solution" ... where there may not actually be a problem to begin with.

Being smart doesn't mean that whatever you're doing is.

Maybe.

I think I'm also a pretty smart person in certain areas, or at least I should be by now having been doing this stuff for almost 30 years.

While I definitely always want to check myself, it's also part of my job to help the newer/younger engineers remember the golden rule of business technology:

Innovate in the business domain, not in the technical one. Keep technology simple and boring.

It's great to experiment, and sometimes the results of the experiments are winners. But generally you can tell winners from losers based on whether they make everything less or more complex.

NodeJS, for example, reduced complexity because it's one language everywhere and has a very simple single threaded execution model. Now with es6 modules, commonjs fading, we simplify even more.

Rollup/vite reduce webpack's complexity, but will also be replaced as native solutions emerge.

REST reduced SOAP/xml-rpc's complexity.

HTML5 replaced flash (and most of the browsers plugins), because flash had grown too complex with a massive attack surface and was impossible to properly secure. Remember ActiveX controls that would load and run in the browser with full permissions? I do. I wrote several.

Linux as an app server reduced Windows' cost and complexity.

Anything that adds needless complexity to the stack, no matter how interesting, will end up being a fad and replaced within a few years.

From https://firecracker-microvm.github.io/:

  Firecracker is an alternative to QEMU that is purpose-built for running serverless
  functions and containers safely and efficiently, and nothing more. Firecracker is
  written in Rust, provides a minimal required device model to the guest operating
  system while excluding non-essential functionality (only 5 emulated devices are
  available: virtio-net, virtio-block, virtio-vsock, serial console, and a minimal
  keyboard controller used only to stop the microVM). This, along with a streamlined
  kernel loading process enables a < 125 ms startup time and a < 5 MiB memory 
  footprint.

Basically they removed anything that wasn't needed to run a Lambda. All your I/O, CPU, memory, etc are all going to be limited to one very simplistic implementation, in addition to whatever KVM exports. So rather than saying "what does it limit", it's more like "imagine anything that might take advantage of/depend on hardware, or even host<->guest or guest<->guest, and just forget about it".

I think that is GP's point: Firecracker shouldn't be considered a general-purpose solution for more secure containerized workloads.

> Why would you want to migrate a firecracker vm?

I wouldn't. I pointed out the lack of live migration as an of example why firecracker can't cut it as the basis of general purpose VM orchestration platform.

> then you can use a different VM tech that supports migration (KVM/Xen/etc)

"KVM/Zen/etc" are in the dark ages compared to the state of the art of Kubernetes container orchestration.

The reason for that isn't technical; KVM is incredible and full-featured VMs conceivably could be orchestrated with the same fidelity as containers. The reason is business: the cloud behemoths invest only in their serverless business models; thus k8s, firecracker, etc. I contend there is an enormous market for orchestration of more complex workloads that VMs suit well.

> If you want to be able to migrate a running VM, do you still really need it to start super fast?

Yes. "You" is two different people in this sentence. The person doing the migration is often not the person who owns whatever service is running in the VM. They have uncorrelated needs. As a platform operator, I want to migrate VMs to be able to balance load and evacuate dying hardware. As a service operator on that platform I want high uptime - or, more accurately to this case, low downtime.

One of the reasons someone goes for a long-lived service on a VM platform is because the service is in some sense critical: it's Bad News when it drops a connection or doesn't respond. So the operator of that service wants something with a fast restart so that if it ever does need to be bounced, it's back quickly and they return to whatever redundancy level they originally designed to as fast as possible.

"Long uptime means a slow boot is fine" has never been true, in my experience.

15 minutes is on the high end, but 5-10 minutes is typical for Windows. You can’t deploy a ready image, usually, you have to deploy one that is sysprepped so Windows can do device discovery on first boot.

You can deploy them with the devices already set up, but sometimes (literally sometimes, it’s hard to tell when it happens) Windows gets really upset if any of the hardware changes. I think Windows 10/11 (and Server equivalents) has made that easier though (MS still recommends sysprep).

Ubuntu. It's not the OS.

(Granted it's Arch, but my home laptop boots to a graphical desktop in ~13s. … our servers don't have to worry about X.)

Absolutely. With firecracker, AWS seems to have solved the multi-tenancy problem that has plagued HPC since its inception.

I've just been waiting for some way to implement that in a sane way for on-prem infrastructure. If not for the root requirement, this would be perfect.

Singularity does get us part of the way there, but it's just another abstraction layer that researchers have to work around. And it's far from perfect.

Interesting. So Talos is both a Linux and a k8s distribution?

I am less interested in running k8s itself in Firecracker (or kata, or similar), but the pods started by it. I think there are advantages to that.