Firecracker MicroVMs | Hacker News Reader

Firecracker MicroVMs

firecracker-microvm.github.io

176 pointscolinprince60 comments

I bookmarked this about firecracker's internals in HN in the past https://www.talhoffman.com/2021/07/18/firecracker-internals/

Best example of current usage that comes to mind (outside of AWS’s own Lambda & Fargate) is Fly.io - they they use Firecracker for all the containers.

Additionally, I remember browsing through BuildBuddy's project and saw they used it for build run isolation in their enterprise offering.

- https://www.buildbuddy.io

- https://github.com/buildbuddy-io/buildbuddy/blob/master/ente...

There’s also Koyeb. I’m not affiliated.

We also use it at Sourcegraph (to run arbitrary LSIF code indexers in a sandboxed environment)

From their docs, their product looks amazing. However I haven't successfully deployed my app on their platform: https://community.fly.io/t/first-app-fails-to-deploy/2754/3

This might be related to my network problem, I will try other deploy ways when I get some spare time.

Anyone any thought on using this in a homelab (just random servers such as laptops and microservers) with K8s vs normal containers?

Is there something new that prompted this posting?

If I were to guess, I'd guess because Firecracker is something "that good hackers would find interesting" https://news.ycombinator.com/newsguidelines.html

I think the question is raised because Firecracker was released years ago and has been posted and discussed many times before.

“Are reposts ok?”

“If a story has not had significant attention in the last year or so, a small number of reposts is ok. Otherwise we bury reposts as duplicates.”

https://news.ycombinator.com/newsfaq.html

I think the question is more along the lines of - was there another news, recently posted, that prompted this post?

Like for example if fly.io was announced yesterday, and they said they used Firecracker, then someone would submit this link about Firecracker.

So the original poster in this thread was wondering if there was something to that effect. Not decrying something being posted that was not new or already discussed before...

Would be interesting to see a project that unveils these "chain of influences" for tracking HN engagement on stories or even comments

as long as we're quoting the rules, "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

assume that the parent's not saying "you shouldn't have posted this, it's old", but rather "i'm interested in firecracker, was there some new development i missed that prompted this submission"... if the answer is no that's okay, but if the answer is yes i'd love to hear about it.

Yeah, I’m actually curious if there’s a new major release or feature launch that I missed. I keep tabs on the project, didn’t think anything new suddenly happened.

Easy way to earn points.

A past thread:

Firecracker: Secure and fast microVMs for serverless computing - https://news.ycombinator.com/item?id=22512196 - March 2020 (103 comments)

For those using firecracker, what's your method for building a rootfs?

I build them from docker images: https://blog.davidv.dev/docker-based-images-on-baremetal.htm...

I build them using Buildah, then mkfs.ext4, mount, cp, umount. It's a little bit annoying that I'm still using root at least for the mount part.

OpenWrt's build system has a method of building rootfs ext4 and squashfs images without any root, it's somewhere in that large Makefile mess.

(sorry, reposting this as I first replied to the wrong parent)

I similarly have built bootable disk images with various tools including buildah and have never been able to fully get away from needing root for various chrooty/loopbacky parts of the process. In principle, it should be very possible to point grub at a filesystem-in-a-file and be like "install to that", but I could never make it happen; it always wanted to be trying to infer things about how to configure itself from examining the host system.

And yes, I've studied the OpenWRT build to no avail. I would be delighted for someone to dissect whatever it is that goes on in there and write it up.

No experience with Firecracker specifically, but if squashfs images are sufficient, one should be able to build a tar archive of the filesystem without root (where all the files have the correct owners, mode, etc.) and then convert it to squashfs using `tar2sqfs` in https://github.com/AgentD/squashfs-tools-ng, also without root - I've done something like this to create squashfs images in constrained build environments, which worked well.

Directly from Docker image: https://github.com/combust-labs/firebuild#high-level-example. I’m the author.

nb, building from docker is also what fly.io does: https://news.ycombinator.com/item?id=22514017

Yes, I can also build from Dockerfile:

    sudo $GOPATH/bin/firebuild rootfs \
      --profile=standard \
      --dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile \
      --cni-network-name=machine-builds \
      --ssh-user=alpine \
      --vmlinux-id=vmlinux-v5.8 \
      --tag=combust-labs/consul:1.9.4

That SSH bit is no longer required as I'm using MMDS instead.

We're fancier now, and use containerd with LVM.

https://fly.io/blog/docker-without-docker/

That's super fancy, indeed.

Thanks for these write-ups. They make for a riveting read, even when more than half of it usually is beyond my technical know-how.

This is incredibly interesting. Is it alright if I email you with some questions?

Sure. My email address: radek@gruchalski.com.

see document here https://github.com/firecracker-microvm/firecracker/blob/main...

Does anyone know of any good orchestrators for Firecracker VMs? While the technology is great, the CLI and API are still a bit too complex and low-level to implement some basic use cases in production.

Kubernetes using Kata containers[1] as a containerd[2] backend

[1] https://aws.amazon.com/blogs/opensource/kata-containers-1-5-...

[2] https://github.com/kata-containers/kata-containers/blob/main...

How does that compare to firecracker-containerd?

https://github.com/firecracker-microvm/firecracker-container...

This repository enables the use of a container runtime, containerd, to manage Firecracker microVMs. Like traditional containers, Firecracker microVMs offer fast start-up and shut-down and minimal overhead. Unlike traditional containers, however, they can provide an additional layer of isolation via the KVM hypervisor.

Take a look at Weaveworks Ignite for an open source tool for running Firecracker VMs from container images: https://github.com/weaveworks/ignite

And of course AWS Fargate and Lambda use this tech under the hood transparently, so that's always an option if you don't want to host and operate it yourself.

Ignite looked intruiging when I checked it out recently - but I need to import rootfs tarballs directly, without going through any registries. Any helpful pointers are appreciated :)

There's a firecracker driver for Hashi's Nomad. Haven't tried that driver, but nomad isn't bad.

https://www.nomadproject.io/docs/drivers/external/firecracke...

It has problems on shutdown and few months ago didn’t work properly with namespaces, haven’t checked recently. Can be worked around…

Depending on what you want, OpenNebula (opennebula.io) has been making a thing of Firecracker.

I did this some months ago: https://github.com/combust-labs/firebuild. Plan is to come back to it sometime towards the end of the year.

If you want some easy-to-hack --but very hackish-- shell scripts, you may find this interesting: https://ongres.com/blog/automation-to-run-vms-based-on-vanil...

I just deployed a WordPress container on the same host that runs everything else. Makes me a bit uncomfortable. Was thinking about running a VM just as a Docker Swarm worker node, but this post reminded me that these virtualization runtimes exist. Looks like there's a drop-in containerd replacement using firecracker. Kata containers look a bit better supported, though?

Or am I too paranoid about container escapes?

What are you worried about them doing? If they take over the app inside the container, they can access your database and seed malware in your users' browsers.

I don't think you can be too paranoid about container escapes. We actually don't consider containers to be a reliable security boundary - we take it where we can, but if we need real isolation it has to be combined with another approach.

what is the advantage of server-less if you have to have to manage the servers?

This is more meant for providers. Best example being AWS's serverless offerings, as well as fly.io.

If you're hosting someone else's code.

If you work at a large enough company it all starts to look like someone else’s code.

The malicious person can always find the worst way to break everything, but dumb luck can get you pretty close.

Aw geez. Seems like we're going to need a Firecracker awesomelist now then!

Note that this Amazon developed technology does not work on Amazon Web Services instances.

It does however work on Google, Azure and Digital Ocean instances where you can configure nested virtualisation.

It works on AWS's metal instances, doesn't it?

That is definitely a constraint though, as the smallest available metal instances are pretty large, hence costly on a per instance basis.

Technically speaking you are right. If you are not limited to AWS, then even a $5 DigitalOcean instance is be able to host 4-5 VMs.

> does not work on Amazon Web Services instances

Why not? And what is Google, Azure and DI doing differently

Firecracker relies on KVM in the Linux kernel which relies on VT-x or similar. Offering VT-x inside a VM that's already using VT-x is called nested virtualization. EC2 VMs do not support nested virtualization, while others do.

That said, it works fine on EC2 Bare Metal instances.

(n.b., I work on the virtualization stack on Google Compute Engine)

Wouldn’t you want to run firecracker on bare metal anyways. Nested VMs come with a decent perf hit.

Thanks @jsolson for explaining

@bushbaba I wonder how large the percentage hit could be, if it's like 1% or 10%, if you happen to know

And how that percentage hit compares with gvisor performance wise

It does work on AWS on the instances that support nested virtualization, same as every other provider.

anyone knows this can create gpu-enabled microvms ?

No support for PCI devices at the moment, but I believe it's planned for some point in the future.

Maybe try Cloud Hypervisor instead. https://www.phoronix.com/scan.php?page=news_item&px=Cloud-Hy...

This is tricky for Firecracker, both because of security (and the attendant switching costs of keeping exactly one VM connected to a GPU at a time) and because of the Firecracker tenet of supporting oversubscription. They've been talking about it for a couple years. I think they're actually talking about forking off a GPU-capable variant of the project to work around the drama.

Reply on news.ycombinator.com