Firecracker internals: Inside the technology powering AWS Lambda (2021)
talhoffman.com
talhoffman.com
> It is highly recommended to read the source code of this amazing project and explore it yourselves - https://github1s.com/firecracker-microvm/firecracker.
Yes! The Firecracker source is super readable, and a great way to learn about this stuff in detail. There's very little magic there, partially thanks to the efforts of the team to keep things accessible and well documented, and partially thanks to how Linux's KVM APIs abstract away some of the hard and hardware-dependent stuff.
For more on the context of how we use Firecracker at AWS, check out:
- Our NSDI'20 paper, on Firecracker and Lambda: https://www.usenix.org/conference/nsdi20/presentation/agache
- Chris and Julien talking about Lambda internals at reInvent'22 (including details of the Firecracker-derived technology behind "snapstart"): https://www.youtube.com/watch?v=EplOzQqgstA
- "Lightweight Virtualization, Opportunities and Challenges", a talk I gave in 2020 which covers some serverless/Firecracker topics: https://www.youtube.com/watch?v=ADOfX2LiEns
- The Security Overview of AWS Lambda, which looks at this stuff from the security perspective: https://docs.aws.amazon.com/whitepapers/latest/security-over...
Tangent but that github1s link from the OP (to an in-browser VS Code instance) makes a clear call out that it's not officially from GitHub. Which is fine. But for completion, GitHub does have an official feature for this by changing the url to `github.dev` or pressing '.' on any GitHub repo:
Then again, large companies that already invested in AWS are probably not likely to switch providers for specific workloads like lambdas.
Quite a different offering to lambda. It shares no source in common.
https://ro.linkedin.com/in/raduweiss
They have quite massive R&D offices in Romania, several thousand folks there.
I visited the Iasi math university years ago. They were very good then. Eg lot of Prolog, where others still did Java or Pascal.
what version of the kernel do you use (the github page says 5.10 but isn't that quite old?)
what (extremely minimal, I imagine?) kernel configuration
What do you use to build the 'micro' images (I'm guessing many won't even have a classic pid-1 such as systemd, but put their software as pid-1?).
How do you keep timesync of you're not using a timesync daemon? Can you make one of these daemons work on af_vsock (I know firecracker gives a virtio-backed Ethernet device but what if you only want af_vsocks?).
Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?
To answer the questions:
> what version of the kernel do you use (the github page says 5.10 but isn't that quite old?)
Right, they have tested with 5.10, but it also works with higher kernel versions. Our host currently runs 5.19 and we're planning to upgrade to 6.1 soon. The guest runs 5.15.63, we use a config very similar to the recommended config by FC team (it's in the FC repo). It's important to mention that we had to disable async pagefaulting (a KVM feature) with more modern kernel versions, as VMs could get stuck waiting for an PF resolve.
> What do you use to build the 'micro' images
We created a CLI that creates a rootfs from a Docker image. It pulls the image, creates a container and then extracts the fs from it to an ext4 disk. For the init, we forked the open sourced init from the Fly team (https://github.com/superfly/init-snapshot) and changed/added some functionality.
> How do you keep timesync of you're not using a timesync daemon?
IIRC we expose the time as a PTP device (handled by kvm) and run phc2sys to sync the time in an interval. Firecracker has some documentation on this, where it recommends chrony. It can also be done with vsock, but it would be more manual.
> Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?
The init forwards stdout/stderr of the command it runs to its own stdout, which Firecracker then logs out by itself. A supervisor reads these and writes the logs to files.
You guys don't happen to have a public writeup about how this works, do you? Maybe it's as simple as it sounds, but Fly and CodeSandbox both have some magic to turn Docker images into VM disks that I'd like to know how to build :)
Thanks :-)
> A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm project's implementations. Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor.
> The goal of the Cloud Hypervisor project differs from the aforementioned projects in that it aims to be a general purpose VMM for Cloud Workloads and not limited to container/serverless or client workloads.
Firecracker is such a great piece of technology. I'm amazed that AWS actually open-sourced it. All kudos to them. We're using Firecracker at our company to allow API companies build interactive demos like this one we built for Prisma [1].
The fast/efficient asynchronous io_uring backend now in released Firecracker makes it a good alternative for full-sized VMs as well as the original short-lived micro-VM use.
"[2019] Firecracker: Lessons from the Trenches by Andreea Florescu and Alexandra Iordache" - https://youtu.be/yULy6IFy49o
Image is out of focus for most of the presentation, but not the slides and audio is good.
I have to offload requests to state machines in AWS Lambdas which is a pain when prototyping
Basically the lambdas, even when you return promises, or set long timeouts, will conclude before completing a request
If you’re doing 5 lambda invocations you don’t really see this, but when you’re doing a lot of them you can see in the statistics a huge failure compared to GCloud Functions UNLESS you offload each fetch to its own state machines, which is technically the correct way to do it but is way over engineered compared to GCloud
The default Docker containers that CodeBuild uses (you can create your own) and the shell script it uses to parse the yaml configuration file (mostly a list of shell scripts) are all open source and the entire process can be run locally.
https://github.com/aws/aws-codebuild-docker-images
https://docs.aws.amazon.com/codebuild/latest/userguide/use-c...
Disclaimer: I work for AWS. But nowhere near the team that developed Firecracker
Does anyone know what open source competitors exist? For example is there a GCP version to check out and compare?
There are many companies offering such services, would be interesting to know who is using what!
I’ve been playing with Firecracker on a Raspberry Pi 4, but never could get Docker inside a Firecracker uVM to work. Should this be supported at all?
One possibly is if your running directly from a Initramfs without a block device then docker needs DOCKER_RAMDISK set as a environment variable.
Otherwise it’s possible the minimal kernel your Firecracker config uses doesn’t support it out of the box. You can use a regular kernel but you need to make sure modules can be loaded from somewhere.
Issue opened in 2021: https://github.com/weaveworks/ignite/issues/874