EC2 Boot Time Benchmarking
daemonology.net
daemonology.net
That's actually pretty solid. If it were 5-10x faster than that you could probably fit that into a lot of interesting use-cases for on-demand workloads. The bottleneck is the EC2 allocation itself, and I'd be interested in seeing what warm EC2 instances can do for you there.
That said, I think for the majority of use cases boot performance is not particularly important. If you want really fast 'boot' you might just want VMs and containers - that would cut out the 3-8 seconds of instance allocation time, as well as most of the rest of the work.
Curious to see a follow up on what's going on with FreeBSD - seems like it takes ages to get the network up.
> The first two values — the time taken for a RunInstances API call to successfully return, and the time taken after RunInstances returns before a DescribeInstances call says that the instance is "running" — are consistent across all the AMIs I tested, at roughly 1.5 and 6.9 seconds respectively
“Running to available” is what’s in the table, ranging from 1.23s to 70 or so.
That's basically lambda although you lose control of the kernel and some of the userspace (although you can use Docker containers and the HTTP interface on Lambda to get some flexibility back)
Under the hood, Lambda uses optimized Firecracker VMs for < 1 sec boot
>I think for the majority of use cases boot performance is not particularly important
Anything with auto scaling. I think CI is probably a big use case (those get torn up and down pretty quickly) and you introduce hairy things like Docker in Docker unprivileged builds trying to run nested in a container
I'd love to see Lambda support custom kernels. That would require a similar approach: document the actual VM interface provided to the kernel, including the virtual hardware and the mechanism used to expose the Lambda API. I'd guess that they haven't yet due to a combination of 1) not enough demand for Lambda with custom kernels and 2) the freedom to modify the kernel/hardware interface arbitrarily because they control both sides of it.
How does that work?
How do Firecracker VMs differ from containers on lambda or fargate?
The kernel surface is part of their security model. There's some details here https://www.bschaatsbergen.com/behind-the-scenes-lambda
E.g. exposing the kernel would undo some intentional isolation
In any case, Firecracker+Nitro seem like they'd be a sufficient security boundary.
And yeah, Firecracker is pretty sick, but it's also something you can just use yourself on ec2 metal instances, and then you get full control over the kernel and networking too, which is neat.
FreeBSD rc executes all rc.d stripts sequentially, in one thread. OpenRC AFAIK can start daemons in parallel, but unfortunately switch to OpenRC was abandoned: https://reviews.freebsd.org/D18578
Our BIOS boot loader is very slow. I'll be writing about FreeBSD boot performance in a later post.
Do you know if there is any plans on FreeBSD creating a super minimal server version that can be used as a docker host ... similar in size to Alpine Linux.
"launch GPU spot instance" to "I can actually use the GPU, at least by running nvidia-smi".
I find that this can take up to 10 minutes and for the more expensive instances, this can mean non-negligible amount of money.
Great article BTW
Perhaps I should have been using "Clear Linux 34640"
TBH the bigger issue than boottime for AWS is lack of physical resources to fulfill the demand - this is what most big players are struggling with.
It's not a big edge, but knowing about downtime before others could certainly be an edge
Not sure if the data is freely available, but I used to be part of the team at AWS that calculated these. See [0].
[0] https://www.gartner.com/en/cloud-decisions/benchmark-library
It doesn't seem like they'll give you up to the hour information and you do have to pay for "Gartner for Professionals" which I'm not familiar with but it's definitely out there.
The key to this has to be supporting it cross cloud IMO -- people have the AWS vs GCP vs Azure conversation so often and I think most people know that GCP is best performance for the buck, but the other reasons to pick the other two are numerous. Might be nice to have some numbers on just how much performance differs against the others since that's one of more quantifiable things to weigh.
I've tried changing the various parameters under my control: assigning a specific network address rather than making AWS pick one from the VPC, disabling metadata, using a launch template, not using a launch template, using the smallest possible AMI (just a kernel and static binary), using Fast Snapshot Restore on the snapshot in the AMI, etc. None of those makes the slightest difference in instance start time; still ~8 seconds.
What else they did would probably require booting into Clear Linux myself and checking it out, but disabling services you don't need so systemd isn't wasting time, baking all kernel modules you know you'll need directly into the kernel instead of storing them as modules, and not using initramfs if you don't need it is probably the "general" answer for how to get a faster boot.
If you really want ultra-fast, you can turn Linux into a unikernel by compiling the kernel with your application as init, not using any other services at all, and booting via the EFISTUB instead of using a bootloader at all. I don't know if the AWS EC2 services provides any way of doing that, though. On your own machine, you can either use efibootmgr from a different booted system on the same BIOS, or by setting boot order in the BIOS, or naming the EFISTUB whatever that magic name is that EFI automatically loads if a boot order isn't configured in NVMEM.
Median is an odd metric to use. I think I might truly be more interested in the mean; maybe the geometric mean.
I am also very interested in worst-cases with this sort of thing, so the 80th and 95th and 100th percentiles would be helpful.
It would be interesting to see plots of distributions, as well: are boot times really unimodal? They seem like they could easily have multiple modes, which could really mess with all of these measures (but especially the median!).
Because the script is open source (thank you!) I may try this myself.
That said, I did my final measurements from inside EC2 (rather than from my laptop over wifi) so it probably wasn't necessary.
Anyway - probably not a big deal, probably mostly academic, although modal behavior would be really interesting.
C is an interesting choice of laguange for this task. They even implemented a rustic XML parser in there.
Also on boot, I /think/ EC2 pulls down the AMI to EBS from S3 so theoretically smaller AMIs might be faster but not sure
I managed to create a 1GB AMI of debian. Will be interesting to test this.
And I've tested AMIs that are no bigger than a Linux kernel and a single static binary; times are still on par with what's listed here. Still takes 7-8 seconds before the first user-controlled instruction gets to run.
That's a bit disappointing but I guess AWS has spent a lot of time optimizing Lambda vs EC2
https://gist.github.com/hashbrowncipher/17a92c6afb9642503876...
Or are you talking about cloud-init which is run by AWS themselves before running userdata scripts ?