Hacker News Reader

Comment by gkbrk | Hacker News Reader

Based on the kernel boot time (28 ms) given by the author, this process takes 1.97 milliseconds.

I'd consider this less "haha dum code is slow" and more "you are booting the kernel way too much, even a 1.97 millisecond delay is noticable".

View on HN

I think the “Firecracker” referenced in this Tweet is Amazon's “microVM” environment[0]. VMs running in this environment are sometimes booted to service scheduled/batch jobs, but are, I think, more often launched in direct response to (and with the intent of serving) an incoming HTTP request (and then stick around for a few minutes afterwards in case there are any subsequent requests). Network latency from the continental US to the nearest AWS datacentre is typically <30ms, so even after you add the response time of the application running in the VM, an extra 2ms boot time still probably makes up >2% of the total cold response time.

[0]: https://firecracker-microvm.github.io/

This will also make a lot of difference once we can run lightweight, userspace kernels ("rump kernels").

> once we can run lightweight, userspace kernels ("rump kernels").

Er, didn't NetBSD accomplish that... like a decade ago, now? Or is FreeBSD looking into supporting rump kernels?

Yes, NetBSD did accomplish that, but I expect it to become more commonplace.

but question is, is this solution for people running software on other peoples computers ? or solution for running my own code on my own computer only ?

Does it matter? Running kernel pieces in userspace is useful in both directions (rather like docker), and in both cases we'd like it to run as quickly as possible.

For my particular case it's about something similar in purpose to sandboxing, but with providing the compartment (ie a process subtree) with an alternative kernel to talk to, to minimise the attack surface between that container and the host kernel.

So wouldn't this fall under the category of "booting the kernel too much"? Why not find a way to keep already-booted kernels laying around and recycle them, rather than rebooting them afresh on every request? This pattern has already played out in other situations and been solved this way, such as with object pools or thread pools.

This is the "AWS Lambda cold start" use case: you're not doing this every time, but you are doing it on scale-up, initial deploy, etc. Depending on complexity, those costs start to hit your p99 as well as your p100.

All that sounds like it adds much more complexity than swapping out bubble sort for qsort, and the other various optimizations that have been going in to reduce boot time. Plus optimizing common code improves boot time for everyone running FreeBSD - routers, firewalls, etc, whereas pooling just helps that one use case. If you can make it startup up fast enough that you don't need any of that extra machinery, why wouldn't you?

Yes, Firecracker supports snapshotting for that reason. But you still have the famous "cold start" lag. Not to mention that, for some use cases, you'll potentially want to add restarts for security reasons (Firecracker is designed to be able to execute malicious code).

The problem here isn't the boot time, but the booting at all!!

Having a microVM handle a rare endpoint and then exit is often cheaper than leaving an EC2 instance running constantly. An EC2 instance also has extra administration needs. At some scales the savings can be significant.

Shaving ms off the boot of that VM is a performance boost of that lambda (or whatever the cloud vendor calls it).

Why does anything have to boot? What does booting do?

Maybe you're calling the serverless function from another serverless function in the same datacenter (or more simply, maybe you're instantiating a local firecracker VM to run some user-defined function within the context of your currently executing program), in which case the latency would be more on the order of 5-10ms.

Yeah, and Ahmdal's law comes directly into play here. At 7% of the total boot time, this bubble sort may be the slowest part of the kernel boot. But, even if you optimized away the sort entirely somehow, you could still only save 1.97 ms. And once you do that, the next slowest thing probably doesn't take more than ~1ms to run. Eventually, you get to a state that resists these kinds of targeted optimizations, where there are no bottlenecks, and the only improvements can come from new algorithms and data structures, i.e. "uniformly slow code."

Of course, calling anything that takes place in less than a blink of an eye "slow" is overstating things... lol :-)

If only I could get my desktop environment to come up in less than 28ms :-)

https://en.wikipedia.org/wiki/Amdahl%27s_law

https://wiki.c2.com/?UniformlySlowCode

exactly, it is not problem of making it slower / faster or which is slower/ faster.

BUT

that we use solutions across multiple problems instead providing solution specifically for that problem. if you have KNOWN vm, on KNOWN hardware then why you need all kinds of hardware and other bull.. initializations, checks.... ? just provide list of things it has to have then after list is run over without problem, continue. also why would amazon/gcp/azure care ? they are making money from you running inefficient solutions....

main point of running unixy stuff is exactly that i can mold my solution to specific problem EASILY. thats why it is so common in embedded market ( routers, kiosks,drones, car infotainment, TV... )

Sure but Firecracker is used for AWS lambdas, so you're going to be booting those extremely frequently. (Now I wonder if Amazon is using FreeBSD for AWS lambda ..?)

AWS Lambda does not support FreeBSD.

It would be great if that changed though...

We now boot so often kernels this is important.

My tv boots a kernel, Alexa's, laptops, cars etc.

And this might be not specifically critical but it's a sign of people just doing something less optimal

Do you think those devices boot every time you send them an input, or just when they're plugged in? Unless you're turning the TV on and off thousands of times per day this would be unnoticeable.

Certain systems on a vehicle make sense to power off. Do you really need the unit controlling your backup camera to be booted into standby while the car is turned off?

No, but I'm not taking thousands of six-inch family vacations per day, I'd walk if I wanted to travel a distance for which two milliseconds was noticeable.

I think you're mixing (bad) analogies to make some type of point but I'm not sure what it is.

The original comment you replied to mentioned cars in the context of booting devices often, so I simply pointed out that yes, actually it makes sense to boot up some devices when they go to receive input and not to leave them in standby as you suggested. In a vehicle that may sit off/idle for several weeks, it's best not to drain the battery unnecessarily.

I probably boot a few embedded devices per day.

If someone did not optimize this this might indicate other optimizations

Well, a microcontroller in a microwave is a bit different than fully fledged POSIX system and both are "embedded devices".

But yes, I also hate when home router takes forever to reboot and reconnect

Do we? And... do we know that we can make it significantly faster? To wit, will the code be bigger to do a smarter sort such that that itself causes problems? Or, even with a faster sort, is it still 1.6ms?

2.031 ms, I rounded down.

You lack the imagination to come up with a use for some improvement off the cuff, that doesn't mean everyone else (some of whom are more motivated) does as well.

Why do we even need to waste any consciousness-seconds on watching electric circuits initialize? Computers should just start working

Probably true. My question was "how often is this run?"

I assume it's not just once when I first turn on my laptop from fully off. Is it once every time I restart a Docker container? (Assuming FreeBSD images.)

Is it something that runs a couple times a day, a dozen times a day, a thousand times a day? Fifty times every time I open an Electron app?

I suspect this doesn't matter and is more a curiosity, but I wouldn't be shocked to learn otherwise.

So that's a very impressive number, but how long does it take to bring up userland?

Years (to be a little hyperbolic). FreeBSD still doesn't have a good pid 1 story. It still has a very simple /sbin/init that just spawns whatever shell script happens to live in /etc/rc. That shell script essentially does a topology sort of service dependencies and then starts each one individually and serially.

Reply on news.ycombinator.com