FreeBSD/EC2 boot performance over time | Hacker News Reader

FreeBSD/EC2 boot performance over time

daemonology.net

109 pointscperciva35 comments

Presumably seconds since I'm not sure what else would make sense here. It'd also be helpful if the y axis had consistent scale between each graph and horizontal lines are set y axis intervals

Seems zfs is quite a bit faster than ufs

The different X axes are because I figured there's no point having a bunch of graphs which are 90% empty. For a long time we only had base/UFS images. (Even arm64 came a few years later than amd64.)

But you aren't able to compare them then.

Your FreeBSD on AWS work is appreciated, @cperciva.

That's an impressive drop from 30 minutes in 2019 to under 10 minutes today.

No, wait .. maybe that's seconds? milliseconds?

Microts!

Seconds, yes. Sorry I figured that was obvious, I'll add it to the graphs I generate next week.

Thank you! Was about to ask for axis labels as well. I assumed seconds but I had some doubts because there were no labels!

Good to hear.

Seconds was my best guess .. but that doesn't make it certain for the viewer.

No real drama, but FWiW I've looked at a lot of data across engineering, math, geophysics, mining, energy, etc and unlabeled data and graphs are a major annoyance.

If you're taking requests, then can the Y axis start at zero, please?

https://handsondataviz.org/images/14-detect/gdp-baseline-mer...

And what is going on with that Y axis? There is no consistent spacing between the marks!

https://imgur.com/HaJmTSU

It's a bit hard for a log axis to start at zero.

Well start it at 0.00001 then. There's no need for a log axis, as all the values are within 2 orders of magnitude of each other - between 1 and 100. They will fit better, and will no longer be distorted, on a linear Y axis starting from zero.

I wouldn't bother. It is super obvious.

Nice improvements in boot speed. Perhaps a little blurb / intro / summary would be helpful to the post to help with understanding the achievements made.

Did the patches ever make it into Firecracker for booting FreeBSD as a guest? I looked back at the paper trail and it seemed like it may have stalled.

Does anyone know?

Things stalled; there were a couple outstanding bugs (one I know has now been fixed, not sure about the other) and I had to take over FreeBSD release engineering and didn't have time to follow up on the Firecracker bits.

I understand that NetBSD can boot in Firecracker using the same patches so I'm hoping they can resolve any remaining issues and prod the Firecracker developers into merging things.

Fixing instance types was probably wrong.

You’re getting progressively legacy (and more likely to be degraded) hardware. This impacts how tightly packed the instance type as a whole is, which impacts launch instance performance

Fixing instance types was probably wrong.

Depends what you're trying to measure, I think? My goal as a FreeBSD developer is to look at FreeBSD performance -- this is both to show the improvements which have been made over the years and to alert me to any performance regressions (I generate these graphs automatically when I build the weekly snapshots).

If you want to compare EC2 to other clouds you would definitely want to use the latest instance generation, of course.

Would be nice to see how this compares to Linux, I think, for perspective.

I did that comparison a few years ago: https://www.daemonology.net/blog/2021-08-12-EC2-boot-time-be...

Re-running the comparison with Linux AMIs from 2024 is on my to-do list.

Dang, Clear Linux boots in one second? How is it ten times faster?

Optimized for automotive use case (max 2s to enable rear camera), https://www.phoronix.com/news/Clear-Linux-Kernel-3s-to-300ms

> They started out with around a three second kernel boot time but cut it down to just 300 ms. Among the optimizations carried out to really speed-up their boot time were ensuring more asynchronous driver probing, only initializing a small amount of RAM at start and then after booted hot-plug the rest of it in parallel via systemd, optimized root file-system mounting, disabling unnecessary kernel modules, and similar approaches. Moving forward they are still looking at optimizations for the boot process around in-kernel deferred memory initialization, SMP initialization changes, ACPI tweaking, and user-space/systemd optimizations.

It's been possible for a long time:

https://lwn.net/Articles/299483

Is there an architectural change and/or approach to which the boot performance increase(s) can be attributed?

If not, which is understandable, is there something specific to stable/14 for interested parties to familiarize themselves with?

My impression is that cperciva has been working on FreeBSD/EC2 boot times for the last six years, not just specifically some changes in 14.

The most accessible summary of his work is probably on his Patreon: https://www.patreon.com/c/cperciva/posts. If you go digging there are also some anecdotes on his Twitter too; most recent I can find is https://x.com/cperciva/status/1833735559614988526

I gave a talk at BSDCan 2022 about this ("speeding up the FreeBSD boot") -- I'm on my phone right now so I can't provide a link but it's on YouTube.

There's also a somewhat out of date summary of most of the work on the FreeBSD wiki: https://wiki.freebsd.org/BootTime

https://www.youtube.com/watch?v=10XRCiBtyhA

The best summary I can provide is "profiling and lots of small wins".

That suits our deployments where for HA we simply terminate and replace unresponsive EC2 nodes with new ones. I’ll have to talk to other developers to see how much work would it be to compile to freebsd instead of Linux

Linux probably boots faster fwiw. As a starting point, see this bench: https://www.daemonology.net/blog/2021-08-12-EC2-boot-time-be...

What are the y-axis units?

I would guess hamburgers eaten :)

On a more serious note, the only thing that really makes sense would be seconds.

wow they make it impressively hard to contact them.

me @FreeBSD.org. I figured most people reading this would know who I am.

I have no idea who you are, nor do I check who submits what (the merit of the argument, not who said it and all that), nor do I have a reason to know (I have no reason to follow FreeBSD at this point).

I understand you do important work, but some people, me included, do other stuff. And there's a lot of people here.

cperciva is the HN comeback of all time:

https://news.ycombinator.com/item?id=35079

(And folks have had since 2007 to top it.)

Funny enough, that started with a "I don't know you" as well:

https://news.ycombinator.com/item?id=35071

Reply on news.ycombinator.com