Not to provoke predictable responses, but I find it interesting that the tech-talented VPN providers are not using BSD in favor of Linux, especially with requirements like diskless operation, kernel customization, and tighter security.
In terms of diskless, I've run 25k+ iPXE deployments on diskless blade servers using a highly customized Ubuntu, and it was fantastic.
Regardless of OS choice, being diskless is also quite nice... if there was a security issue or you need an upgrade of some sort, you just reboot. Only thing is that it takes a while to reboot 25k servers... even on gigE. It was a bit of work to build the scheduling system to make that happen reliably, but it worked out quite well.
Last time I managed a small «supercomputer», 50x IBM blades running Suse, it wouldn't support PXE/NFS without kernel customization, but that would void support contracts and finicky third-party software. Made a switch to FreeBSD, where everything worked out of the box one hour later. That was over 15 years ago, I have no idea how much the situation changed.
This was effectively 25k PS5's... much more powerful now.
Even better if you had boxes with 10 gigE and the smaller image. Would take your times down from like 6-10 hours to 1.5 hours.
Also, I doubt a full 25k restart all at once you probably had underlying applications that expected rolling, blue/green or even % or nodes that can go offline at once.
I did not have a choice over the hardware design.
Each worker was individual. Underlying application didn’t care.
We had a couple full power outages. They were set to boot automatically. So yea… full restarts.