But let's go further - while we're trading flexibility for performance, why couldn't the entire kernel and userspace be precompiled+loaded into a single image file - like a Smalltalk/Lisp image - and loaded directly into memory, modulo some really-truly-has-to-be-done-upon-power-up initialization?
Sure, there's hardware that has to be brought up - so, right after initializing basic access to the SD card and RAM, this hypothetical bootloader could initialize another core and use that to bring up the random SoC subsystems while the kernel image is being copied into RAM.
And, yes, this would mean that you would need to re-compile that image every time you updated the kernel (or any of the userspace stuff inside) - but, again, we're sacrificing flexibility for performance.
Is this possible? Has anyone done something like it before?