The loader+initrd+userspace time there sounds unreasonable.

I've had libvirt bog standard qemu-kvm (not a microvm) creating a new Ubuntu VM from a disk image & booting to a login prompt in under 10 seconds for more than a decade. This is without fiddling with virtio, doing hardware scans for PCI, VGA, SATA and such, and booting via Grub (your "loader"). Those should be pretty comparable!