I personally wen't this route last year and eventually got things working (networking etc. OK with only the kernel). My company then took a direction in which each compute node has its custom kernel, so that we can control the initial stages of boot better (initrd is dynamically downloaded from intranet, given that the network drivers are baked-in) to manage heterogeneous nodes with different "roles" (GPUs, add-on devices, etc.).

Something I have been surprised about is how much functionality exists in the Linux kernel and how much more reliable the modules are compared to many user-space tools (e.g. race-conditions).