Building a kernel isn't difficult (and used to be pretty much required). Build support in many distros is quite robust.

The real challenge is that once you've compiled a kernel, that's all you've got in terms of support. If you need to add filesystem support, a networking capability, additional driver support, etc., you've got to configure and build a new kernel, and test it, which is distinctly less convenient than autoloading an existing module (or even one you've newly compiled in many cases, on a running kernel).

I seem to recall a kernel option or syscontrol, possibly from OpenBSD/FreeBSD, which prevents loading of additional modules once it's been set. This allows you to boot and load modules, but then no more. If your boot media are read-only, this gives a fairly high level of confidence.

I cannot find a reference though.