Is that an apples-to-apples comparison (either both monolith or both all-modules kernels, both with all drivers enabled?)
I recall, back in the long long ago, running through `make menuconfig` and disabling what I didn't need to get a smaller kernel + shorter build-times.