It's not just the kernel design, it's the driver design as well. Fuschia has drivers being ELF binaries that hook into Device Manager process[0]. I believe they are developing a standard API for how drivers interact with Fuschia.
Linux doesn't really have this today, which means that the driver must be in the kernel tree to have it keep up with the kernel's interface. And with ARM lacking something like a BIOS for phones, each chip that Qualcomm et al make requires a full BSP to get it working. And from what I understand, Linus and others don't want these (relatively) short-lived processors to be checked into mainline linux (plus you have the lag time of that code being available). Android's Project Treble[1] aims to address this some by creating a stable API for hardware makers to use to talk with Linux, but it is not an ideal solution.
[0] https://fuchsia.googlesource.com/zircon/+/HEAD/docs/ddk/over...
[1] https://android-developers.googleblog.com/2017/05/here-comes...