This is ideology. In real-life, some OEMs are binded by NDAs (so they can't do all what they'd like), and the split between hardware and software is quite blurry (chipset vendors don't want to disclose their secrets by open-sourcing their code).
I wish like you that more code would be supported in mainline kernel (just as I wish every single board would support SystemReady/UEFI or any similar solution (ideally more lightweight) so that they could boot a generic kernel just like x86 does (and regardless of the availability of the proper dts for this specific board, which may be hard to download after several years)). But this is not the world we live in.
Because of this "stable-api-nonsense" ideology, we end up with piles of junk (e.g. all old smartphones and boards that become unsupported after 2-3 years... and we only have one planet !) while we can still easily use 10-15 years-old PCs.