The Kernel has no stable ABI for drivers.
Manufacturers only ever develop a driver for their chips once, and then send that to the OEM. They never update.
The Linux Kernel LTS gets 2 years of updates, Google's fork about 4.
From the day a Kernel is released, to the day it ships in a phone, usually 2 years are spent integrating the blobs and code drops from the chip manufacturers.
On every kernel upgrade that breaks the ABI, those 2 years would have to be redone from scratch.
Linux can't mainline support for every exotic piece of hardware that ever shows up in a device.
Manufacturers can't keep maintaining several developers to update every single chip they release.
Google can't keep Android on 6 year old kernels forever.
Now combine these facts, and you'll see the issue.