(upstream arm-soc kernel maintainer here :)
It's not really about anything Android-specific that's in the "android kernel", it's about all the other code that's needed for a platform to run well.
The main problem is that besides the base kernel, most mobile platforms require a lot of code that hasn't been upstreamed. In some cases, "hasn't yet", in some cases "probably never will be" -- it depends on the vendor involved if they have any such ambitions.
Most of this is drivers of various kind. Some vendors are better than others at upstreaming them and the other pieces that are needed, but nearly none of them have upstreamed sufficient amount of power management to make a real device useful and have reasonable battery life. There's also usually drivers missing for modems, etc.
And, of course, graphics is a very sore topic in this area -- no vendor today ships a phone that uses an open graphics base. Ironically enough, Nvidia is the vendor that has done best here, with work happening in the open on their DRM drivers (but no products have shipped with those drivers at this time).
The vendors that have done best are normally those who have more embedded-type platforms and not primarily mobile phone chipsets. By the time upstreaming of a mobile platform is done, the next generation is already out and nobody will build new products with the old one. It _does_ get better over time as more and more share code goes in, but most vendors have enough of a backlog that they don't see those benefits yet and as a result don't prioritize it as high as I wish they would.
Then there is of course some vendors who don't participate at all, or does very very little. The Chinese manufacturers used to be notorious here, but even some of them have started doing better as of late (Rockchip in particular, but MediaTek has started posting some patches too).
The vendor that traditionally has done best is TI, but they've gotten out of the mobile business. ST-Ericsson was making a good attempt too, and they also got out of it. Nvidia has actually been really good at working upstream on their 32-bit Tegra support, it's unfortunate that it's taken them this long to get going on the 64-bit support upstream.