Embedded development is hard. Really hard. And much of that difficulty is intrinsic to the problem space, so it isn't going away anytime soon. But much of it is just embedded developers being masochists and embedded tool developers being sadists. For phones, the pain is probably 90% avoidable. 90 percent!
No, I don't know why we live with it either.
The reason your phone is different than your PC is that the PC's OS has all the drivers for everything built in, then loads whichever drivers your specific machine needs. Embedded Linux images are built specific to one model of hardware, with weird crappy OEM-provided drivers baked in and nowhere else available. It doesn't _have_ to be this way. But it is.
So to support your phone, the vendor basically maintains a custom-patched kernel with a bizarre build process. And that's done more or less independently for every model they're supporting. It's a huge headache!
Why deep embedded is this way is a little more understandable. (It's also where I prefer to work if I'm doing software; I don't like embedded Linux. It's just not my thing. So if I'm wrong when talking about it, that's why.) Deep embedded is often working with very limited hardware, custom circuits, and general strangeness. It also can serve the industrial, automotive, or medical markets, which naturally tend to longer product cycles. So the product is generally less standard, meaning it makes more sense (at least to me) why the support flow is so bespoke.