There's a LOT that the phone OS and hardware manufacturers can do at the hardware and firmware level to make things better. A good model would be the security strategies used on game consoles, the recent generations of which have moved critical pieces of the system to embedded processors (sort of like the secure enclave on recent iPhones, but with more isolation).

Not putting device drivers written by random external engineers into the trusted part of your OS would be a good start. So would not letting any page acquire executable permissions without being signed (not sure what you do with jitted code -- if you can't effectively sandbox it, you probably don't want to run it, and yeah, that's a drag). Forbidding external processors (e.g., the radio module) from becoming physical bus masters is also a great idea. Encrypted memory (at the cache line level) has been in consoles for over a decade; it's not that expensive and it's a nice first defense against stuff like RowHammer and folks mucking with physical wires in a lab.

Consoles take this level of security seriously because it's not just about customer privacy or security or similar pesky stuff, it's about the revenue stream. (Secondarily, it's also about limiting the actions of bad actors on hardware under their control -- it's much harder to cheat on a console).