I agree on the TPA not using the secure element on the phone is an issue, but I wasn't addressing low quality authenticator apps, but for apps or for tokens you're assuming that the developers haven't cheaped out.
> A smartphone OS ...
Yes, and the same applies to tokens albeit with simpler logic, and the same applies to any PC. So at some point you're saying "I trust that the company producing the product I'm using is not lying in their security documentation".
> .. Reviews ...
Correct, because reviewers are talking about feature set, competent phone companies provide extensive documentation of the security architecture of the devices. Security researchers periodically write up their investigations.
> These are opaque devices, with a huge attack surface.
No, by the standards you have presented all your devices are opaque, but unlike every other device you own the default security model of a well designed phone is far stronger than any other device you have. They don't run arbitrary code, they don't support loading code into kernel space, user partitions are completely separated from the read only system partitions, etc.