If I can get a word in on this subject, I'd like to say this:
If Android is stripped completely back to minimum and rebuilt fully from source, it's arguably a trustworthy platform, but who has the time to basically do the equivalent of Gentoo on their phone, potentially weekly?
Even if the binary blob problem was miraculously solved, a rooted android device is basically a sitting duck (https://www.reddit.com/r/netsec/comments/3hr9f0/i_am_john_mc...), but an unrooted device isn't sufficiently hackable (flexible) to ensure its continued security (ie, unrooted + depending on a central service for Android updates = no thanks).
To me, I personally don't equate "Android" with "secure" in any way shape or form; I consider the platform practicably unsecurable.
I was just doing a bit of thinking. How about: use the 100MHz secure processor to run Linux or some another open-source lightweight kernel, and add a second CPU (something run-of-the-mill but decent, 1GHz+) that the first one can switch on and off. Both CPUs can see the GPU (running a low-res display, to make it easier - and cheaper), controlled only using open drivers, and the CPUs arbitrate for who has control of the GPU.
I see two software use cases for such a model.
First, you could use the 100MHz secure processor to actually run the phone. The resulting UI would be pretty basic, but open and verifiably secure (this is not currently possible with any other device AFAIK, and would get you a noteworthy demographic). You could use the 1GHz+ secondary CPU in lieu of hardware GPU decode - as in, you setup the fast CPU with libx265 on a unikernel, and feed it data via DMA. That sidesteps the blob problem, and lets people securely chat via/watch video.
Second, you could use the secure processor to do basic system tasks (again, providing a minimal UI), and provide an option to boot Android on the second processor. Caveat emptor, but that would cater to the people who only want to go so far, and all on the one piece of hardware.
Hmm. A modem is just straight CDC with no weirdness, right? Also, are there cellular-class Wi-Fi chipsets with open drivers out there?
To me, I absolutely envisage a secure phone as a secondary device. I might not want it in my possession all the time. Under certain circumstancs it might make sense for me to do a lot of activity on another phone so I do generate decipherable noise. I might want different/unusual notification policies (eg, maybe calls shouldn't even vibrate under certain circumstances).
The above is just me in stream-of-consciousness mode - but I've personally wanted a truly secure communicator for a very long time, not because I actually have anything to hide, but because I find the idea of being able to achieve near-perfect security (in particular, secure boot) really compelling.
I can understand why x86 was the only viable solution for the desktop, and kudos for just going ahead and making the effort with that design. I think that for mobile communications, being able to send and receive simple text messages using a secure hardware design that's running carefully-vetoed software would just be really really cool.
PS. Host USB would be incredibly useful. I really like your idea of having the port lock down though.