Which phone (secure computing platform) do you trust to securely store your wallet key? All it takes to lose is an update to the wallet app that sends the key to a server.
Think of it like selling a mail-order gold bar cozy. 100% of your customers have a physical item that's vulnerable to crowbar enhanced recovery.
Besides, if all you're doing is some kind of chain of authority you could do it through fdroid which enforces a constant key. No need for a standalone phone which will anyway have the same components as any other phone.
You could use BTC online wallets on Apple Phones, but it was difficult to find one where your iPhone itself stored the wallet.
I personally find that more risk than benefit, but traditional banking's security mechanism seems adequate for many people.
If crypto currencies cannot offer the same level of convenience/security, then how can they compete in the mass market?