Would that it were.
You are using the application processor (the "computer") to do that work, but there are two other computers inside your phone - the baseband processor and the SIM card.[1] Your carrier has access (OTA updates, etc.) to the baseband processor and can load new code/functions on it without your knowledge at any time. Depending on the SOC your phone is based on, the baseband processor can have DMA access to your application processor. What that means is, the baseband processor (which you have no control over whatsoever) can read your RAM directly.
Your cryptosystem that you describe probably works quite well on a desktop or laptop computer, but your carrier completely and totally owns your phone and everything on it.
... and we haven't even gotten to what they can do with the SIM card ...
[1] Yes, the SIM card is a computer with its own processor, RAM and programs running on it right this moment.
[0] http://www.supremecourt.gov/opinions/13pdf/13-132_8l9c.pdf