That sounds endlessly frustrating. Imagine trying to use your computer and the primary way to accomplish anything is to convince/prompt-engineer some two-bit LLM into doing it. First you have to convince it that what you're doing is ethical and doesn't violate its safety guidelines, then you have to get it to actually translate your natural language into the actions you want to perform, which will be more difficult the less common those actions are, and finally you need to check its action plan for any subtle errors or misinterpretation. It would be like operating a computer by way of a phone call with an annoying relative who doesn't quite know how to use one.
This will be the case until we have AI agents that are as intelligent and competent as a top-percentile human personal assistant. At the point where that's possible though, I doubt the future of phone interfaces will be the most interesting change.