Biggest downside of CLI for me is that it needs to run in a container. You're allowing the agent to run CLI tools, so you need to limit what it can do.
It breaks most assumptions we have about the shell's security model.
You'll quickly realize that this is not feasible.