I've been wondering why Gemini CLI (and others) don't have this feature yet (letting the agent interact will full-screen/interactive terminal like VIM, or a running node server, a ruby CLI etc. so I tried to make a PoC to see what was the technical challenge.
Gemini CLI did add a kind of "pseudo terminal" support (
https://news.ycombinator.com/item?id=45605823) but that still does not let the agent execute commands.
So gopilotty is my vision for a more "interactive" agent: a two pane window with a terminal on the left, a chatbot on the right, and the terminal is a combination of a "pseudo terminal" running bash, linked to a terminal emulator (for escape characters etc), and you can ask the bot to make edits in VIM (it knows how to exit !), run SQLite etc... Run it in a VM for safety :)