Even bigger could be sandboxes. It's certainly possible to sandbox your agents and still use a GUI, but SSH is the easiest way.
And not sandboxing your agents seems insane.
There's no reason you couldn't present a GUI over an SSH connection. Whether the agent is running locally or remotely is orthogonal to the type of interface used.
Most of us sshing in would be very annoyed by the GUI and associated overhead and latency. "Why doesn't this exist?“ Because people don't want it at scale.