This looks really cool, especially on the remote control part. I've experienced exactly the same problem, but ended up building my own tool through an opposite architectural approach than Saggar.

The main difference is that I want to keep my existing workflow in Kitty and tmux/zellij. Since all of them provide solid remote control protocol, I wrote captain-miao (https://github.com/hyperlogue/captain-miao) which is a standalone TUI app that runs inside these terminals, observing the agent sessions, managing tabs and windows, and allow me to quickly jump to the exact window/tab that needs my attention. The sessions are still native windows/panes inside the terminal/multiplexer that I configured and polished for years.

One difficulty I found when trying to expanding the agent support beyond claude code and codex, many agents don't provide a good way to inject hooks to listen to the session status change. wondering how Saggar solves this problem. do it infer state from the screen buffer?