Hacker News Reader

Comment by tptacek | Hacker News Reader

It allows Claude to take screenshots and generate keyboard inputs. It's like TUI Playwright.

Maybe I'm not understanding it (totally possible!) but could Claude just do that by reading standard out and writing to standard in?

I had a really hard time getting anything like that to work (you can't just read stdout and write stdin, because you're driving a terminal in raw mode), but it took like 3 sentences worth of Claude prompt to get Claude to use tmux to do this reliably.

I would love to see your prompt if you ever post it anywhere.

For Claude, it's enough to prompt "use tmux to test", that usually does the work out of the box. If colors are important I also add "use -e option with capture-pane to see colors". It just works. I used it regularly with Claude and my TUI. For other agents other than Claude I need to use a more specific set of instructions ("use send-keys, capture-pane and mouse control via tmux" etc.)

Since I have e2e tests, I only use the agent for: guiding it on how to write the e2e test ("use tmux to try the new UI and then write a test") or to evaluate its overall usability (fake user testing, before actual user testing): "use tmux to evaluate the feature X and compile a list of usability issues"

Thank you!

I tell Claude code to use an existing tmux session to interact with eg a rails console, and it uses tmux send-keys and capture-pane for IO. It gets tripped up if a pager is invoked, but otherwise it works pretty well. Didn’t occur to me to tell it to take screenshots.

`tmux capture-pane`.

Also many CLIs act differently when invoked connected to a terminal (TUI/interactive) vs not. So you’d run into issues there where Claude could only test the non-interactive things.

So by screenshots you mean tmux capture-pane, not actual screenshots. So in essence it is using stdout, just not Claude’s own.

"In essence" but terminals do stuff to render stdout that you do not want a LLM to have to replicate, I think. If your TUI does stuff in fullscreen or otherwise with a bunch of control codes, that is simple work for a terminal but potentially intractable for a LLM.

Reply on news.ycombinator.com