The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
You can, for example, hook it up to a logging system and have it fix errors as they occur on your platform.
I’d be curious about:
- your setup. How it all works - The types of errors it fixed and how quickly - Any regressions or issues it caused - The cost
Thanks!
It works surprisingly well. The errors fixed are both genuine errors in the harness itself, but increasingly so upstream bugs (in the underlying agent apps like Codex, or in Herdr, which is used to expose uniform programmatic access to all those different apps) for which it needs to come up with workarounds. No regressions so far.
The cost is hard to judge on a subscription, especially when you're running really heavy tasks otherwise that dwarf any harness work.