Ask HN: How do you catch regressions when you change your AI agent's prompt?

1 pointyakshithk_2 comments

i've been building agents for a while and kept running into the same problem: change the sys prompt, swap models, or tweak for the agent calls a tool — and something subtly breaks. The only way I found out was manually running it or a user reporting it.

View on HN

Any AI is is non-deterministic by nature, this is more about statistics, but not dedicated test cases. So, even if you swich to the model which is better in metric results you still cannot guarantee it will be better in your specific use case or do the same. What you can probably do is to automate your important test scenarious and run each every time you change model or system prompt

I've been building a tool to do this - build a dataset based on tasks from your repo, then A/B test the agent with whatever change you're making to determine the impact prior to actually shipping it. If you want to check it out - stet.sh

Reply on news.ycombinator.com