Online vs. Offline AI Evals: When to Use Each

inngest.com

10 points·aldersondev··3 comments

> That's fewer moving parts, one load-bearing system instead of two...

When I saw "load-bearing" I remembered the other discussion in the frontpage explaining how to prevent Claude from saying that so often.

Not-so-hot take: you need to measure your agent's performance in production.

Maybe-a-little-hotter-take: you should be using production data for evaluation rather than synthetic events.

Enter Online and offline evals; one measures performance in real time. The other, off the execution path, either before deployment, or after the agent run has happened. When to use both is explained in our latest guide!

I've found a good balance in looking through sessions based on user feedback (ex: user thumbs down and submits it into a review queue) to build an understanding of usage patterns and failure modes. This is very boring but critical to understanding how the LLM is trying to navigate the problem. Then you use your insights to create offline evals. Then hill climb those. Then A/B test the changes with your uses.

I don't know if this is the end state, but I think this is pretty close to what I was seeing a year ago and what we will be seeing next year. Yea, maybe we use agents to do the looking through but you will still be shepherding that.

Reply on news.ycombinator.com