AI Models Like GPT-4o Change Without Warning. Here's What You Can Do About It

> On top of all the other things that add uncertainty to our software (flaky servers, third-party APIs going down, scaling issues),

Plus the elephant in the room, that prompt injection and hallucinations are a fundamental way of life for the algorithm, even when it's not getting updated in the background.

People have been complaining about AI models surreptitiously changing underneath them for a while now, and we found evidence of it happening in the wild. We build an LLM monitoring and testing tool called Libretto, and we saw GPT-4o start to behave significantly differently on one of our prompts this week. This is a write-up of how we detected the change and what it means for building on top of LLMs that can change at any moment.

Just goes to show how important having control over the model is for actual production applications