Do few-shot examples generalize across different LLMs?

Hey there, I'm the founder & CEO of Libretto, which is building tools to automate prompt engineering, and we have a new post about some experiments we did to see if few-shot examples' performance translates across LLMs.

We took a prompt from Big Bench and created a few dozen variants of our prompt with different sets of few-shot examples, with the intention of checking whether the best performing examples in one model would be the best performing examples in another model. Most of the time, the answer was no, even when we were talking about different versions of the same model.

The annoying conclusion here is that we probably have to optimize few-shot examples on a model-by-model basis, and that we have to re-do that work whenever a new model version is released. If you want more detail, along with some pretty scatterplots, check out the post!