Then you’re either not testing your prompts or doing something trivial.

Remember: a good model with a good prompt will generate bad outputs sometimes.

A bad model with a bad prompt will generate a good output sometimes.

That is simply a fact with these non deterministic models.

You have to do many iterations for each prompt to verify they are working correctly.

> I’ve not had much problems moving between LLMs…

If you want to move your prompts to a different model, you’re effectively replacing one:

f(prompt + seed) => output

With different black box implementation.

Unless you’re measuring the output over multiple iterations of (seed) and verifying your prompt still does the right thing, it’s actually very likely that what you’ve done if take an application with a known output space and converted it to an application with an unknown output space…

…that partially overlaps the original output space!

So it looks like it’s the same.

…but it isn’t, and the “isn’t” is in weird edge cases.

Unless you’re measuring that, you simply now have an app that does “eh, who knows?”

So yes. Porting is trivial if you don’t care if you have the same functionality.

…but reliably porting is much harder (or longer).