Exactly! With even just 3 chained calls, 98% individual chance of failure ^ 3 calls ~ 94.1% joint probability of success or 5.9% failure global rate. That's a failure rate of 1 out of 17.

Problems like those and my experience as a reliability statistician are why I built ModelGym: https://modelgymai.com/

I want to help LLM app developers deliver on their visions and accelerate the realization of AI's potential.

It's free to try. If you have any feedback, I'd love to learn of it.