A simple fix for LLM tail latency

engineering.myhoai.com

2 pointszhixuan1 comment

What if everyone did this? Then you'd send the same request thrice? Four times? Ten times? Does this eventually lead to crowding out the provider?