As long as they revert the charge when notified of scenarios like this , and they have historically done so for many cases, it's fine. It's an acceptable workaround for a hard problem and the cost of doing business ( just like Credit Cards accept a certain amount of loss to fraud as part of business)
Overcharge protection doesn't have to be free. It could be +5% on prices or a fee of 25% when you reach the threshold.
They would have financial interest in calculating cost in real time and it'd magically become more and more precise over releases.
It's absolutely not fine to be at the mercy of other people, that's what we buy cloud products or really any products for: So that we are not at the mercy of hardware faults, bad weather, bad teeth, hunger, thirst, [insert anything]
It sucks, but that's unfortunately the world we live in until something changes.
The US could rely on an agency like the CFPB to prevent this, but that was gutted under the current admin.
If for some resources you can't sample measurements fast enough you could weaken it to "triggers within one dollar or five minutes after cost overrun, whichever comes later". But LLM APIs are one of those cases where time isn't a factor, your only issue is that if you only check quota before each inference a given query might bring you over
What makes you think that?
Don't tote the party line.
Same reason why Azure AI only has easy rate limits by minute, not by day or week or month. Open source proxy projects do it easily tho. Think about the incentives.
Going over a hard cap by 3% would be a reasonable failure to make, not by 30000%.