> They won’t notice those extra 10 milliseconds you saved
Depends what you're doing. In my case I'm saving microseconds on the step time of an LLM used by hundreds of millions of people.
Depends what you're doing. In my case I'm saving microseconds on the step time of an LLM used by hundreds of millions of people.
In most real jobs those ten milliseconds will add up to what, 5 seconds to a minute?
Those 10 ms may quite well mean the difference between success and failure... or they may be completely irrelevant. I don't know if this is knowable.
* ... if they're at ClosedAI or Facebook or something. If they're at some startup selling "AI" solutions that has 10 customers, it may be wishful thinking that they'll reach ClosedAI levels of usage.
If it's about saving the users time, it's very non-linear. And if it's not a scalable read-only service, the costs will be very non-linear too.