I don't think you can ever make a case for it on cost savings in general. Inference is very obviously the kind of problem where things are cheaper at scale, and this is still true for smaller models.
The only reason to run locally is privacy.
The only reason to run locally is privacy.
Things get cheaper at scale but that's where the provider's margins come in!
I do think there's also an interesting idea: you buy a box like this and run it at a fixed-ish cost (well, electricity). Your demand goes up but your supply is fixed... and that back pressure means that you still have good cost control.
With cloud providers it's a _biiiiit_ too easy to just increase spend.
Sometimes it's OK for things to just be slow.
Renting tokens from open model providers is cheaper but it incurs the same issues: unexpected changes in model quality, inconsistent speeds, service outages.