Can you explain better how OpenAI, for example, would simultaneously subsidize tokens while overcharging for them? Not sure I get it.
Looking at reserved capacity cost for PTUs on azure, which I think they’d probably not subsidize but can’t be sure, I’m inclined to not agree with the vast undercharging for tokens hypothesis.
But the biggest discount people see is subscriptions. You get a few thousand dollars of work from a couple hundred dollars.
And this one is easy to calculate: take your monthly API spend to K3, then rent a stack of 8xB300 for a month and see how much it costs. I would say you're about to save 10-15k dollars per month if you have enough traffic compared to pay per token pricing.
It's not very complex math, and the hardware of course is cheaper if you bought it last year and if you have extra GPUs waiting in your warehouse (depending on if you can produce enough energy cheaply).