That being said, the difference between OpenAI and AWS cost ($1 vs $17) is huge. Is OpenAI just operating at a massive loss?
Edit: Turns out AWS is actually cheaper if you don't use the terrible setup in this article, see comments below.
A100 TDP is 400W so assuming 4kW for the whole machine, that's a little more than $5k/year at $0.15/kWh. Again, the difference is in the tens of thousands per instance. Even at 50% utilization over three years, if you need more than a dozen machines it's much cheaper to buy them outright, especially on credit.
even with the subs and api charges, they still let people use chatGPT for free with no monetization options. Sure they are collecting the data for training, but that's hard to quantify the value of.
If you keep reading past there, they get it down significantly. The 8 tkn/s number AWS was evaluated on is really funny, that's about what you'd get on last year's iphone and it's not because apples special, it's because theres barely any reasonable optimization being done here. No batching, float32 weights (8 bit is guaranteed indistinguishable from 32 bit, 5 bit tests as definitely indistinguishable in blind tests, 4 bit arguably is indistinguishable)
So in reality AWS is cheaper for a much better model if you don't go with a wildly suboptimal setup.