Unit Economics of LLM APIs

lesswrong.com

Key takeway from the article: OpenAI's ChatGPT APIs might actually be, surprise surprise, profitable.

"As of June 2024, OpenAI's API was very likely profitable, with surprisingly high margins. Our median estimate for gross margin (not including model training costs or employee salaries) was 75%." "Once all traffic switches over to the new August GPT-4o model and pricing, OpenAI plausibly still will have a healthy profit margin. Our median estimate for the profit margin is 55%."

Ye the article is pretty interesting - I was trying to calculate inference provider's profit margins for say Llama-3.1 8b including electricity costs and the power consumption of GPUs, and I seem to always get to around $0.20/1M tokens as the minimum cost if you amortize a GPU's cost over 1 GPU. If you amortize it over 2 years, then maybe $0.15/1M tokens.

I'm very intrigued about OpenAI's profit margins, if they are indeed true.

4o is heavily quantized perhaps and runs on hardware that conceivably could be used for 3-4 years?

Fair point! Ye if it's 3-4 year amortization that's reasonable - quantization could be another reason why some benchmarks for 4o aren't that good vs pure float8 / float16, and also how they made it much faster

Reply on news.ycombinator.com