> we don't switch to heavily quantized models
That sounded like a press bulletin, so just to let you clarify yourself: Does that mean you may switch to lightly quantized models?
That sounded like a press bulletin, so just to let you clarify yourself: Does that mean you may switch to lightly quantized models?
I am willing to bet large amounts of money that OpenAI would never release a model served as fully BF16 in the year of our lord 2026. That would be insane operationally. They're almost certainly doing QAT to FP4 for FFN, and a similar or slightly larger quant for attention tensors.
(The loose version of this that’s true is that there may exist eval data contamination in pretraining. This is a hard problem to fully solve.)
The tokens/sec of the model is basically directly proportional of the memory bandwidth of the hardware it runs on. So either OpenAI has to gimp model performance for its entire life, or somehow magically speed it up 4x on the first day.
(I used the adjective heavily because that’s what the original post said. I have no intention of making misleading but technically true statements.)