Hacker News Reader

> until hardware FP8(FP4 anyone?) becomes more widely & cheaply available

Nvidia 40xx Tensor Cores support FP8 (but not the fancy async stuff on H100s). For some reason it's not used for models (AFAIK).

Some people report throttling issues in some cases, though.

View on HN

I think PyTorch recently added FP8 & even Intel's Neural-Compressor has experimental support. But yeah most HF LLM models are currently just INT8 or FP4/NF4 loaded thanks to BitsandBytes by Tim Dettmers.

Reply on news.ycombinator.com