Hacker News Reader

Comment by JPLeRouzic | Hacker News Reader

The author says:

"It used Voronoi Cells and reduced most math to 8- and 16-bit integer calculations with one or two single-precision floating-point calculations."

I wonder if it can be used to speed up LLM inference.

View on HN

That's what quantization and stuff is, basically. So yes. Been done for years on all neural net types not just LLMs.

Please can you provide some links to intro level articles?

This seems worth the read but I didn't dig too deep https://medium.com/@sergiopr89/float-point-quantization-the-...

Quantization compresses the model’s parameters and you lose precision in the process. My understanding from the article was that precision wasn't lost - 'It actually had higher resolution than Extended-precision floating point math!'. The point of quantization is to make the model smaller to fit into available ram. I think the OP is referring to computation which is a different beast.

You may be conflating distillation, which may be a fault of other people getting it wrong. Going from floating to fixed point doesn't change precision or size. It changes representation. For example, if all your values are between 0 and 1 or -1 and 1, you're not using the entire range of a standard IEEE float. If you instead use Q31 format, you get basically 31 instead of 23 bits of mantissa. So it's an increase in precision if and only if you can basically stretch your sub-range that you were using over a larger range of bit representations. You have to think of the total number of values. 32 bit is 4 billion. 80 bit extended precision is a lot more than that. But he didn't say I think what his fixed point bit width was. That's what matters.

Now maybe you don't need to represent 1/2^32 in precision. Maybe you just need to know to 1/8th. That's where you can quantize to a lower precision to save space.

But if you start with 32 bit float and quantize to 32 bit fixed, I struggle to think how that saves storage.

I wasn't conflating distillation. You are basically repeating what I said - you're describing quantization. The OP was talking about changing the computation. The article says the 8- and 16-bit integer approach actually had higher resolution than the floating-point approach. You said we already had quantization to do this but quantization isn't what the OP was describing.

Reply on news.ycombinator.com