Hacker News Reader

Comment by qwertox | Hacker News Reader

What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading.

It's 4 bit floating point, at twice the speed of 8 bit floating point. There's also FP6, doesn't offer faster compute than FP8 but manages to take advantage of the better memory bandwith and cache use of the 6 bit format.

Apparently some people are drawing connections to this paper [1] on 4 bit LLMs, which has one NVIDIA employee among its contributors

1: https://arxiv.org/pdf/2310.16836.pdf

>bit misleading.

Only partially, because in LLMs FP4 isn't half as useful as FP8. So if you have gear that crushes at FP4 then that's what you use and you benefit from that increased speed (at minimal accuracy loss).

Definitely some marketing creativity in there, but its not entirely wrong as a measure of real world usage

curiously, what real world usage actually uses FP4? AFICT, most of the LLMs still use BF16, and even the quantizations down to 4bits and 2bits end up back to 16bit or INT8 for actual computations.

Llama.cpp and many others support 4 bit weights and lower

Half the reason why they move back up to 8/16 bit is that current hardware doesn't properly support 4 bit floats, and you get better performance from the conversion. I think once this hardware hits, most of the computation will shift to native 4 bit just for efficency's sake.

...assuming the recent 1.58b paper doesn't render the entire float quantization approach obsolete by then.

The 1.58b approach is good for everyone including for quantization. It means that current quantization schemes have room for improvement.

It's good for everyone on the software side, but it's not so good for the hardware side, because it means that whatever you're designing now for tapeout in 6 months and release in 12 is going to be obsolete in 2 weeks.

https://arxiv.org/pdf/2310.10537.pdf

Discussed in a previous post https://news.ycombinator.com/item?id=37930663

Yes this is it!

How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?

For training FP4 sounds pretty niche, but for inference it might be very useful.

- Training isn’t done at 4-bits, to date this small size has only been for inference.

- Research for a while now has been finding that smaller weights are surprisingly effective. It’s kind of a counterintuitive result, but one way to think about it is there are billions of weights working together. So taken as a whole you still have a large amount of information.

Intuitively, there is a ton of redundancy and we still have a long way we can still compress things.

Each token is represented by a vector of 4096 floats. Of course there is redundancy.

Maybe the rounding errors are noise that is somewhat useful in a big enough neutral net. Image generators also generate noise to work on

> - Training isn’t done at 4-bits, to date this small size has only been for inference.

Wasn't there a paper from Microsoft two weeks ago or so where they trained on log₂(3) bits?

Edit: https://arxiv.org/pdf/2402.17764.pdf

They don't "train on log₂(3) bit". Gradients and activations are still calculated at full (8-bit) precision and weights are quantised after every update.

This makes network minimise loss not only with regard to expected outcome but also minimises loss resulting from quantisation. With big networks their "knowledge" is encoded in relationships between weights, not in their absolute values so lower precision work well as long as network is big enough.

The various sigmoid activation functions have the effect of keeping bit growth under control, by virtue of clamping to the +/- 1 range.

There are research papers where even 1 bit (not floating point) was enough, with some quality loss.

4 bits is effectively 16 different float point numbers - 8 positive, 8 negative, no zero and no NaN/inf. 1 bit for sign and 3 bits for exponent, 0 bits for mantissa, mantissa is implied to be 4. It’s logarithmic - representing numbers in the range from -4^3 to 4^3, smallest numbers are 4^-3.

Thanks, I was thinking that zero, negative zero, inf, negative inf, and the NaN's were included like in IEEE 754

>1 bit (not floating point)

I like how you specified that it's not floating point.

Thanks. First source i see for what fp4 is. Gotta say I'm surprised: I would have chosen to lose one value, but have a zero. (though I have no doubt those people are much more clever and knowledgeable than I am)

If the weight is zero it doesn’t need to exist

The fundamental 'unit' of NN computation is not an individual vector element but rather an entire vector. One of the first results you often learn about in linear algebra is that some axes are more important than others (principal components, singular value decomposition). Thus, it totally stands to reason that the underlying field of the vector is inconsequential but rather the entire vector machinery. All you have to do is make sure that there are enough elements in the vector to get the job done for whatever bit size of element.

I see, so the idea is that enough of the quantization errors are sort of averaged out across the dimensions of the vector space to still be useful?

The way I think about it is finally it will end in a binary feature vector similar to 20Questions (male or female, alive or dead ...) just with 100s of dimensions

That's right.There was mention of a precision aware transformer engine which might make it easier to use fp4, but it's not 30x faster in a like for like way. This shouldn't be surprising since it's more or less two hoppers next to one another on a slightly improved process node. 2.5x seems more likely in cases where you don't exploit a new feature like that or the increased memory.

yes

Reply on news.ycombinator.com