Comment by w-m | Hacker News Reader

Comment by w-m | Hacker News Reader

I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner workings of floating point representations. About zero values it says [0]:

> Yes, the floating point standard specifies both +0.0 and −0.0. This concept is actually useful because it tells us from which “direction” the 0 was approached as a result of storing value too small to be represented in a float. For instance -10e-30f / 10e30f won’t fit in a float, however, it will produce the value of -0.0.

The authors of the LLM paper use the values {-1, 0, -1}. Connecting the two ideas, I'm now wondering whether having a 2-bit {-1, -0, 0, 1} representation might have any benefit over the proposed 1.58 bits. Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")?

Also, I've seen 2-bit quantizations being proposed in other LLM quantization papers. What values are they using?

[0] https://ciechanow.ski/exposing-floating-point/#zero

> Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")?

Probably, but is it worth the cost? One of the goals behind BitNet and this paper is to find a way to implement LLMs as efficiently in hardware as possible, and foregoing floating point semantics is a big part of it. I'm not sure if there's a way to encode -0 that doesn't throw out half the performance gains.

But if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.

You can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.

Can processor perform addition on them effectively?

How exactly would you do that? 3 states need 1.58 bits which is a tad more than 1.5. Two 3-states have 3²=9 states while three bits only give you 2³=8 states.

I wonder if there's some encoding tricks you can use to reduce it to 8 (or less?) effective states, given that you're only using them with a reduced set of mathematical operations. E.g., can you automatically convert all (-1, 1) to (1, -1) and save one encoded state, since they add up to the same result anyway?

You can use a bit for zero or non-zero and then use bits only for providing the sign to non-zero values, for example. The sign part will be variable length but can probably be made very fast with hardware support.

Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0?

> Could the additional -0 carry some pseudo-gradient information

It looks like training was done on fp32 or bf16. Low-bit quantization is approximated with STE during training. I'd expect training itself cause each point to "polarize" towards 1 or -1.

> 2-bit quantizations being proposed

Symmetric (i.e. without 0) exponential values were pretty popular IIRC.

> how do you use -0 in the add

In my mind the two zero values would represent a tiny epsilon around 0, let's say -0.01 and +0.01. Looking at them like this, it would mean

  +0 +0 -0 = +0
  +0 -0 -0 = -0
  +1 * +0 = +0
  -1 * +0 = -0

Performing addition with the same sign count in each group would be problematic. How to decide on the sign of +0-0 or +1-1, other than flipping a coin?

maybe they could be stored together in two words until they're operated on and lose their pairing?

Or use -1, 0, 1/2, 1 where the new half-weight is still a cheap bit shift.

I would guess that having 2 zeros is not that useful for NNs, but in general with 2 bits we could encode 4 states, so are there 4 possible states that would be useful to encode? Sure, but would this be better than encoding 3 states? That's the entire question imo. I would guess that 3 states are probably better, because negative/neutral/positive seems the minimal signal that we need these weights to provide.

You could use a negative-two base, and encode {-2, -1, 0, 1}. See https://en.wikipedia.org/wiki/Negative_base

Or you could use the regular positive-two base and encode {-2, -1, 0, 1} the normal way with two's complement.

You might also use a basis of negative-two and use two bits to represent {-2, -1, 0, 1}.

Negative bases are fun. See https://en.wikipedia.org/wiki/Negative_base

Reply on news.ycombinator.com