Hacker News Reader

Comment by dragontamer | Hacker News Reader

At this rate, we're going to end up with FP1 (1-bit floating point) numbers...

I guess that's nonsensical. 1-bit in all FP is the sign bit. So I guess the minimum size is 2-bit FP (1-bit sign + 1-bit exponent + 0-bit implicit 1 mantissa)

View on HN

Binary neural networks, where weights and/or activations are just 0/1s, are an active research area. In theory they could be implemented very efficiently in hardware. But in contrast to FP16 (or to some extent, int8), just quantizing FP32 to 1 bit doesn't work very well. There have been successful methods in practice. There was a company called Xnor.ai that was built partially around this technology, but it was sold to Apple a couple years ago. I don't know what's the current SOTA in this area, though.

We have created a binary embedding to help us with the embedding sizes. I think in the future we will see a lot more research in reducing the model and embedding sizes.

https://medium.com/ozonetel-ai/compressing-bert-sentence-emb...

*binarized

At FP2, you are probably better off with {-1, 0, 1, NaN} (sign+mantissa) rather than sign/exponent. You basically bit pack.

FP3 gives you sign, 1x "exponent", 1x mantissa, so still kinda bit packing.

I could see FP4 with sign, 1x exponent, 2x mantissa. Exponent would really just be a 4x multiplier, giving +/-0,1,2,3,4,8,12

Or invert all those, so you are expressing common fractions on 0..1

Real life has the E3 series: 1, 2.2, 4.7, and then 10, 22, 47, 100, 220, 470, 1000, etc. Etc.

EEs would recognize these values to be the Preferred Resistors values for projects (though more commonly the E6 series is used in projects, the E3 and E1 values are preferred)

That's 3 values per decade, which is slightly more dispersed than a FP4 that consists of 1 sign + 3 exponent + 0 (implicit mantissa 1 bit).

Or the values -128, -64, -32, -16, -8, -4, -2, -1, 1, 2, ... 128.

Maybe we can take -128 and call that zero instead, cause zero is useful.

--------

Given how even E3 is still useful in real world electrical engineering problems, I'm more inclined to allocate more bits to the exponent than the mantissa.

> Real life has the E3 series: 1, 2.2, 4.7, and then 10, 22, 47, 100, 220, 470, 1000, etc. Etc.

Took me until today to realise that sequence is a rounded version of 10^(n/3) for integer n.

Almost so, there are some manual adjustments to help with overlap from tolerances at various places, but that pure math layout of it would be where you'd probably want to do it for ml

E3 is only relevant to decimal numbers. I don't see how computing can benefit from it, unless base conversion for human interaction is a particularly large part of the problem. And there is a reason why stuff like BCD isn't used anywhere near anything resembling high performance computing: it's practically never worth it.

If you're going to bother doing floats you should probably make them balanced around 1.

And exponent seems to be much more important for these small sizes. The first paper that shows up for FP4 almost has negative mantissa bits. Their encoding has 0, 1/64, 1/16, 1/4, 1, 4, 16, 64.

1 bit would work fine - make the values represent +-1 or so

That once we get into asymmetrical number coding so that you could use numbers that take fraction of bits.

I think I read somewhere it only goes as low as int4. Can't find the reference.

1-bit unsigned float would probably be useful to some

So +0, +infinity, -0, -infinity, no NaNs?

Reply on news.ycombinator.com