Hacker News Reader

Comment by andy99 | Hacker News Reader

Title misses that it's "weights and activations" as in they (from my skim) are doing all the math with mostly 4-bit values. Normally quantized weights are converted to 32-bits (edit: or more generally a type for which there are native machine instructions to multiply/add, as mentioned below) as they are applied during a forward pass which saves memory but incurs extra processing. They are keeping everything in (mostly) 4-bits to make it faster.

View on HN

It's pretty common to convert to bfloat16 nowadays since that's much faster. e.g. bitsnbytes now defaults to bf16 compute for 4bit quant.

You only get 80 characters for the title so I had to rewrite it to fit.

Might use activations rather than weights in the title then. Weights are commonly done, activations are not.

Alas, the edit window has closed. But I'll keep it in mind for any future submissions on that topic, thanks.

Huh, I thought this is what was usually meant by quantising

I believe the distinction they're drawing is they're also not "cast" to float32 during processing (which I also didn't know)

Yep I didn't know other models cast during processing

It's a nontrivial problem to do faster arithmetic operations with quantized values (which are not just rounded to fit in 4 bits but are usually block quantized with one or more scaling parameters stored at full precision) faster than to convert them to floats that the cpu/GPU is already optimized to multiply/add. That is what this project is proposing a solution to.

If you have a 4 bit signal multiplied by a 4 bit weigh, you can just use a lookup table for any operation with any output type you want. A 256 entry table of 32bit floats fits in 1K for example, which will fit in cache with plenty left over. This is independent of the quantization scheme so long as it's not dynamic.

It's quite unfavorable on modern hardware. A Sapphire Rapids core can do 2 separate 32 half-precision FMAs (vfmadd132ph, [1]) per clock, which is 128 FLOPs/cycle. It is not possible to achieve that kind of throughput with an 8-bit LUT and accumulation, even just a shuffle with vpshufb is too slow.

[1]: https://www.intel.com/content/www/us/en/docs/intrinsics-guid...

That's absolutely wild. If you really only needed vpshufb, the throughput is the same in terms of values, because there are twice as many values per register and you get to retire half as many instructions, but it takes a bunch more instructions to combine the two inputs and apply a LUT of 256 values :(

>> It's quite unfavorable on modern hardware.

Fair point. It might help if the system is DRAM bandwidth limited, so reducing the data size helps even though individual operations take multiple instructions. But that is not the situation with todays hardware.

Hmmm is there anything preventing dedicated AI chips to have this LUT built in and to vectorise it too?

Reply on news.ycombinator.com