It looks like this is a quantization method to flatten matrices for vector addition. Can anyone explain how this could allow LLMs to reach current benchmarks without losing performance?
Once you do that, a dot product is just addition/subtraction. Matrix multiplications are just dot products, so you've removed multiplication.
Then they built custom hardware that presumably only does that operation and doesn't use much electricity
Since this is just "more aggressive quantization" it's not too surprising that it reaches similar performance to other quantized models.
The network should structurally look the same as any other llm, it's still using transformers etc (afaiu)