The Inference Hardware Revolution of 2026
spectrum.ieee.org
spectrum.ieee.org
Of the areas mentioned in the article, which are the most likely to have the most prominent innovative impact, and what will they entail?
which is the Hennesey and Patterson computer architecture book would serve the role of the "dozens of books" hyperbole rather well.
1. having more memory on the card/chip and/or faster access to that memory;
2. integrated memory and compute units optimized for matrix and vector multiply add operations;
3. optimized load circuitry to e.g. read memory in the stride and span (next row, next column) access patterns common to matrices or ensure that no/few parts of the chip are stalled waiting on data or operations to complete.
Another aspect is quantizations. These are similar to SIMD vector operations in that you are performing an operation on a block of n-bit data values at the same time, so can have optimized circuitry.
For 2 or 3 valued quantizations you can reduce various addition and multiplication operations to logic operations, avoiding circuitry for things like the half-adder, full-adder, and carry-lookahead.
Then there's adding specific circuitry for common operations such as ReLU like is done in hardware acceleration of image, video, etc. processing. There's a trade off here as optimized hardware would perform better at the specific operations but if those are too specific then they can't be used by different/newer model architectures. (Though it does make sense to try and optimize common operations/logic where possible.)
It would be interesting to see if these designs can/will benefit training as well, as that would bring down the time/cost/energy of training large models as well as making it easier for local fine-tuning.
Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.
How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hiding tricks don't matter when you're bandwidth constrained.
As far as the interconnect to the GPU/CPU - thats a different story. But with Nvidia acquiring Mellanox and Nvlink getting faster and faster I assume well get there
I like how the author uses the analogy of scrabble word creation to describe LLM training but unfortunately the analogy didn't continue to inference and I got lost trying to keep up.
Did not know about this cool trick about storing numbers as exponents! Is there a name for this technique? Wouldn’t there be overhead in converting back and forth between the exponent and the number?
normalize the two numbers A and B to have the same exponent, add the mantissa, then convert back to IEEE 754?
The idea is rather than storing a number x as (exponent, mantissa), just store (log x) as a fixed precision number. Multiplying two such numbers is just addition, dividing is just subtraction. TBH I didn't read the article, but my reaction is that yes, that works, but one must sum all those products, and now summing becomes an expensive operation. Maybe the total cost saves area and power, but it beggars belief that it is 10x more efficient. They must be doing PR math: our low precision log scheme is 10x more efficient than a higher precision traditional approach.
Another thing to keep in mind is a lot of inference is done using very low precision math and so the cost of doing multiplies isn't that bad. Yes, it is still (n bits) squared, but as n gets small, n^2 still isn't too bad.
ps. Also used in the original circuits of the Yamaha DX7 synthetiser ( https://www.righto.com/2021/11/reverse-engineering-yamaha-dx... ).
I knew of this but not the $ amount. Wow