Hacker News Reader

Comment by AnimalMuppet | Hacker News Reader

Can anyone comment on the performance of this hardware? How does it compare to state of the art, human-designed hardware? Is this actually an improvement? (To get to recursive self-improvement, you first have to improve at all.)

View on HN

This is the smallest unit of a typical AI ASIC, for example Google's TPU would have several dozen more compute units inside of it per chip.

In essence this is the simplest unit of an entire AI chip. The more complicated units of AI ASICS are actually the periphery, especially around PCIe and Ethernet and the sub-systems that link many AI ASICs together to move huge amounts of data around ultimately to each TPU.

So its missing ALOT

Thanks. But that wasn't my question. For this part, how is the performance? State of the art? Better? Or worse?

Its a SYSTEM on Chip, evaluating 1 function on performance is superficial. This could have the best performance in the world, and it doesn't matter if the bottleneck is upstream

It's at 80~90% the max perf it can achieve on this hardware... Which is DDR3 speeds. I'm working on getting FPGA's with DDR4 and HBM2 next.

Why is there any point in trying with faster memory if you are only at 80% perf on DDR3? You'll just go to 40% perf on the next iteration?

You should only upgrade when the bottleneck becomes memory, right now the bottleneck is still on compute until you are at 100% max perf

Yeah thats true, chip to chip communication and KV cache management is make or break for an AI ASIC.

Reply on news.ycombinator.com