Hacker News Reader

Comment by bschne | Hacker News Reader

but isn't that aspect common among this, CPUs, GPUs, ...? And it feels like the whole NOC thing would add quite some overhead to moving things around.

Are you saying proximity here more than offsets this vs. e.g. each core having its own cache as I think they do in a "normal" CPU? And if so, is this more true of ML inference workloads than other workloads, for some reason?

View on HN

I think the distinction with ML inference workloads is that you often have very little control flow, so this type of architecture lets you match layers to adjacent cores so that each operation gets it's data directly from the step before rather than from RAM.

Right, so latency is less of an issue (because control flow is predictable) and performance becomes about bandwidth.

I think the NOC approach is fundamentally similar to Intel's rings that they used for core interconnect back in the mid-2010s. It works.

https://www.realworldtech.com/includes/images/articles/snbep...

https://en.wikichip.org/wiki/intel/microarchitectures/sandy_...

They still use this today, and the ring interconnect is also the topology inside zen3/zen4 CCXs (with 8 cores). Ring is one of the simplest and best systems for >4 cores, until you get to about 8-10 cores (at which point you generally split it into multiple “tiers” like multiple CCXs etc).

Intel switched to a mesh model: https://en.wikichip.org/wiki/intel/mesh_interconnect_archite...

not for consumer chips, which still use the same ringbus design.

e-cores do have a CCX/core cluster, but the clusters themselves go on the ringbus lol

Reply on news.ycombinator.com