Comment by Const-me | Hacker News Reader

Comment by Const-me | Hacker News Reader

Parent Full thread

Indeed, automatic vectorizers do such simple things pretty reliably these days.

However, if you build your software from kernels like that, you leave a lot of performance on the table. For example, each core of my Zen 4 CPU at base frequency can add FP32 numbers with AVX1 or AVX-512 at 268 GB/sec, which results in 806 GB/sec total bandwidth for your kernel with two inputs and 1 output.

However, dual-channel DDR5 memory in my computer can only deliver 83 GB/sec bandwidth shared across all CPU cores. That’s an order of magnitude difference for a single threaded program, and almost 2 orders of magnitude difference when computing something on the complete CPU.

Even worse, the difference between compute and memory widens over time. The next generation Zen 5 CPUs can add numbers twice as fast per cycle if using AVX-512.

For this reason, ideally you want your kernels to do much more work with the numbers loaded from memory. That’s why efficient compute kernels are often way more complicated than the for loop in your example. Sadly, seems modern compilers can only reliably autovectorize very simple loops.

On the other hand, enregistered value access is free; and L1 cache is .. what.. 2 cycles?

But you're right. It's hard to come up with enough computing to interleave with the actual expensive part, which is accessing memory. Even L2 cache isn't really fast enough to not be a bottleneck for typical vectorized operations.

If you look at TPU architectures, the the general pattern is: fast local large L1-cache-grade memory, preferably multi-ported, or multi-banked. Plus compute (whatever). The important bit being the memory architecture, not the compute. Plus blistering fast communication between cores over which results get streamed at speeds that are still not really fast enough.

Interestingly, I sat in on architecture meetings three decades ago, where Intel architects were privately telling us: "compute doesn't matter any more; it's all about memory speed".

> L1 cache is .. what.. 2 cycles?

On Zen 4 CPU, I believe the typical latency of L1D is 4 cycles. However, if you (or your compiler) write AVX code which adds these floats, will still bottleneck on memory even if both inputs are in L1D cache. Each Zen 4 core can sustain two vaddps instructions per cycle, two loads per cycle, and one store per cycle. Due to the load and store bottlenecks, that kernel will only do one vaddps per cycle i.e. will waste 50% of theoretically available compute power.

> Intel architects were privately telling us: "compute doesn't matter any more; it's all about memory speed"

To be fair, that’s only true for automatically vectorized code, kernels like in the GP’s example. With sufficient efforts spent on software development, for some practical problems it’s possible to write codes which do saturate compute.

An example of such problem is multiplication of large matrices. A carefully written manually vectorized implementation should bottleneck on compute not memory, because theoretically required memory bandwidth scales as N^2, while theoretically required FLOPs scale as N^3 where N is size of the matrix.

That’s precisely what many BLAS libraries are doing under the hood. For the same reason GPU vendors report ridiculously high numbers of theoretical TFlops when multiplying low precision matrices with these special AI blocks, wmma/mfma instructions on AMD, tensor cores on nVidia.

Reply on news.ycombinator.com