Hacker News Reader

Comment by incrudible | Hacker News Reader

Naive array programming is not really high performance in my book, because performing a lot of trivial arithmetic on large arrays leads to poor locality and high bandwidth pressure. The better alternative is SPMD, i.e. something like CUDA or ISPC for CPUs. This is possible with some type of JIT if the numpy style of programming is to be maintained, for example tinygrad.

View on HN

Memory traffic is certainly the biggest problem that the array paradigm presents for implementation, yes. I'd quibble with calling that "poor locality": when working with large arrays, if any part of a cache line is accessed it's very likely the entire line will be used in short order, which is the definition of good locality as I understand it. The issue is simply high memory use and a large number of accesses.

I think it's an oversimplification to say SPMD is a better model full stop. Array programming has the advantage that the implementer optimizes a specific function, and its interaction with the rest of the program is much simpler: all the data is available at the start, and the result can be produced in any order. So things like multi-pass algorithms and temporary lookup tables are possible, particularly valuable if the program isn't spending that much time on arithmetic but rather "heavy" searching and sorting operations. Not that I claim NumPy or CuPy do a good job of this. Sections "Fusion versus fission" and "Dynamic versus static" are relevant here, I think: https://mlochbaum.github.io/BQN/implementation/versusc.html#...

If you like high-performance array programming a la "numpy with JIT" I suggest looking at JAX. It's very suitable for general numeric computing (not just ML) and a very mature ecosystem.

https://github.com/jax-ml/jax

You are absolutely right, naive array programming might be much faster than raw python but it will never be high performance because you can't use caches and memory bandwidth effectively.

The unstated major premise here is that you're working with data that's big enough that this becomes your major bottleneck.

There's also a subset - possibly even a silent majority - of problems where you're doing fairly intensive calculation on data that fairly comfortably fits into a modern CPU cache. For those, I'd still expect SIMD to be a great choice from a performance perspective.

I'm not exactly sure what this means, but the bandwidth to the level 3 cache isn't actually more than memory. If you only have a few megabytes to work with, it's a lot less likely performance is going to matter anyway, and if it does you would still want to do multiple operations on each array item if possible, because if you put a few more lines in a loop you're talking about registers at that point, not cache.

L3 cache bandwidth is often more than main memory; most systems at https://old.chipsandcheese.com/memory-bandwidth-data/ give at least a ~2x difference from whatever sample I clicked.

And on a recent generation Xeon, your mid-level data cache can total up to 64MB, depending on model. That's maybe the more interesting one? L3 cache is shared among cores, so access to it can be more heavily impacted by memory concurrency concerns and not just pure bandwidth concerns. Meaning there's potentially a strong incentive to keep your task size below 2MB per core.

What are doing that is computationally intensive and are using python and a CPU with 64MB of cache for, but is under 64 MB total? How do you know you aren't losing performance by doing single operations on each array in python?

Model training and inference, for example. Despite the hype, there's actually a lot of data science work that uses modestly-sized sets of structured data and doesn't really need a billion parameter deep learning model. And even when you're running inference on larger datasets it's often fine to just use mini batches.

I don't really think of it as a "losing" performance problem. It's all about tradeoffs. SIMD tooling often offers an easier programming model than SPMD tooling, and spending an extra 60 minutes of development time to save myself 60 seconds of run time is generally not a worthwhile trade in my problem space. Especially if I also need to run it on more expensive compute instances to realize that speedup.

It seems like instead of a "silent majority doing intensive calculations" what you're really saying is "I don't think of slower speeds as a performance loss, I don't pay attention to that, I don't care about slower speeds, my data is small and my programs run almost instantly".

If someone says "this isn't really high performance" and you say "I don't personally care, my stuff is small and doesn't take long time to run" that isn't a coherent reply.

It's really more about acknowledging that, unless you're explicitly talking about something like HPC, "high" is a relative term. It quite reasonably means different things to different people in different contexts. In Python I can often improve a task's run time by multiple orders of magnitude, getting its run time from "irritating" to "not really worth thinking about" in the process, just by making sure I'm doing vectorized operations that can make use of SIMD instructions in the appropriate places. That's more than high enough performance for my needs in most cases. A similar situation may be true for the author of this library.

I find it to be much less coherent, nearly to the point of knocking down straw men, when people get lost in arguing semantics over a term that we should all know by now is perennially ill-defined.

It's not ill defined to everyone else. If I say my phone is fast but it's nowhere close to the speed of a new phone, no one else would say it's fast.

High performance means you show that its speed is competitive against the other fastest programming methods.

You labeling something "high performance" because you wrote something slow and made it faster has nothing to do with releasing something and calling it "high performance" publicly.

Reply on news.ycombinator.com