Hacker News Reader

Comment by dmos62 | Hacker News Reader

To someone just hearing about SIMD, anyone care to give an experience infused introduction? Is it worth the hassle only in rare cases?

View on HN

GPUs are SIMD machines, does that color whether it seems rare? Coming from a graphics background and working on GPUs, I am super biased, but I say it's very worth the hassle.

The easiest intro IMO is check out & play around in ShaderToy. You don't have to know much about SIMD to write shaders, but once you start paying attention to how the machine works, you can make it go really fast.

In general, using something like CUDA is similar to C++ you just have to make sure all your threads do as close to the same thing as possible in order to see good perf.

If you are just hearing about it, there's a good chance that you're not doing any kind of elaborate matrix math, so it's tough to say if it's useful.

It's incredible useful if you're doing a lot of stuff involving matrices (graphics, image/video processing, neural-net stuff, stuff like that). If you have any interest in that topic, it's absolutely worth learning how to use SIMD (or at least learning a library that takes advantage of SIMD in your language of choice).

It's certainly useful for quite a lot of mathematical science code. In my experience compilers are not very good at autovectorizing anything but the most simple loops and writing SIMD intrinsics is necessary to obtain the maximum output of the processor.

The cpu just takes a bunch of numbers and applies the same operation on them in one instruction instead of a bunch single ones. E.g. if you are writing some simulations/game code where you deal with 3d vectors a lot the use case is there a lot. Is it worth writing the custom instruction intrinsics yourself? Most likely no as the compiler often (!) does a good job at auto vectorizing or the underlying library already uses them (I think c# vector types does).

Compilers are really smart, so in many cases they will vectorize without even asking - no hassle involved. Some examples here: https://llvm.org/docs/Vectorizers.html

GCC and Clang will try their best to vectorise simple code, but if you try to write more complex algorithms, they will have a more difficult time and you may end up trying to massage your code to be more vectoriser-friendly. Auto-vectorisation is no substitute for a predictable SIMD programming model.

They tend to stumble over data dependencies and for floating point math they can't reorder instructions as well (unless you explicitly disable IEEE754 compliance).

There are a number of structured approaches to rewriting code to be SIMD friendly, but you do need a degree of explicitness to get what you want out of the compiler.

Some proprietary compilers provide tools to explicitly instruct a block of code to be vectorized via annotations

> Is it worth the hassle only in rare cases?

Vectorization is not always faster. It's important to understand that modern processors can perform work on >100 instructions in a given cycle, and not all instructions take equal amounts of time. So reducing a dozen instructions to a single instruction doesn't necessarily mean that the single instruction is going to be faster.

This is misleading, at least if you're looking at a single core.

A single core can have >100 instructions in flight, but most of them will be waiting for their operands to arrive. The theoretical maximum throughput is gated by the instruction decoders and execution units, and is an order of magnitude lower.

That's fair to call me out on that, what I said wasn't clear. I think the broader point still stands though: don't look at the number of ISA instructions and think that fewer instructions necessarily means performance improves.

I think that we also miss a piece of the puzzle in terms of what happens once you leave ISA instructions and get into uops. There's magic inside there and it's completely opaque.

> So reducing a dozen instructions to a single instruction doesn't necessarily mean that the single instruction is going to be faster.

Are you saying dozen scalar instructions can be faster than one vector instruction? That's wrong 100% of time on modern CPUs.

Yeah, I think there exists a case where this is possible. Let me play around with it this weekend and I'll comment here if I find anything.

I know it can happen on some older ARM designs, microcontrollers and weird DSP-like etc. chips. But I can't think of any case on modern x86 chips at least.

Some low-performance ARMv7/8 designs can split NEON SIMD instructions into multiple clock cycles, but I think even then NEON is going to perform better.

I started digging into it but I don't have an intel machine with linux easily available and that's a prerequisite for looking at microop performance. I think you win.

I don't need to win, I just want the truth to win. In other words, I'd consider it a win if I, you or anyone else learns something new.

SIMD is highly optimized at this point, with sustainable two instructions per clock throughput. It's hard to imagine scalar getting anywhere near even at 4 inst/clk rates.

One corner case that exists is that using AVX instructions imposes a frequency limit, although this isn't the case for SSE instructions.

There exist some vector instructions that are going to be slower than a non-equivalent sequence of scalar instructions: VPGATHER is going to be an easy such case.

However, I doubt there are going to be any cases where a vector instruction will take fewer clock cycles than its equivalent scalarized instructions. There are some where it might be equivalent--a vector of 2 elements performing an operation that can be issued twice a cycle is an easy example--but I can't think of any where it would be worse. If that were the case, then you should just implement the operation in hardware by scalarizing the uops (and some instructions appear to be so implemented--e.g., gather/scatter).

VPGATHER is actually significantly faster than scalar loads at least on Skylake, possibly on some earlier CPUs as well (Broadwell?).

On Haswell where it was first introduced... yeah, not very fast, like you mentioned.

Reply on news.ycombinator.com