Hacker News Reader

I've worked on a project to compile a domain-specific description language down to both SIMD (by using Intel SPMD Program Compiler) and GPGPU (by using CUDA and OpenCL) Monte Carlo simulation code. We also have a Scala/Java interpreter for debugging.

Are there workloads that benefit from very wide SIMD vectors that aren't good fits for GPGPUs, as long as the GPUs support 64-bit floats close enough to IEEE-754 for your needs? I understand the overhead in shuffling data between main memory and GPU memory, and synchronization overhead, but most code I'm familiar with that does heavy number crunching suitable for very wide SIMD tends to do that number crunching off on threads that don't have much synchronization with threads doing more general-purpose computation.

On a side note, ISPC's input language is deceptively close to C, but little traps lie in wait. I remember helping an intern debug his port of some of the Java code I wrote over to ISPC, but it turns out that mulitplying a long long by a double in ISPC results in a long long. Our attempt to scale down a random 64-bit integer to the range [0.0, 1.0) was always resulting in 0.0. I get that integer calculations are faster, and I could understand disallowing implicit casts, but making a language so close to C, but with different implicit casting rules is just asking for trouble.

You're in a bit of a different domain - I've never really done all that much stuff with heavy number crunching (I've done a bit of work on random forest traversal, but that's more about logic than about FP). And I have never worked with ISPC.

I think number crunching workloads are typically quite suitable for GPGPU - I'm certainly not trying to "debunk GPGPU", just saying that there are a lot of integer/logic intensive workloads that involve rapid switching back and forth between control/GPR-based-logic sides and "SIMD tasks" (e.g. Hyperscan switching between NFA/DFA simulation and "acceleration", which was SIMD-based character skipping).

Dan Luu does brilliant things parsing strings with SIMD instructions but the programming model is busted if you have to recompile (and maybe hand code assembly) for each chip.

I was involved with wrapping up a deep neural net application just as GPGPU was about to take off. We used hand coded SIMD, performance was fine, but we were not going to recode for the next generation from chips, change our error function, or change anything SIMD related.I

Before there was MMX computers came with vector units that did not have a programmer visible size, such as the Cray 1 and the vector unit for the 3090 mainframe. As it is, people hardly use vector instructions or if they use them it is something a few generations old so people pay for vector units that they don't use.

simd is a better fit than gpu for a lot of data warehousing workloads where you aren't doing much computation on each individual value and it's not worth the overhead of sending the data to the gpu. also true for search index performance optimization.

I'm sold, and not sold. Having parsed terabytes of JSON, I was impressed by your work. Regexps and other operations are common-place, and if libraries like yours were in broad use, I could definitely see the appeal.

I'm sold on this as a technical pathway to performance.

Problem is, I do it in Python. I have a hard time laying out the breadcrumbs between simdjson and hyperscan, and practical, general-purpose code running more quickly on Intel CPUs. My code will run faster on AMD CPUs in practice even if, in abstract, given a few extra hours of coding, it could run faster on AVX-512. I'd rather wait a few hours for my computer than port my code to simdjson. I think that's true for 99% of the code out there.

Didn't we go down this path with Pentium IV's deep out-of-order pipeline and Itanium's VLIW before? Where Intel had great theoretical performance, but software didn't keep up?

For this to make a dent, Intel would need to hire a teach of engineers so Firefox and Chrome both took advantage of the new instructions. That's perhaps more practical than in the Itanium and PIV era, but that doesn't mean it will happen.

> Where Intel had great theoretical performance, but software didn't keep up?

> For this to make a dent, Intel would need to hire a teach of engineers so Firefox and Chrome both took advantage of the new instructions.

That's more or less Intel's standard practice with its Intel compiler suite and MKL-and-similar libraries.

If there are performance gains from SIMD being left on the table, some of it is on Intel for not building the middle tooling necessary to make it easy.

(1) Most software isn't built with the Intel compiler suite or the MKL-and-similar-libraries.

(2) Even if Firefox/Chrome were built with those, they JIT Javascript. It's the JIT that matters, not what compiles the browser.

Oh, I should note that an interest in large SIMD is not confined to Intel - the ARM world is (slowly) heading towards SVE (or so my friends from that quarter assure me). So it's not just a partisan pro-Intel or pro-x86 thing on my part.

RISC-V is also adding Vector extensions. The cool thing about those is that they are variable length, so code can take advantage of newer hardware / degrade gracefully on less capable one.

https://gms.tf/riscv-vector.html

> so code can take advantage of newer hardware / degrade gracefully on less capable one

I never user variable-length SIMD on CPUs, but I have substantial experience with current SIMD up to AVX2, and GPGPUs. I’m not sure I like that variable-length stuff.

1. Variable-length SIMD is only suitable for vertical operations. Many useful algorithms are only possible to vectorize with shuffles and such.

2. With registers size unknown to compilers, it’s impossible to create a calling convention to pass data in registers, as opposed to RAM. RAM is slow on modern hardware, and is becoming slower over time (in relation to compute speed; nanoseconds are fine). GPUs are OK with that because they are massively multithreaded, they switch threads instead of waiting for data from memory. CPUs can’t do that, they only have 2 hardware threads per core.

I think going for variable length is a Bold Move, and am not sure I like it for an ISA that otherwise is trying to be conservative. I'll be more optimistic about it when I see SVE (a) on more than one HPC machine and (b) implemented in a variety of widths. I'll be impressed if it works, but I'm not going to be impressed ahead of time with it while it's an aspirational design because it has aspirations.

The small, medium and large sizes built into arms neon accomplishes some scalability without going back to what vector units were like in the big hair 1980s.

"...SIMD is underutilized in general purpose workloads... Daniel Lemire and I built simdjson to show the power of SIMD in a non-traditional domain..."

To your point, the recently front-paged simdjson writeup made me regret ignoring SIMD. I had dabbled in 3D graphics and matrix math stuff in the 90s and somehow continued to associate those extensions with that kind of work.

My bad.

Thank you for daylighting this stuff.

> AVX2 had downclocking issues when it first appeared too - but these were rapidly mitigated and forgotten.

Speaking of Daniel Lemire, he recently published a paper https://arxiv.org/abs/1910.05109 mentioning downclocking in section 4:

> On some Intel processors, AVX-512 instructions are subject to downclocking: whenever AVX-512 instructions are executed, the processor lowers its frequency. We do not observe any downclocking on our processor, nor did we find any Intel documentation referencing downclocking for this processor. To test for downclocking, we used the avx-turbo benchmarking tool: it reports the processor frequency after issuing long sequences of expensive instructions of various types,including AVX-512 multiplication and floating-point instructions.

> partly due to Intel's fecklessness in promoting SIMD - including gratuitous fragmentation - e.g. a new Comet Lake machine with Pentium branding doesn't even suport AVX2 and has it fused off

So, Intel's problem is that it is now run by marketing droids, not engineers.

Since you sound quite expert -- I would love to learn from you (or others), at a beginner's level:

Is the design + production of a chip an exercise in optimizing the combination of:

-- size of die (and therefore overall cost/yield?)

-- power consumption

-- number / type / purpose of compute and memory subcomponents packed into the area

-- speed

Is that basically the envelope that you're doing an optimization within, and until a technology jump, even Intel/AMD/ARM-based are all simply choosing values between these dimensions? And each different choice of parameters within these basically goes after a certain segment of user?

Thanks!

I am not a hardware design expert, but... yes. Everyone makes their choice between speed, power consumption and cost. I attended a event once at Intel where someone told the anecdote that, "If someone comes to me and says, gee, I'd like a chip that runs faster, consumes less power and is cheaper to make I tell them 'you are overpaid for whatever it is that you do'".

The technology jump is of course the thing that allows you to "have it all" - until it doesn't. So it's clear that the optimization of Intel's 14nm++++++ is better for speed right now than anything they have at 10nm, which is better for density and power consumption. It used to be that the shrinks were unambiguously better but we've entered a strange new world.

Honestly, from the design perspective, I'm a enthusiastic spectator - to the extent I have insider knowledge it's out of date, poorly understood and not shareable in any case.

Microarchitecture is still very important - a better uarch and get you big effects without process changes. The last unambigiously successful one of these was Broadwell to Skylake. I think it will be fascinating to see the backport of Sunny Cove to 14nm ("Rocket Lake") as the difference between Rocket and Comet Lakes will give us a great natural experiment in how much microarchitecture changes matter without the usual distractions of process changes.

That used to be the "tick" in "tick/tock" but the uarch change is much bigger than usual and the process changes from Comet to Rocket will (I think) be quite small.

Doesn't AVX-512 have a problem with frequency scaling?

https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...

https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html

Given this, and the fact Ryzen doesn't support AVX-512, I would be very wary of the technology.

From the comment you are replying to:

> The biggest meta-problem with AVX-512 is that due to process issues, the pause button got hit just as it appeared in a server-only, 'first cut' form. AVX2 had downclocking issues when it first appeared too - but these were rapidly mitigated and forgotten.