Comment by jerf | Hacker News Reader

Comment by jerf | Hacker News Reader

I post this in the spirit of being corrected if I am wrong, the Internet's core skill.

If I'm reading this correctly, the competition for this technology isn't CPUs, it's GPUs. If that's the case, it seems like they'd be able to show a lot of artificially inflated numbers vs. CPUs simply by running already-known-GPU-heavy workloads on their hardware vs. CPUs and show massive performance gains vs. the CPUs... but what about an optimal GPU algorithm?

Traditionally, making the hardware that can support reconfigurability at a highly granular level (FPGAs, for instance) has never been able to catch up with conventional silicon, because the you can't recover the quite significant costs of all that reconfigurability with improved architecture. The conventional silicon will still beat you with much faster speeds even if you take a penalty vs. some theoretical maximum they could have achieved with some other architecture.

Plus, again, GPUs really cut into the whole space of "things CPU can't do well". They haven't claimed the whole space, but they certainly put their marker down on the second-most profitable segment of that space. Between a strong CPU and a strong GPU you've got quite a lot of computing power across a very wide set of possible computation tasks.

I am not an expert in this space, but it seems like the only algorithms I've seen lately people proposing custom hardware for at scale are neural network evaluation. By "at scale", I mean, there's always "inner loop" sorts of places where someone has some specific task they need an ASIC for, encryption being one of the classic examples of something that has swung back and forth between being done in CPUs and being done in ASICs for decades now. But those problems aren't really a candidate for this sort of tech because they are so in need of hyperoptimization that they do in fact go straight to ASICs. It doesn't seem to me like there's a huge middle ground here anymore between stuff worth dedicating the effort to make ASICs for (which almost by definition will outperform anything else), and the stuff covered by some combination of CPU & GPU.

So... how am I misreading this, oh great and mightily contentious Internet?

You are correct that the main competing technology is throughput compute (GPUs, TPUs, ML processors) rather than "general-purpose" CPU-style compute with batch size of 1. The way we use CPUs is fairly unique, though, where we time share a lot of different code on each core of a system. Trying to move CPU applications to an FPGA-style device would be a disaster.

I used to believe that FPGAs could compete with GPUs in throughput compute cases thanks to high-level synthesis and OpenCL, but it looks like most applications have gone with something more ASIC-like than general-purpose (eg TPUs and matrix units in GPUs for ML).

CPUs __ARE__ an ASIC.

You can get a commodity CPU today at 4.7+ GHz clock and 64 MB of L3 cache (AMD Ryzen 9 5950X). There's no FPGA / configurable logic in the world that comes even close. AMD's even shown off a test chip with 2x96 MB of L3 cache.

GPUs __ARE__ an ASIC, but with a different configuration than CPUs. An AMD MI100 comes with 120 compute units, each CU has 4x4x16 SIMD-lanes and each lane has 256 x 32-bit __registers__. (AMD runs each lane once-per-4 clock ticks)

That's 32MB of __registers__, accessible in once every 4th clock tick. Sure, a clock of 1.5GHz is much slower but you ain't getting this number of registers or SRAM on any FPGA.

> I am not an expert in this space, but it seems like the only algorithms I've seen lately people proposing custom hardware for at scale are neural network evaluation

The FPGA / ASIC stuff is almost always "pipelined systolic array". The systolic array is even more parallel than a GPU, and requires data to move "exactly" as planned. The idea is that instead of moving data to / from central registers, you move data across the systolic array to each area that needs it.

----------

CPUs: Today's CPUs __discover__ parallelism in standard assembly code (aka: Instruction level parallelism). It spends a lot of energy "decoding", which is really "scheduling" which instruction should run in which of some 8 to 16 pipelines (depending on AMD Zen vs Skylake vs Apple M1). Each pipeline has different characteristics (this one can add / multiply, but another one can add / multiply / divide). Sometimes some instructions take up the whole pipeline (divide), other times, multiply takes 5 clock ticks but can "take an instruction" every clock tick.

CPUs spend a huge amount of effort discovering the optimal, parallel, execution for any chunk of assembly code, and today is scanning ~400 to 700 instructions to do so (the rough size of a typical CPU's reorder buffer). That's why branch prediction is so important.

In effect: CPUs are today's ultimate MIMD (multiple instructions / multiple data) computers... with a big decoder in front ensuring the various pipelines remain full.

-----------

GPUs: SIMD execution. If parallelism is known ahead of time, this is a better architecture. Instead of spending so much space and energy on discovering parallelism, you have the programmer explicitly write a parallel program from the start.

GPUs are the today's ultimate SIMD computers (single instruction / multiple data) computers.

-------------

Systolic Array: Will never be a general processor, must be designed for each specific task at hand. Can only work if all data-movements are set in stone ahead of time (such as matrix multiplication). Under these conditions, is even more parallel and efficient than a GPU.

Remind what the ASIC acronym means

Application specific integrated circuit.

The Intel / AMD CPUs are an ASIC that execute x86 as quickly as possible. Turns out to be a very important application :-)

My point being: you aren't going to beat a CPU at a CPU's job. A CPU is an ASIC that executes assembly language as quickly as possible. Everything in that chip is designed to accelerate the processing of assembly language.

--------

You win vs CPUs by grossly changing the architecture. Traditionally, by using a systolic array. (Which is very difficult to make a general purpose processor for).

I think it's at least a fair question to ask what makes "execute x86" such an important application?

Sure the x86 cpu is the best way to execute x86 instructions, but so what?

I do not actually care about x86. I don't write it or read it.

I write bash and c and kicad and ooenscad and markdown etc..., and really even those are just todays convenient means of expression. The same "what's so untouchable about x86" is true for c.

I actually care about manipulating data.

I don't mean databases, I mean everything, like the input from a sensor and the output to an actuator or display is all just manipulating data at the lowest level.

Maybe this new architecture idea can not perform my freecad modelling task faster or more efficiently than my i7, but I see nothing about the macroscopic job that dictates it's already being mapped to hardware in the most elegant way possible translating to x86 ISA and executing on an x86 asic.

> I think it's at least a fair question to ask what makes "execute x86" such an important application?

x86, ARM, POWER9, and RISC-V are all the same class of assembly languages. There's really not much difference today in their architectures.

All of them are heavily pipelined, heavily branch predicted, superscalar out-of-order speculative processors with cache coherence / snooping to provide some kind of memory model that's standardizing upon Acquire/Release semantics. (Though x86 remains in Total-store ordering model instead).

It has been demonstrated that this architecture is the fastest for executing high level code from Bash, C, Java, Python, etc. etc. Any language that compiles down into a set of registers / jumps / calls (including indirect calls) that supports threads of execution are inevitably going to look a hell of a lot like x86 / ARM / POWER9.

----------

If you're willing to change to OpenCL / CUDA, then you can execute on SIMD-computers such as NVidia Ampere or AMD CDNA. Its a completely different execution model than x86 / ARM / POWER9 / RISC-V, with a different language to support the differences in performance (ex: x86 / POWER9 have very fast spinlocks. CUDA / OpenCL has very fast thread-barriers).

There's a CUDA-like compiler for x86 AVX + ARM-NEON called "ispc" for people who want to have CUDA-like programming on a CPU. But it executes slower, because CPUs have much smaller SIMD-arrays than a GPU. (but there's virtually no latency, because x86 AVX / ARM-NEON SIMD registers are in the same core as the rest of their register space. Like... 1 clock or 2 clocks latency but nothing like the 10,000+ clock ticks to communicate to a remote GPU)

----------

Look, if webservers and databases and Java JITs / Bash interpreters / Python interpreters could be executed by something different (ex: a systolic array), I'm sure someone would have tried by now.

But look at the companies who made Java: IBM and Sun. What kind of computers did they make? POWER9 / SPARC. That's the fruit of their research: the computer architecture that best suites Java programming according to at least two different sets of researchers.

And what is POWER9? Its a heavily pipelined, heavily branch predicted, superscalar out-of-order speculative core with cache-coherent acquire/release semantics for multicore communication. Basically the same model as a x86 processor.

POWER9 even has the same AES-acceleration and 128-bit vector SIMD units similar to x86's AVX or SSE instructions.

You get a few differences (SMT4 on POWER9 and bigger L3 cache), but the overall gameplan is extremely similar to x86.

There are some strange assertions there apart from the definition of ASIC. ISAs aren't assembly languages. I could believe there are may be more ARM and, particularly, RISC-V chips without all those features than with. Since when has C the language specified what it compiles to, to exclude Lisp machines? I read about Oak (later Java) on my second or third generation of SPARC workstation, when I'd used RS/6000. Few people remember what Sun did market to run Java exclusively. IBM might argue about the similarity of our POWER9 to x86 but I don't much care.

In comparing with CPUs I would also want to see thorough measurement of latency too.

Yes you are

Reply on news.ycombinator.com