Hacker News Reader

Comment by dist-epoch | Hacker News Reader

Not only that, but you need to decode the SIMD instruction only once, and not N times. Same with other stages of the pipeline.

View on HN

Yep, I've heard that actually something like 80% of the energy of the CPU is used by the front-end! Beyond decoding, you also have all the energy spent on scoreboarding data dependencies and reordering the instructions and scheduling to execution ports. If you can do that 8x less as in the case for SIMD, that's a huge win!

That's definitely wrong (unless you use an incredibly expansive definition of "frontend"). The simplest way to prove that it's wrong is that heavily vectorized workloads can easily cause a 30% increase in power consumption over scalar workloads even though the instruction density is similar.

Wouldn't that support the idea that most energy consumption is in decoding? If you're getting 2x, 4x, 8x ect as much value computation per instruction and yet only a 30% increase in power then clearly most the power is not used by computing the values.

no, because there's a lot the CPU does that is neither decoding nor execution. There's also caches, register renaming, branch prediction, inter core communication for atomics, and a dozen other things.

Sorry poor terminology use on my part. I mean more broadly that most energy is used on frontend and middle end, rather than backend and that this is what vectorisation improves in regards to energy consumption. Register renaming and branch prediction energy consumption should be improved in the same factor as decoding. Caching probably less so (depending if we are talking instruction, data or combined).

I don’t think inter-core communication is too relevant when comparing vectored and non-vectored on a single core, but definitely would be when batching across multiple cores.

I wonder how efficient a modern x86-64 CPU could be — on a scale from “CISC” to “single-purpose ASIC” — if you could compile and load microcode directly into cache lines, pre-baking all of this just-in-time silicon allocation decision-making into a static microcode loop, so that the frontend can shut off entirely.

(Presumably doing this would only be possible in the context of a single-threaded uninterruptible unikernel workload with system-management-mode functions disabled — but I’m sure a lot of “one powerful single-core SoC”-type embedded systems would be happy to make that trade off!)

Forgive me if i'm misunderstanding, CPU's have micro-op caches that will bypass decoding instructions altogether if its in that cache. This means that variable length instructions will be decoded to a micro-op in the cpu and will get reused for as long as you have cache hits. Which does mean you"ll have ghost performance penalties if your code/gcc doesn't respect the size and limitations of hidden caches. Otherwise you can pretend that any instruction length gets shortened to its micro-op.

I think I was imagining a useful operation level lower than the regular uOp — one less akin to programming a RISC processor, and more akin to the direct control over individual RTL signal lines — similar to the "instruction word" of signal-line states directly encoded by a row of the op-decoder PLA of a 6502.

Right now, AFAICT, even when executing a uOp stream from L0 cache, several "planning" systems are still active — juggling caches around and deciding routing between them; renaming registers; switching between power "license" states; etc. None of these decisions are explicit even in uCode for modern CPUs, so they have to be made, over and over again, even when running from uOp cache. Which means the silicon that's making these decisions can't ever go dark, even when running from uOp cache.

A "nanocode" would be a version of microcode that burns in all these "planning"-system decisions — and which can thereby put all the "planning" silicon to sleep. There would be "nanoOps" for explicitly wiring registers to other registers or cache buffers, for adding precise numbers of delay cycles, etc. And these would all happen at Nth-of-a-cycle-precise points in the execution stream, indicated by either entire nOp instructions, or pragma bits on nOp instructions.

(Given this, to usefully function, these "nanoOps" would either likely need to be bytecode and be decoded+executed at some ridiculously high frequency relative to the regular CPU clock — or they would need to be some arcane parallelized packing of what was originally a serial stream of uOps, such that you get VLIW nOps where all the pragmas you want to have happen for the next CPU cycle are indicated in the instruction-word at once as individual signal-line bits. Just like the 6502 PLA output "instruction word", actually!)

I know this probably sounds like nonsense — the instruction stream would be so much more bloated than microcode that it probably wouldn't be worth it. But that assumes an instruction stream that needs to live in RAM and get shuttled through layers of caches to reach the decoder. But what if the instruction stream could be loaded into a reserved SRAM area within the CPU itself — an SRAM that would effectively act as EEPROM (in that it would be loaded from CPU NVRAM on boot); and which you could directly drop the instruction pointer into (in nCode decode mode), skipping RAM+caching entirely?

I mention this, because I've always had a vague hypothesis that Intel x86 CPUs specifically already have something akin to this implementation of a "nanocode" + reserved SRAM area that holds some of it — specifically for the purpose of programming custom dynamic RTL for instructions after CPU release to hotfix CPU errata. It's something they had to learn the necessity of over and over again, after releasing CPUs with bugged instructions, all the way back to the FDIV bug.

Sadly I wouldn't know enough about microarchitecture to understand whether nanocode would actually be a benefit. surely if CPU's became big and complex enough there would be a benefit to have hierarchical translations down to micro-ops and smaller subdivisions for execution units. What you're describing seems to come close to FPGA's. The possibility to write very complex sequences to pipeline different payloads trough the system while maintaining proper state. Though I think the economical solution might be more boring, the intel ME and AMD PSP hardware is probaly responsible for alot of management of state that could flush or change how instructions are executed with very basic logic.

If esoteric hardware interests you I would recommend IBM system/360 as reading material. Sadly hardware like that has fallen out of fashion and we've been stuck with the x86 and arm dichotomy. if anyone else has interesting/newer non-x86 hardware to link I'm more than happy to delve into that.

Memory traffic too, assuming the SIMD code is properly optimized, you can slam nicely aligned blocks of memory straight into a SIMD register rather than loading each element one at a time.

this is less important than you would expect because memory gets fetched into caches 512 bits at a time. As such, the difference in power use is only the cost of loading from L1 cache (since the first instruction will bring the line to l1)

Given the massive # of instructions that an M3 core can keep in flight, I suspect that Apple’s CPU engineers could (but won’t) write some really interesting papers on this challenge.

Reply on news.ycombinator.com