Hacker News Reader

Comment by throwaway31131 | Hacker News Reader

"You don't think about memories when you design hardware"

I found that comment interesting because memory is mostly what I think about when I design hardware. Memory access is by far the slowest thing, so managing memory bandwidth is absolutely critical, and it uses a significant portion of the die area.

Also, a significant portion of the microarchitecture of a modern CPU is all about managing memory accesses.

If there were a hypothetical high-level CPU language that somehow encoded all the information the microarchitecture needs to measure to manage memory access, then it would likely tie in performance, assuming the CPU team did a good job measuring memory. Still, it wouldn't need all the extra stuff that did the measurement. So I see that as a win.

The main problem is, I have absolutely no idea how to do that, and unfortunately, I haven't met anyone else who knows either. Hence, tons of bookkeeping logic in CPUs persists.

View on HN

Keep in mind this article is almost 20 years old.

Memory access used to matter less because CPUs were a lot slower relative to memory. When I was a kid you could get a speedup by using lookup tables for trig functions - you'd never do that today, it's faster to recalculate.

Wait, what?

Oh, yeah. Jeebus.

2008 was SEVENTEEN years ago.

I turn 60 soon. Anything 2xxx just seems like almost yesterday, 199x isn't that far away in mind, and even 198x is pretty close.

I love math (B.Sc. Physics, so more math than anything else) but sometimes math hurts.

Damn you, beloved math.

This certainly rings true with my own experiences (worked on both GPUs and CPUs and now doing AI inference at UK startup Fractile).

> If there were a hypothetical high-level CPU language that somehow encoded all the information the microarchitecture needs to measure to manage memory access,

I think this is fundamentally impossible because of dynamic behaviours. It's very tempting to assume that if you can be clever enough you can work this all out ahead of time, encode what you need in some static information and the whole computation just runs like clockwork on the hardware, no or little area spent on scheduling, stalling, buffering etc. Though I think history has shown over and over this just doesn't work (for more general computation at least, more feasible in restricted domains). There's always lots of fiddly details in real workloads that surprise you and if you've got an inflexible system you've got no give and you end up 'stopping the world' (or some significant part of it) to deal with it killing your performance.

Notably running transformer models feels like one of those restrictive domains you could do this well in but dig in and there's plenty enough dynamic behaviour in there that you can't escape the problems they cause.

> I think this is fundamentally impossible because of dynamic behaviours.

I was thinking that too as it sounds awfully close to a variant of the halting problem.

But since I am not sure that it's actually equivalent, and there might be some heuristic that constrains the problem, in theory, to make it solvable "enough" I went with the softer "I don't know how to do it" rather than "it can't be done"

Also weird because pipelining is very literally "insert memories to cut some bigger process into parts that can occur in parallel with one another"

This is almost the opposite of what you're referring to I think, but interestingly Itanium had instruction level parallelism hard-coded by the compiler as opposed to determined by the processor at runtime.

"memories" could also refer to registers, or even a stack-based approach

Some folks use the term "state element" to describe the set of things which hold state between clock cycles. Memories (e.g., SRAM) and registers are both examples of state elements.

I took the "You don't think about memories when you design hardware" paragraph from Yosef to mean that SRAM is so highly tuned that in most cases your best bet is to just use it "off the shelf" rather than invent some novel type of state element. And if you use SRAM, then you are stuck sucking through a small straw (e.g., an SRAM can hold 1K words, but you can only read 1 word/cycle).

Reply on news.ycombinator.com