deRerum

Yes the examples you have all have multiple processing elements but they are vector processors. I was talking about simple and cheap scalar processors.

GPUs rely on symmetry to simplify the hardware design. Multiple (like 64) processing elements share the same instruction decoder. They have to access adjacent registers. So they become vector processors.

CPUs devote a huge chip area to caches and instruction pipelines. GPUs took out much of that area and complexity and replaced it with raw floating point computing power. For certain applications this has proven to be a good trade off.

What I described was a similar trade off...replacing a few heavily pipelined processors with massive amounts of cache memory with smaller cheaper cores. I wonder if it might prove to be the optimal micro-architecture for certain applications and given certain languages and certain data patterns.

That is an interesting idea. I wonder if a small processor core could be stamped out on an fpga and hundreds of small processors can run simultaneously.

Then I wish my c++ objects each ran in a separate (virtual) processor instance. They could have event signalling built into the language as well as the hardware.

It might force people to partition their design for more fine grained parallelism. C++ objects use function calls for interfacing with objects. Using object methods for event handling is just a ridiculous hack. Each object should have its own memory space.

Anyone can prefetch data into the cache, but letting programmers control cache eviction would open the door to all kinds of user errors which would kill performance.

A 10Mhz processor has a clock cycle of 100ns (0.1 millionths of a second). Those are just rough representative numbers I picked...any particular RAM delays would be different and the actual latency would be complicated by bus speeds and protocols etc.

In the past (like around the time most programming languages were invented) memory speeds were faster than processor speeds. So all variable accesses were instantaneous. Languages like C did not have to worry about memory hierarchies.

If memory speed is 100ns then you would notice the memory bottleneck around the time when your processor speed is 10Mhz. This point was reached in the mid 1980s with the 286 processor. Yet through the addition of cache memory this bottleneck was hidden from most software. They continued to operate in a bubble as if they were still running on the hardware of the 1980s.

It’s a bit like life itself...we land mammals carry around bags of water under our skin and our cells are still batched in fluids as if we are still living in the environment of the oceans hundreds of millions of years ago.

Many programming languages have been invented since the 90s but as far as I know none of them explicitly model memory latency and make reference to memory hierarchies. It’s as if they still need to maintain the illusion that they are running on the hardware of the past.

(Note: I once read about a language called Sequioa developed at Stanford that explicitly modelled the memory hierarchy. I don’t know what happened to it).

GPUs are latency hiding engines...they address the mismatch between processor clock speeds and memory latency by a unique scheme. Since they can’t improve memory latency, instead they parallelize memory and vastly increase the bandwidth available.

Once they have increased memory system bandwidth to be able to feed the multiprocessor throughput, the rest of the architecture is designed to make the most efficient use of it.

They spawn thousands of threads and schedule them in and out really quickly... so the processor utilization is always as close to 100% as possible. When a thread is waiting for memory it is put to sleep in a few clock cycles. When it’s data is available it wakes up and does it’s business. Since the workload is split among thousands of threads therefore there is somebody ready to be scheduled. This is why GPUs only make sense to run on massive workloads.

The same trick can be used to hide SSD latency.

Agreed. Way back when, when I was young and stupid and knew everything, my head was full of logic puzzles and programming tricks. So it was easy to solve them in an interview. But today, there are so many things in my head that I cannot keep them all there at the ready...so I usually dump less important things (like puzzles) out of my brain's fast-retrieval cache, and store them in long term memory.

Also, I work on multiple problems at once so I have to do a slow context-switch, and fill my brain's cache with the working-set of the problem (usually by googling stackoverflow etc.) and then solve it quickly, before dumping the cache.

As I developed more wisdom, I usually don't keep small details in my long-term memory, but only repeated patterns and useful details.

Interview questions are more appropriate for a simpler time when programmers were more single-threaded in their heads. They are also heavily biased towards the programming issues of the 1970s and 1980s...doing all of the things which were left out of the standard libraries...recursion, dynamic programming, sorting algorithms, string parsing, set theory, data structures etc. In my humble opinion interviews should focus on questions of style--take this code, refactor it. Refactor it again given new constraints. Multithread it. Make it secure. Refactor it given that you cannot touch this non-optimal code that someone checked in in a hurry and cannot be removed in a short time frame. Etc.