ARM Pioneer Sophie Wilson Also Thinks Moore’s Law is Coming to an End

> Why? Who knows.

Possibly for the same reason it's already increasing today: as software becomes more complicated (or complex, depending on your POV), we are shifting toward languages and abstractions better suited to manage that complexity, often at the cost of CPU cycles.

Examples: dynamic dispatch, interpreted code, libraries on top of libraries on top of libraries, boxing of machine-native types, bridging between old unsafe languages and new safe languages, automatic memory management, automatic bounds/overflow checking.

I kinda like the idea of this, and that's also one of the reasons I find architectures like the Adapteva Epiphany (the chip that makes the Parallella [1] interesting) interesting. At the moment they're more curiosities because each core is so weak, but they're at least exploring the space of architectures designed for ultra-high core counts (their latest chip is 1024 cores [2], and what makes them different from GPUs is that these are 1024 fully independent cores, running different instruction streams).

The problem, though, remains, that for most systems, unless each core is tremendously weak, you can run most of the ongoing tasks on a typical current desktop system on a tiny timeslice of a single core.

OSs certainly can paralleise more. E.g. AmigaOS relied far more on using multitasking as a basic OS primitive 30 years ago than most modern OSs, because message passing was exceedingly cheap (no memory protection, context switched involved moving a few registers). Something as simple as cut and paste could involve nearly a dozen processes (device drivers for mouse and keyboard, input handler, console device and higher level console handler, a process specifically managing the console clipboard (other apps might handle it themselves), the clipboard handler, the clipboard device, a filesystem handler for wherever the clipboard was mapped, the underlying disk device - usually ramdisk but could be anything, and more depending on setup). That was to make latency predictable first and foremost, at the cost of throughput, but the overall model could be adapted.

But part of the challenge is to write applications this way without just ending up being bottlenecked on the message exchanges.

Even if you can make that cheap enough, we still have the problem of the vast number of tasks that are simply very hard to parallelise. For some of them splitting them up and spreading them out certainly could make things like pipelining processing help us cut latency when something is done repeatedly, but whatever we do, we will have big challenges ahead.

[1] https://www.parallella.org [2] http://www.adapteva.com/announcements/epiphany-v-a-1024-core...

>Assume applications increase their CPU demands significantly over the coming decades. Why? Who knows. Maybe (very plausibly) extremely advanced (compared to today) AI use & integration.*

I'm willing to bet that "extremely advanced AI use" will not come to applications in the "coming decades". It's another fad (in fact a cyclic fad, it had made the same promises back in the 80s), like all the buzzwords that take over an industry for 3-5 years and then give way to the next.

In 2040, for all the predictions for its transformation or demise, the business and home desktop will look more or less like it does now.

I think the best way to quickly get a sense of the difficulty is "Somehow it's harder than using photons that wiggle back and forth by 193nm to carve out 14nm shapes."

Typing is faster and more accurate than voice recognition. The advantage of voice is hands-free operation.

The laptop has been somewhat eroded by tablets and hybrids like the Microsoft Surface, but until we get rollup screens the form factor of 10"-17" portable screen with keyboard will continue to exist.

>No useful voice recognition/speech synthesis, no gesture tracking etc.

Those are gimmicks anyway. Unless incapacitated for some reason, nobody wants to talk to their programs -- except if it is combined with very advanced/strong AI to just utter some command and have them figure out all the rest (try talking out aloud for an hour for e.g. writing a post -- it gets old very soon). Making gestures in the air even less so, unless you want to switch on the lights (e.g. something you do very rare) or play a gesture tracking game to get fit. Those Minority Report style UIs is something that goods look in movies, but is not a good idea in practice (just like the translucent glass monitors they use in those movies. What's the idea there, who thought that, outside of a HUD, it makes sense to have anybody/anything on the other side of the glass compete with the on-screen data for your attention?).

>a clunky device requiring lots of cables and holes in the casing, showing flat image at 4K at best and average quality audio, typing every single letter of a every command to get something done. No useful voice recognition/speech synthesis, no gesture tracking etc.

You make the pinnacle of modern technology sound bad. Perhaps born quite recently and lacking some context?

Some of the sneer doesn't even make sense. "4K at best" -- the eye has a certain physiology, and for a laptop screen that's 15 or even 17 inches beyond 4K is marginal returns (and after that, no returns at all).

And "average quality audio"? Not sure what you have in mind, but audio in laptops is constrained by the speakers, not by the soundcard or CPU. It's trivial to have a great DAC in a laptop that can do 192Khz/24bit that's better than what a human can hear (or even tolerate: 24bit dynamic range allows for building-crushing audio).

> This only if we want to keep the laptop in its current definition: a clunky device requiring lots of cables and holes in the casing, showing flat image at 4K at best and average quality audio, typing every single letter of a every command to get something done. No useful voice recognition/speech synthesis, no gesture tracking etc.

Laptops can be roughly the size of a screen and a battery, and very sleek. They can do all power and I/O through a single USB-C cable. Laptops can do 3D, and if you find a screen that's better than 4K they can do that too. The only real limit is that it's fundamentally one screen, which is pretty minor. Laptops can have great audio. They can do voice recognition and speech synthesis. They can do gesture tracking.

And doing all those at the same time only requires a couple processors.

> Intel, AMD, ARM, etc could learn from supercomputers from 1970s/1980s era. They had to deal with heat problems and found solutions, look at e.g. Cray 2

You have to be more specific here to support such a claim.

Cray 2 was a cabinet-sized minicomputer. They "solved" the heat problems with airconditioning and fans.

The issues revolving around nano-meter scale heat bottlenecks is fundamentally different. You're smaller than airconditioners and fans.

> The problem on CPUs is that memory is flat and considered equal.

CPUs have a similar memory hierarchy, the difference is that you're a context switch away from losing your preciously cached data. Even in GPUs memory is more often than not a constraint on the speed of computation, it is very easy to starve a GPU by accident.

My personal favorite trick to see if I'm using my GPU effectively is to run a benchmark on it to max it out and register the power consumed with an e-meter. As long as I don't hit that power level in my own application there are still improvements to be had (and I almost never manage to reach the maximum but I'm happy with 60 to 80% or so of that). It's one of the dumbest debugging tricks ever but it is surprisingly effective at figuring out if I've laid out memory access patterns properly.

Latency is more important than just bandwidth, and since RAM is pretty far away from the ALU you'll quickly hit limits there.

The memory wall is not a bandwidth issue, but a problem of (concurrent - many more strands of execution than memory channels) latency and there are as far as I know pretty much no ideas how to fundamentally address that (DRAM alternatives might, but so far were all rather disappointing even compared to much older DRAM).

You can add more channels and more ranks into the system, which gives some small linear increase w.r.t. to concurrency, but brings a a big (channels) or minor (ranks) power increase. This doesn't change non-concurrent latency, so does not help sequential workloads.

What? In my experience SiP tends to be more energy efficient than CMOS interconnect transport.

What are some practical things done with GA's chips? (My own belief is that almost nothing can be done with them but I'm always glad to be surprised, certainly with something like this where the chip itself is a masterpiece of circuit design shunning every convention of the art.)

I've always admired the Sh-Boom. Ahead of its time and hence the subject of a lot of patent litigation when time caught up.

https://www.dnd.utwente.nl/~tim/colorforth/PTSC/IGNITE_Proce...

Faster than clock processing before that was a thing.

https://patents.google.com/patent/US5440749A/en

As a longtime GPU programmer, agreed 100%! There's still room for more parallel-friendly algorithms and higher memory bandwidth before the party is truly over IMO.

But...

One has to embrace parallel languages instead of single-threaded scripting languages or the music is going to stop playing RSN, no?

Chip manufactures aren't going to be able to keep it up though. Look at all the problems getting a 10nm chip working, look at how those issues are going to get much worse at 7 and 3nm scales. Advances now seem more putting all the optimizations back in that they ignored in the quest for the physical bottom.

I wasn't able to find the exact slideshow, but here's something that gets the point across more or less.

Page 37 is what you're looking for.

https://fas.org/irp/agency/dod/jason/exascale.pdf

Joules, but the cost of memory access in terms of speed has an even worse disparity.

GPUs are more complex than what's needed for AI, take a close look at Google TPU for example.

I'm not sure what you mean. EUV at 13nm (Extreme UV, after marketing with the name X-Ray Lithography failed) is much easier to make work at 10nm than the current multi-patterned immersion technology based on 193nm ArF excimer.

http://www.anandtech.com/show/10097/euv-lithography-makes-go...

The problem with EUV is that the source power (laser excitation of plasma) is too low, making exposure times too long for the expense of the equipment.

To be fair Toshiba's sale has nothing to do with their semi business (they lost so much money in their nuclear misadventure that they have to sell the only profitable portion of their business to pay off remaining debts)...and the semiconductor manufacturing has been consolidating for 20 years. I actually think that there is a chance for some newer customers to spring up in China due to the massive investment being underwritten by the Chinese government. The existing companies involved in chip manufacturing will be kept from merging by regulatory fiat (see AMAT-TEL, Lam-KLA getting squashed on the equipment side)...there's no way that TSMC, Intel or Samsung will be allowed to buy any of their competitors, and they can't sell to China because of CIFUS (or the Taiwan/Korea equivalent) interference. This means that if China is serious about getting into wafer manufacturing, they have to start from scratch. Since NAND is likely supply constrained for at least another 5 years, there is a chance they can pull it off.

Oops... misread. It is from 2017.

Moore's Law trumps Amdahl's Law when you decide to simply solve two problems at the same time.

Instead of a 19X speedup on rendering, you get an 18X speedup on both rendering, physics, and AI, and gain some extra responsivity on the mean-time.

And that's true with some workloads, and maybe some systems that use raytracing, but not raytracing itself. The only overhead is combining the final data from each processor, and that's log(n) in the number of processors with a very small constant. A system with a million independent processors can raytrace very nearly a million times faster.