Hacker News Reader

Hacker News

Comment by phkahler | Hacker News Reader

Full thread

phkahler

Everything over the last years tells me we are very close to the end game in chip production.

1. power stopped scaling

2. clocks stopped - 4x over 20 years?

3. node names became fiction.

4. EUV now differentiates the top from everyone else.

5. IPC improvements are a trickle.

6. Tons of non-moores law ideas are here:

A) GPUs

B) chiplets

C) stacked die

D) specialized accelerators

7. Governments are now involved in dividing up the tech. Since its finally mature / done.

It has been incredible to watch the progression over the last 50 years.

View on HN

I used to agree with you until I saw this presentation by Jim Keller, one of the biggest names in chip architecture. It really illustrates how many more options we have to improve performance.

https://www.youtube.com/live/oIG9ztQw2Gc?si=vCBGGm0tkhG-VOyd

I like that one. Building a chip involves many different manufacturing steps, each of which has room for improvement. I liked how he even mentions that advances in sandpaper allow us to make better chips.

Depends what you consider as part of chip production. If you consider new instructions and the related software as new chip production, then the performance improvements have continued quite dramatically. Single chip inference has gone 1000x over the last 10 years according to Bill Dally from Nvidia [1] at 10 minutes (probably not original source). Main gains according to the video have been different numeric types (fp16 for example) and more specialised hardware together with more complex instructions. Note that process improvements have only caused a 2.5x from that 1000x. He also mentions that Google‘s TPU has no fundamental benefit over dedicated instructions in GPUs. Quite an interesting video IMO.

[1]: https://youtu.be/kLiwvnr4L80

So going from 65nm to 3nm only gave us 2.5x speed? If by speed we only understand frequency, then it might be true. But speed depends also on IPC and having more transistors means higher IPC.

Ah yes thanks! Makes sense. So, more transistors allow for the more complex instructions, which allow for more effective instructions per cycle (such as SIMD).

Dennard scaling stopped years ago. There's definitely room at the bottom for different topologies — probably 5–10 generations worth. But! These are probably 5 year steps, not two year steps. Also, there's just as much room for the improvement of HW design as there is in peeling off SW abstraction layers above the HW.

I'm guessing we're not done, we're just reaching the endgame with silicon dies. Next steps will be new tech sometimes with a long lead time until it has been developed to the extent that it's competitive with silicon. Things like integrated photonics (or a completely photonic CPU), quantum computers, or diamond instead of silicon (or other materials instead of silicon).

Are we not still getting exponential growth in capability vs time?

It's impossible for growth to be exponential indefinitely. Eventually it'll end up as a sigmoid curve.

Secondly, progress isn't smooth. It comes in phases and leaps, and it's possible for it to stagnate for a long time.

Certainly. We're near the top of the computational capacity for silicon chips. However, the computational capacity of a cubic centimeter of our universe is mind bogglingly enormous, what we can do now is a drop of water in the ocean (or much much smaller). There are lots of other ways to compute and many of them in active development.

Folks have been saying exponential growth is over for many decades now. It may have slowed in some ways or growth might be going in different directions, but it's still growth and I see no signs that the end is near or we're close to a major longterm slowdown of growth.

I think we'll start with a cubic cm of silicon before space.

However if you like the topic and want to read some fiction involving it, try the Jean Le Flambeur trilogy by Hannu Rajanieme (sp?)

I think we are only limited by the amount of matter and energy in the Universe. We are far from reaching any limit.

Dead wrong. You are limited by the speed of causality. Even if the universe is infinite your growth rate would cease to be exponential since the speed of causality is linear and the reachable sphere is growing only by the third power.

5% growth per year and two thousand years later you coloinize thousands of observable universes. The observable universe has a length of 90 billion lightyears.

And no, time dilation won't save you unless you think of dumb ideas like cloning yourself so that the humanity that spreads is considered "you". Otherwise you will only get to go to a single planet.

I think there was a lot of research to use other kind of materials beside silicon for making chips. One of them was carbon. I even heard about optical chips, using light instead of electricity.

We have a long way to go before we get to the end of the road in where we can go in chip design. The real sore spot moving forward is the grind it's going to be to improve price/performance ratios.

Increasingly I'm adjusting by being willing to spend more than I would previously consider reasonable on big ticket tech purchases, rather than waiting for prices to drop.

I think us software types may be more sad about the end of the "free lunch" than the buying public. The 90s in particular had amazing performance increases, but you also needed to buy a new computer (which were more expensive then) every ~3 years, or upgrade roughly every 18 months if you wanted to stay near the top.

Anyway, my optimization skills will stay in demand :)

I am just an average dev working in python and occasionally c++, but I'd like to improve my skills and learn more about optimization. Would you be able to point me to a structured resource (book, course) where I can learn more about optimization?

I’m not into hardware at all. But my friend is. He works on niche application chip firmware. What he tells me boggles my mind. And I think there’s a ton that can be improved on that side for sure. The whole tool stack is so arcane. Most people working in this field are older and conservative. No one wants to change anything.

Yeah, well. I'm a software guy, not a hardware person. But think about it if the hardware guys were like "move fast and break things". It's amazing how much space the hardware people (Asics) have given us the to experiment and explore on their incredibly robust platforms.

Maybe some of them needed to be mature and conservative.

I’m not advocating for another extreme. But the things he told me could be certainly improved a lot without breaking things. In fact things are breaking already. I’m taking even basic things like tools. He tells me many devs have no intellisense of any sort which leads to slow dev pace and bugs. Poor CI setups. Papering over old languages with codegen but not doing it properly thru AST. But literally just string concat in Python. Leads to lots of hard to track bugs. Lots of examples like this.

Every time someone says we've hit peak microprocessing, we in fact have not hit peak microprocessing.

That can be true while approaching an asymptote. Unless there is a switch in technology (optical, different materials, whatever), I don't expect much for the time being. At least in CPUs and especially in single core performance. GPUs and other massively parallel devices are easier to improve with 3D techniques and such.

Point 3 always makes me unreasonably (or perhaps reasonably) angry.

We arent.

There are materials which could make cpus eg 1000x faster, but their life span would be e.g year or two and cost would be 10000x

>Tons of non-moores law ideas are here:

What?

>> >Tons of non-moores law ideas are here:

What?

Those were all things that don't affect transistor scaling. Well some depend on having high density chips, but they don't enable denser chips. Chiplets and stacked die are the things being done because transistor scaling is near its limit.

15 years ago people were looking at those things but they didn't make sense because we could just move to the next node. So they were deemed future possibilities for when scaling wouldn't be cheap and easy. They are no longer "future" tech, they are present day. Meaning scaling is about over, and we are pulling out all the stops on other ways to go fast.

Reply on news.ycombinator.com