Intel was quite correct in predicting that we would continue to get faster transistors, and the Pentium 4 was absolutely the right design to take advantage of faster transistors. The story of the past decade of CPU architecture is all about heat becoming a primary design problem for the first time in history.
> Pentium 4 was absolutely the right design to take advantage of faster transistors
No, it was a horrible design in many ways, irrespective of those constraints and any wishful thinking about the power wall, and it gave the advantage to AMD for years. Not a big surprise as it was designed by Intel's B team. The Pentium M which has been the microarchitectural basis for their subsequent desktop processors was designed by the A team.
To be fair, nobody has ever designed a CPU based on current process technologies. Design teams are always saying "we think the process technology we'll end up working with will look something like this...", and based on past history there was no reason to think that the power issue wouldn't get solved like it had the past N times.
Historically (pre-P4) this was a great tradeoff. The Alpha 21064 was a barn-stormer; it ran at nearly twice the Mhz of its brainiac competitors at the time. This more than made up for a deeper pipeline and simpler execution units.
What killed the P4 was that they hit the power wall. Moore's law scaling of gate size no longer resulted in a corresponding scaling of power usage.
It is hard to blame the P4 designers too much for this, since people have been prophesying the end of Moore's law since the mid-80s at least. They happened to be designing when it actually happened on power. Current designers should look to that lesson when relying on increasing density...
Of course, these lessons are pretty useless, because much like a stock market bubble, Moore's law continues on a given metric until it doesn't. And it is always obvious in retrospect.