Today most software can only scale if you add a faster processor, that has to stop, before that happens, there is not much the processor manufacturer can do, their hands are tied IMO
The programming interface of a modern x86 CPU is best thought of as a virtual machine with a JIT. The JIT performs some optimizations to make old code run more in parallel without the need for recompilation. This is a terrible model for compilers and programmers, since it makes optimizations hit or miss, and of course it's why we're in this whole spectre/meltdown mess at the moment. If we switched to a more reasonable programming model which actually met the needs of software (fine grained interprocessor communication without going through central memory, many more registers, actual software access to pipelines, caches, etc.) then it would make a lot of old software run slower (through emulation), but new software could finally make better use of your hardware...
Seeing how Itanium went, their risk aversion seems pretty wise.
Itanium seems to satisfy most of your desires with its VLIW/EPIC architecture, which exposes much more to the compiler.
> Today most software can only scale if you add a faster processor, that has to stop, before that happens, there is not much the processor manufacturer can do, their hands are tied IMO
These are not recent developments, chip makers could have opted not to use them and software developers would have done otherwise.
The hardware is developed to make the existing software faster, but the software is also developed to run well on the current hardware.