I think the problems have more to do with economics. Creating new modern CPUs requires a lot of capital investment, making CPU vendors more risk averse. That's why modern CPUs by and large are not built to meet the demands of future software - they're built to meet the demands of Excel 97 (exaggerating slightly for effect).

The programming interface of a modern x86 CPU is best thought of as a virtual machine with a JIT. The JIT performs some optimizations to make old code run more in parallel without the need for recompilation. This is a terrible model for compilers and programmers, since it makes optimizations hit or miss, and of course it's why we're in this whole spectre/meltdown mess at the moment. If we switched to a more reasonable programming model which actually met the needs of software (fine grained interprocessor communication without going through central memory, many more registers, actual software access to pipelines, caches, etc.) then it would make a lot of old software run slower (through emulation), but new software could finally make better use of your hardware...