I see what you mean, but I feel that this misses out an important fact - a lot of those speed increases are only available if you modify your code to use an optimized path. Memory I/O is a classic example - it has sped up a lot but only if you make sure that you aren't generating cache misses on every memory access. Branch-prediction-friendly code is another good example that can make a huge difference to the performance of a system.
That means that a lot of the performance gains that we have achieved (not all, certainly, but a substantial chunk) are not necessarily made available to naively-written code. It is not terribly surprising then that we sometimes struggle to achieve the speed-ups the hardware has made available to us.
This problem is only compounded by the fact that performance is one of the leakiest things going when we are talking about abstraction layers. It makes things extremely difficult to think about because you need to know intimate detail of your entire software stack, which these days is huge, if you wish to effectively optimize your code.