We are deep into the Post-Moore's Law era, where a new generation gives, now, 120% rather than 200% of the previous generation's performance. Next generation we might get 115% of this, or 110%.
Another consequence is that the improvements we get have become dodgier and less reliable, so that tiny, irrelevant-looking changes in the code may mean a 2x speedup, but more commonly a 2x slowdown. (This is in no way an exaggeration.) The bargain we get from pervasive penetration of more kinds of caches is that sometimes our programs are faster, but we no longer know how fast they should be, or whether another factor of two or ten has been left on the table.
Sorting algorithms are quite mature now, so that a 20% improvement in a relevant case is important, yet a factor of 2 or more may come from the compiler choosing one instruction over another.
Compiler regression bug reports now routinely complain of a 2x performance loss from that instruction choice, but fixing them would result in 2x losses in some other set of programs, instead. We get new unportable compiler intrinsics to patch the failure, that often don't, for obscure reasons.