I recently listened to an interview with someone working on Intel's C++, and he indicated that, contrary to popular assumption, most of the speed boost you can get with it isn't from better instruction-level micro-optimization, it's from analyzing information flows and maximizing locality of reference.
Perhaps this is an inevitable result of that ever growing processor-memory performance gap. Nowadays, the cost of having to take a walk of shame out to the RAM chips is so great compared to everything else that I would assume that minimizing the number of times you have to do that is by far the lowest-hanging fruit.