Five years ago I was using Chrome and it was fast, lightweight, and stable. Now it's slow, uses 500 MB of RAM, and the renderer crashes on a regular basis.
1) Your definition of what is fast changes. Downloading an MP3 in 5 minutes felt really fast in 1998, but really slow today.
2) What we expect chrome to do keeps increasing. Since Chrome was so fast, people started creating webpages that had to do more (because they could!) It is a form of Induced Demand (https://en.wikipedia.org/wiki/Induced_demand)
Consider how much extra performance game developers are able to wring out of a console after a few years' experience. If the hardware is absolutely stable, and there's no expectation that it can get faster, then people do make a big effort on software.
I suspect the target response time is what determines how slow things are, not the speed of the hardware. As people have pointed out, hardware has gotten faster and many user interfaces have gotten slower, so it seems more likely that: the target response times have been rising (people tolerate higher response times in exchange for more features or whatever), or people with older and slower computers observe much higher response times than the original target (iphone 3gs vs iphone 5 for a given app, for instance).
Stable hardware solves the second issue at least, but not sure about the first one. I think over time people will notice that everything else responds instantly to touch, and computers should too, so they will start rejecting apps that have a high target response time.
As an example of that, I chose my last monitor by looking at http://www.displaylag.com/ to get the minimum latency there. If other people start caring more about the latency of the things they use, hardware and software developers will prioritize that.
"Innovations in hardware are to give programmers freedom to build shitty software and get away with it."
If people knew that their own choice for speed improvements was optimization (and to a certain extent, that is true already), then they might focus more on that instead of adding features.