It probably bothers some people but it does not bother me. I'm on relatively slow hardware. I use a (second?) gen i5, but without an SSD and without a GPU. My laptop is fine for programming, even gaming (CS:GO, CS:S both work! CS:GO admittedly on 1080p but low graphics).
The thing is that I've never used anything faster than an i7 + HDD (with GPU). So when my old one died, I just started using my ever so slightly slower laptop as main device.
I firmly believe that as long as you have not experienced the difference (which I haven't) it does not bother you. Applications don't start lightning fast, when I open the terminal I need to wait a second before I can start typing. So I start Spotify, Firefox alongside my terminal and after ~10 seconds I'm up and running for the rest of the day.
I'm not that bothered about waiting 10 seconds each day.