I'm also incredibly careful with my phones, almost to the point of not really being able to enjoy them for the first few weeks because I'm overly cautious. Nonetheless, my Nexus 5 suffered from the power button issue that seems to be quite common.
You're correct in that the CPU can be a big factor but I don't play mobile games or watch much HD video so it wasn't as much of a concern to me. It should be noted that the Nexus 5 is running a 32-bit SOC whereas the new phones like the Pixel are running ARMv8 64-bit SOCs which have noticeable performance improvements, particularly with regard to encryption. Newer phones will also have more RAM which is actually a noticeable difference for me in day to day use as well as higher speed memory.