I owned the older 2013 Nexus 5 for two years until upgrading to a Nexus 6p, which I am still happily using. The N5 was a great phone, and an exceptional value at the time but the hardware is quite old at this point and I believe Google has stopped updating it to newer versions of Android. The Nexus 5 build quality wasn't that great and had a common problem where the power button would break and stop responding. This happened to my unit after approximately 14-15 months of use.
I've never owned the 5X but I did consider it when it was first released. I ultimately opted for the 6P even though it was more expensive and larger. The 5X has fairly large bezels for it's screen size and the size difference between the 5X and 6P wasn't as large in person as I assumed. The 6P also had a better screen in my view, and far better build quality.
The Pixel is an interesting proposition for me since it is even more expensive than my current 6P. I couldn't justify the upgrade cost since my current phone is still more than adequate, and I found the Pixel's design to be lacking. I've heard the battery life is very good, especially on the Pixel XL.
In short, the biggest reasons for buying an expensive Android smartphone for me are build quality and update support. I'd also add that the more expensive phones tend to place a higher priority on the camera. I never really felt this was a big deal or killer feature, but since getting the 6P, which has a very good camera, I've found myself using it more and more. It's gotten to the point were going forward, camera quality will definitely be an important consideration in a future phone. I'll add that the Pixel's camera is said to be very impressive.
You may also want to consider the more recently released OnePlus 3T[1]. I've heard very good things about this phone and many consider it the spiritual successor to the Nexus line since it offers very good value.
You're correct in that the CPU can be a big factor but I don't play mobile games or watch much HD video so it wasn't as much of a concern to me. It should be noted that the Nexus 5 is running a 32-bit SOC whereas the new phones like the Pixel are running ARMv8 64-bit SOCs which have noticeable performance improvements, particularly with regard to encryption. Newer phones will also have more RAM which is actually a noticeable difference for me in day to day use as well as higher speed memory.