As a comparison, my younger daughter inherited my old Nexus 4 (bought couple of months before I got company-issued iPhone 5), the phone introduced more or less in the same time as iPhone 5, and not only the hardware is failing (the top of the screen doesn't respond anymore), Google stopped supporting the phone after a year and some.
I actually just bought myself an iPhone 5s to replace it because the iPhone 5's battery life finally became unusably short after many years, but I didn't want a big phone to replace it. My 5s is also performing very well with iOS10.
My 5s is a second-hand refurb, markedly cheaper than the SE, which I was reluctant to buy after several of my acquaintances complained of the device expiring days after purchase.
My comparably old Android phone won't even allow a software update and is still on Android 4. My partner prefers Android and has a Moto G (and has a tendency to drop phones hence why I suggested something cheap to replace) and it also won't update to the latest Android version.
So while the Pixel might be the shiniest, I would be reluctant to invest in something which will not have the effective lifespan of an iPhone.
I really wish Google would make a great Android phone so I could have better integration with my Chromebook and G-Suite. Android phones can be used to unlock Chromebooks in lieu of password entry, which would be a cool feature. I use G-Suite for my business so it would be better for that too. I've never owned a Mac computer and I've little loyalty to any company. The iPhone 5s was simply the apex in terms of phone-to-pocket-size/toughness/tactility/features (for me personally).
As a counter point, myself and 2 friends all have the SE, and we all love it. iPhone 5 form factor, better battery life and equal performance to the iPhone 6s made it the winning choice for all 3 of us.
2 of us have had it since launch day, no regrets.
Have you considered that it may be user error?