battery is one.. Android phones in general, even the Pixels, are already less energy efficient compared to the iPhones.. it will be much harder to work around that restriction with a smaller phone...
I use a Galaxy S10e, which is a "flagship" compact phone from few years back, and it has been a great phone that I still use today, but even on the day I bought it, battery life has been much shorter than what I was used (an IP8) to I suspected I got a lemon...