There are plenty of compact and competent phones. A Samsung A40 for example is barely larger than this new iPhone SE, but has a 5.9 inch display. A friend wanted a smallish smartphone and she's quiet happy with the form factor.
That is not a compact phone.
I know I'm in the minority here, but the iPhone4, in my view, was the ideal form factor (3.5", I think). At least half of the time I glance at my phone, the informational payload is not even a full sentence and requires no interaction. Otherwise, I check mail and sometimes use maps. More or less, my phone occupies the niche that I'm supposed to want a digital watch for, but don't.
If I need more screen, there are a plethora of options. I really don't want a tablet in my pocket. And I don't keep it with me like anywhere nearly like I used to - it has turned in to more of a 'desk phone' that I check when I think about it.
4\S was perfection as far as ergonomics go. Timeless design. The natural 2:1 ratio was ruined in 5 by stretching in the vertical dimension. What they should have done was to expand the screen and have it eat the top bezel. They stopped pushing. Supersizing was the easy road.
3:2 ratio?
A major pro of having a smaller phone is being able to reach anywhere on the _screen_ with just a thumb. While A40 and SE may have the same physical form factor, the former does not achieve this.
Usually when using a phone one handed I'm not even touching the other side.