The first touch devices might've been around 3.5-4" (just guessing here) but they've since then been constantly growing to where 6" is nowadays considered normal. How far can you going up in size? It's definitely not comparable to finding the size of a seat, as a phone's "right size" depends on how you intend to use it. There are hundreds of phone models on the market but they all seem to revolve around price points (and hence features/specs). Why not expand the range size-wise as well while at it?
I have fairly large hands and I don't mind manhandling a 6" "phone" but I simply do not want to carry one in my pocket. I can't imagine I'm alone on this. Largest phone I've ever had was 5.5" and that one constantly fell out of my shorts pocket when sitting down and now with the 4" iPhone SE I've grown rather accustomed to the form factor and am willing to trade off usability (more difficult, precise typing required) for it.
And then you have people with small hands.
Edit: Easy one-handed usage is also important to me.