I'm pretty dexterous person with medium-sized hands, and weight and form factor of 4S allows me to easily fetch it from any side, flip it, access any button, and manipulate any point on screen while still keeping weight center in palm of my hand, not fingers or, god-forbid, fingertips.
I sort of could work with Samsung's SII, but SIII I couldn't tolerate.
If the rumours of the iPhone 6 having a bigger screen are true, I really hope they don't gimp a smaller version to have less grunt. That's my main gripe with small-screen android phones - they're technically inferior to their physically larger counterparts and marketed as the midrange option.
And yet, anything bigger than 4 inches is too big for me. I want to use 1 hand. I want to toss it on my pocket and forget about the thing. If I need a tablet, I will get one.
But anecdotally, my friends who carry Android phones keep their chargers nearby in their purses/briefcases.