I have the iPhone as a phone. I make calls with it, keep it in my pocket, and other uses are secondary (albeit common). Insofar as it's uncomfortably small for some uses, that's why an iPad is usually in arm's reach. Sure, 5" is more usable for many applications than 4", but ~10" is much more so. Make the phone more than 4" and I'm apt to just not carry it, as it won't reasonably fit in my pocket and looks/feels like a wall held to my ear.

iPad + iPhone Nano = winning combination. If Apple makes a 6" iPhone, I'd sure hope they at least retain the 4" version, if not shrink & optimize for a much smaller version. I need a phone that excels at being a phone and integrates well into the iOS ecosystem; got my tablet for applications needing a more complex/larger interface.