What concerns me (and I'm sure many others who don't like the idea of a larger iPhone) is that the iPhone 5s is currently the only flagship phone under 4.5". Sure, you can get smaller budget smartphones, but if you want a small high-end phone, it's Apple or nothing.
If the rumors are true and Apple decides to make the iPhone 6 4.7" and 5.5", what do I get next time I'm ready for an upgrade? I think it's silly for Apple to leave money on the table by only selling the current form factor when there's obviously a lot of demand for a larger iPhone, but it's equally silly to ignore the market for smaller phones like every other smartphone manufacturer is.