I think my biggest ergonomic complaint was the earpiece speaker aperture. If I didn't hold it within about 0.001 degree of the sweet spot, I couldn't hear what the caller was saying. (I could go off on a rant about how there was a very good reason why telephones fed a small sample of the caller's voice back to the earpiece as a sidetone for 100 years, but it would apply to all cell phones, not just the iPhone.)
I was (and am) fine with no physical keyboard, but omitting a GPS chip was just plain goofy, considering how nice the Maps application was. Others were complaining about the first iPhone's lack of cut and paste functionality relative to some other phones, the inability to replace the battery or add external storage, the inability to use third-party applications, and the substandard camera.
Bug-wise, if you could get Safari to avoid crashing back to the desktop for more than 30 seconds at a time, you were way ahead of me.
All of this stuff got better in the next generation. It was Apple's dedication to fixing the phone's shortcomings that really made me a dedicated iPhone user.