In fact there were competing phones with Windows Mobile or Symbian that did have similar hardware specs. (Really if you have to pick the single defining hardware innovation of the iPhone, it was the full slate capacitative touchscreen.) And in some areas the iPhone was clearly behind, shipping with a EDGE-only radio at a time where other networks were rapidly deploying 3G hardware.
But the big missing bit was the app store, which didn't exist yet. When the iPhone shipped, it was a very expensive phone with a web browser, and nothing else. Jobs was going on stage saying that web apps were all anyone needed, and people were laughing at him. When the app store was finally released it only barely beat Android to market.
It wasn't really until the 3G/3GS generation that the product really hit its stride, and a few more revs after that before it became the sine qua non accessory for the tech set.
In hindsight, yeah, that was a huge inflection point. But at the time there were a lot of reasons to prefer other devices.