Actually they did, it was called 'Openmoko' and it was released before the iPhone (albeit there wasn't much between the two). It wasn't just an open source smart phone OS, but also open hardware as well.
Sadly it never took off: partly because iOS (and later, Android) were busy reshaping the mobile landscape; and partly because corporations started adopting Linux for mobile OSs which dragged many of the hacker crowd away from independent set ups.
As to whether Openmoko would have succeeded if it was conceived a couple of years earlier; or whether it such a project was too ambitious without corporate backing, i don't know. But for a while there was an a community driven smart phone available to buy.