Android will be doable, and has been done, but it means google still owns your ass.
At present, a lot of hardware documentation is proprietary and only available through large contracts and NDA's, so a truly open source operating system is not possible. Read up on the PowerVR graphics and the Qualcomm chipsets for more info. Some will undoubtedly argue that android is "open source" but without the hardware docs, all you really have is unmaintainable rotting mystery meat.
One of the larger problems is the FCC and similar rules for spectrum use various countries. Many cell chipsets can do a lot more than what is legally allowed in particular places, and hence, not doing it "correctly" results in a lot of headaches. Similar legal issues are also true for some "WiFi" (802.*) spectrum usage.
Of course, the carriers will fight it. The last thing that they want is to become a "dumb pipe" and have their profits diminished.
The last major problem is integration. If you haven't disassembled any modern mobile phones, I'd encourage you to do so. The complexity and compactness required by a handset results in very tight spacing and integration of components. Doing this sort of thing al-a-carte would be difficult, and the vast majority of the population would never know how to put it together (i.e. very limited market). On the flip side, I do remember a company that was doing something like this, but they gave you a list of possible features (hardware) and you ordered it from them assembled. Sadly, I can't remember their name.