Edit; not saying this is a good idea for a phone, but to some extend it would be for a laptop (not ultrabook). Some rugged ones are quite like that.
Modularity often ends up being a _cost_ to the unsophisticated majority of end users. People don't like it in their software (preferring a "complete solution"), nor in their hardware (e.g. cheap integrated hifi versus expensive minority separates).
- like to change/tinker with their hardware - like to change/tinker with their software - like products which do one thing and do it well
When was the last time your parents/grandparents upgraded something in their desktop computer?
I upgraded RAM, video card, sound card, added more hard drives, retired old ones. Later on I replaced CPU+motherboard (unfortunately you can rarely just upgrade the CPU).
If you're not lazy, this can save you quite a lot of money.
Phones are nothing like PCs, and will not be for a long time. It would take some crazy engineering to make them lego-like, or even plain upgradeable.