Why would you -have- to? I've never broken any of those things personally, other than the typical battery charge degradation. If the camera+battery+memory are modules, it may be cheaper just to replace/tradein the main body over doing a repair.
Handwavy on the costs, but modular is definitely more expensive - being able to trick out your phone with bigger memory and large-aparture camera may be worth it to a certain set of customers.
Agree it's a wonderful but hard to implement concept.