I think your fundamental assumption is based on "possible" performance. I would argue that required performance has flattened over the last few years. A nexus 4 is a perfectly serviceable phone still.
I think as technology improves, the advantages of modularity with trump any loss of performance.