Surely its more about the fact that lenses have to be a certain size.
You need chunks of glass at the moment regardless of sensor size. I carry a basic DSLR around with me (D3100) and a decent but cheap lens (AF-S DX 18-55) and a shit smartphone. They cost less than a good smartphone and the results are amazing.