That's not to mention issues like autofocus.
There were a few attempts to bridge this gap, which attach a large sensor to your phone. The best-known ones are the Sony QX1 and the Olympus Air. They never took off. I think that was due to very poor execution. I though the Olympus Air was close, but had a few complete showstopper issues which made it unusable. A v2 might have done well, but instead, Olympus gave up.
A few cameras are getting improved workflows with interfacing to cell phones, so perhaps we'll get there another way.