In my opinion the limiting factor in phone cameras is the size. Of course, for a given size a cheap, crappy sensor will give crappy results. But I wouldn't be surprised that sensors used in flagship phones are actually pretty close to the state of the art.
The issue with the size is that if you increase it for the sensor, you're going to have to increase it for the optics. In an era where manufacturers are convinced we want paper-thin devices, they're going to have to choose the one over the other.