That isn't an inevitability. The modem doesn't negatively affect the lenses and sensors, choices by people in companies are what result in worse sensors/lenses in phones.
It shows that smartphones use small, and crappy, sensors behind even smaller, and crappier, lenses. Phone got a long way, and there is a reason they replaced point-and-shoots. They are still a far cry from "real" cameras.