Here's a reason. I, like most humans, live near the coast. When my home is submerged I'm going to come and live in your neighborhood. And when I run out of food, I'm going to go to your house and steal yours.
Even at a relatively low population density equivalent to that of Los Angeles, you could fit the entire population of the world 3.4 times over in the contiguous United States.
Agriculture is a potential problem, but technological advances are improving the situation. If everyone switched to vastly more efficient plant-based diets, it wouldn't be a problem at all.