No, they don't. There is a lot of empty land in the world that isn't useful for people to live in or to grow crops due to lack of rainfall, which also makes those sites more valuable for solar energy.
We'd consider a utility-scale solar array that's a couple square miles to be absolutely huge, but plotted on a map of, say, the United States it would be a barely visible dot in a massive sea of land-that-isn't-devoted-to-solar-power. The Earth is very big, and when you get outside the areas where people congregate there's a lot of wide open spaces.
This says that US electricity consumption was 3.9 trillion killowatt-hours in 2019:
https://www.eia.gov/energyexplained/electricity/use-of-elect...
39 quadrillion watt hours divided by (36524) gives us average power consumption in watts.
Let's say a pretty good solar panel gets 100 watts per square meter, then reduce that to 30 to account for night-time. If we divide average power consumption by 30 watts per meter and divide by 1,000,000 to convert to square kilometers, we get:
Prelude> (((3.9 1000 * 1000 * 1000 * 1000 * 1000) / (36524)) / 30) / (1000 1000) 14840.182648401827
So, about 15,000 square kilometers of solar panels would satisfy current US electrical needs. Let's double that to be conservative, that's 30,000 square kilometers. Sounds like a lot, right? That's a rectangle 100 km by 300 km. It's slightly more than 10% of the land area of Nevada. Definitely big, but attainable. Compared to the land we use for farming, it's barely anything (and it can be done in places that we don't farm). And in practice it would be spread out, not just in one place. And it also assumes we get 100% of our power from solar, which we wouldn't.