If you're displacing groundwater based desert agriculture, that's one thing, but it's worth keeping in mind that in general, desert != wasteland. Deserts are fragile ecosystems with many rare and specialized plants and animals.
Climate change may push the temperature of some of those deserts too high, wiping out the specialized plants and animals that live there. Perhaps we could use a solar installation to provide shade in areas that are getting too hot.