As a Southern California native, I am intimately familiar with the Mojave, Anza Borrego, and Sonora deserts. I will tell you that these deserts are far from "crappy land that nothing can grow on" as you put it.
The North American deserts are areas of huge ecological and geological diversity. They are also some of the most beautiful and awe-inspiring landscapes that exist on this planet.
We do utilize a lot of area in the Western Mojave for solar and wind power. For instance, there is a HUGE field of wind turbines in the San Gorgonio pass near Palm Springs at the intersection of the 10 and the 62.
That said, I am no way in favor of scarring any more of our deserts by covering "a fifth to a tenth" of them in solar panels. However, I am not against concentrated solar facilities several square miles in area as long as they remain contained.
I have thought about how we can "disperse" a solar panel network into a system of smaller panels and batteries and the best idea I can come up with is requiring all commercial buildings to cover their roofs in solar panels and maintain on-site battery banks. Also requiring all homes and apartment buildings to have at least one solar panel per household. These ideas come with their own trade-offs although.