There will always be trade off no matter where we decide to put them. Either that or we return to living in villages and forget about modern civilization.
Not really sure how you went from not destroying the desert for PV to living in villages and forgetting modern civilization, seems like a big jump. There are other alternatives which I mentioned that we can utilize before we need to make the trade off that you're talking about.