Also managing the lifecycle and production of these panels is another issue. And don't forget that you still need to grow crops somewhere.
Currently only less than 2 % of the energy we use comes from solar energy and 87 % comes from fossil fuels. The cost of replacing the infrastructure is enormous and there is not much incentive to do it as long as we have plenty of cheap oil (which is going to stay with us for at least another century according to estimates).
Why is it so unimaginable that we will still be using oil in 100 years? It might not be 87 % as it is today, it might be 30 % but I am very sceptical that we will be able to completely replace the infrastructure we have build in the last 200 years.