This is similar to my plan. I plan to cool and warm my house using an earth tube or two, which leaves electricity required for very modest use, lighting, charging phones and what not and a refrigerator. Combined with a white or silver metal roof and a good gap between the solar panels to allow wind to flow through in leu of an attic should do me just fine, and on top of that the inside air will always be fresh from outside. The energy saving will be tremendous, and digging a trench for the earth tubes will actually cost less than installing a central air conditioning unit. Additionally you don't really need insulation, since fresh temperate air is always cycling in and is available pretty much endlessly, the limiting factor on efficiency is air flow, not insulation. Also, no condensation in the walls; they'll always be above the dew point in the summer, and in the winter the air will be well below 100% relative humidity since the ambient air is being warmed, not cooled, and so the walls will never be below the outside ambient temperature in the winter.
The only downsides I can see are that I can't set the temperature, it would be whatever it is in the ground, which at the right depth should be around 73F year round, and 100% relative humidity in the summer. That and that the house must be designed for maximum air flow but I don't really see that as a downside, people have designed houses this way forever until the advent of air conditioning, and to be honest I'm not a fan of a house design that doesn't take air flow into consideration.