We discovered/noticed this when the exterior walls were 2 meters high. They had to move around interior walls but that didn't help much.

My wife and I concluded that we got what we paid for, and you're right that in hindsight we should have taken legal action against the contractor. I don't know how breaking down the whole exterior of a house to fix it down to the foundation would feel though. At some point we thought of selling the partially completed structure.