Some people prefer their peaceful cookie cut house and desire nothing much to do with the big cities. More often then not it's zoning, job availability, and affordability.
It's not that common to want wide open fields in front of you. On top of that, it's not that expensive to get a big chunk of land in the middle of nowhere.
What you are describing is a culture that already dislikes being downtown but is only there because the transit sucks. That's not a sign of a healthy downtown.
The very presence of freeways drives people away because they are noisy eyesores and impediments to local travel. In Seattle, I-5 cuts across the street grid and separates downtown from Capital Hill and First Hill, while the Alaskan Way Viaduct cuts downtown off from the waterfront. If both of these were removed (I-5 could take a bypass route around the Seattle city center using what is now I-405), downtown Seattle would be much more livable and would have better traffic on account of being able to reconnect the street grid.
The presence of freeways does not drive away people. If that were true there would be no freeways in any populated location.
If you remove all freeways it would impede people's ability to visit and make the downtown less crowded. Perhaps that's your intention.