What good is wealth if you are living in a boring old town with nothing to do? Also this would obviously not be true if you have a family because kids are a money sink.
For example I have many friends in large metropolitan areas whose lives revolve around eating and going out. On the other hand I much prefer going on hikes to going out to dinner and bars. Also if I want a more cultured experience I have enough money to hop on a plane and go anywhere I want. Overall the smaller town with the lower cost of living works very well for me.
Far away from me are the trivialities and concerns of concrete monoliths. I much prefer being able to walk outside on my own, and see open fields and forests where ever I turn.
Maybe I'm not an exception, but there are more than a few people who don't need much in life. A wood cabin in the mountains with 802.11 and solar power is all I need.
As an example, a friend lives in a 5 bed, 4 bath house and hour and a half out of the city. He has a woodshop, a workshop, fire pit, fireplace, man cave, 4 car garage, loves dirt biking, off-roading, cycling, fishing, flying his racing drone,and can play his drums as loud as he wants. He is also only an hour and a half from the city if he wants to go in for the day or weekend. He would be in a 2 bed, 1.5 bath condo if he lived in the city and wouldn't be able to do any of his hobbies; maybe the cycling.
Not really seeing a downside to this place. Different strokes for different folks.