Not sure wealth is moving out or rather people who cannot afford to live there are moving. People who are wealthy and afford to stay will stay in the coastal cities where most of the outbound migration is happening.
There are other factors at play besides just affordability.