It ends up messing with the local housing market, as the Californians come in with a ton of money for housing (as housing is so expensive where they're from) and all the prices are super low in the new location. Since they have enough money to pay for it, prices go sky-high. Not good for locals (unless you want to sell your house).
You can buy 2 equal sized, equivalent location, equal desirability houses in North Carolina (or insert most other states) and California and end up paying double or triple in California.