With all due respect, the "too many outsiders spoil thing" narrative is hokum, flat wrong, an not-useful myth.
Back when rents were considerably more affordable, most of California received a similar level of visitor and had similar growth rates. The changes have been driven by massive money coming into real estate, restrictions on building, and an economy where real wages are stagnant.