The people who have the talent necessary to change the world, and the people who can afford to buy real estate in SF are two sets with a small intersection. Both are small sets on their own, and the correlation of membership is probably pretty close to zero (being talented makes you more of a candidate for material success in some ways, and less of one in others, so the net effect is probably zero) so you'd imagine a tiny intersection.

It's not that "nobody goes" because it's too crowded. It's that the people you want don't go because it's too crowded with people you don't.