The reason these neighborhoods are exclusive to the wealthy is that housing there is scarce despite growing demand due to job growth. Housing in places like the Bay Area is such a big portion of monthly expenses that stuff like Arby's 4 for $10 or whatever isn't whats hurting people the most.
Lots of these issues just come down to housing and restrictive zoning.