Downtown has never been where the bulk of the ppl in SF live or hang out. It is for the “bridge and tunnel crowd” —- mostly offices and businesses; and not meant to be habitable.
Not true; for a long time, the rich people lived around second street/rincon hill (neighborhood known as "East Cut" today referring to the Second Street Cut, which levelled the area to improve transportation).
This is quite sad and indicative of how poorly planned the city is due to car centric transportation and car culture.