Don't conflate Manhattan and central Paris (the low number Arrondissments) with all of SF.
You can find decent housing in NYC, with pricing comparable to other smaller metro areas in the U.S. like Seattle or D.C. without too much fuss. Same goes for Paris. That's why most people in those cities don't live in the center. Brooklyn or Queens, for example, has a larger population than Manhattan. But only Staten Island has a lower population Density than SF.
The problem with SF is that it has ~Manhattan prices without the number of available units. This means that laughably high housing prices get pushed further and further away from SF proper and out into the Bay Area.
Arguably, if you think of the Bay Area as one city, this makes sense, it's the same in any major city.
It's not so much that someplace like Cupertino or Santa Clara are expensive by themselves, but that you're paying much more for similar housing than you would a similar distance from other similar sized city centers. Yet people move there and saddle themselves with ridiculous mortgages because the alternative is to live in someplace like Stockton.
Dramatically increasing the density of SF proper could make the outer areas and even the East Bay area much more reasonable places to live. However, it would mean lots of people who bought high in the current market would lose their shirts if such a thing were to happen.
I remember the very first time I visited SF, my friend was regaling us with tales of the insane housing prices. No slouch and from a pretty expensive area myself I figured it was just an issue of demand like you'd see in any other big city. But when I actually got to the city I was floored with how low the density obviously was. The solution to the high housing prices seemed idiotically simple then as it does now. Build SF up up up! Areas around every major BART station should also be higher density areas. Relax the density further out from these centers and you'd end up with a much more livable city plan like pretty much every single other city in the country.
By way of comparison, the D.C. metro area (not exactly a well planned urban paradise) has about the same population density as SF Bay Area yet housing can be had for probably around 2/3s the price at comparable distances from the city center.