Manhattan is precisely the right thing to compare to San Francisco. San Francisco's borders are drawn in a way that encompasses what in other cities would just be the dense inner neighborhoods. E.g. Oakland is a separate city while Queens is part of New York City proper, but they're functionally and geographically equivalent.

If you go neighborhood by neighborhood, you'll see that San Francisco isn't really very dense. No neighborhood besides Chinatown and the Tenderloin are more than 50,000 per square mile, and popular areas like the Mission are only 25,000 per square mile. In New York, the UES and UWS are over 100,000 per square mile. Greenwich Village/SoHo (same community area) is 60,000 per square mile. Chelsea is about 45,000 per square mile.