I'd also include Los Angeles (Getty and LACMA), Chicago (Art Institute), and Boston (MFA, beating SF by default because it doesn't even really compete in this category).
Chicago's Art Institute is not good. Chicago does however have a couple good museums: Museum of Science and Industry, Field Museum of Natural History,
Boston's MFA is not good. Boston does however have Fort Warren and the USS Constitution.