Of course, you're right, and I am not in any way annoyed that romance or criminal novels don't cover the same topics as sci-fi. For instance, romance novels are good at things you described, + developing empathy in general.
My only peeve is with people dismissing sci-fi, arguably the most intellectually-stimulating type of fiction (as opposed to e.g. empathy-developing romance novels, etc.), while telling me "read books, they'll make you smarter". I admit I might be biased by the environment I grew up in, but I see this attitude everywhere among non-tech acquaintances, to the point that the expression "it's science fiction" is being used interchangeably with "absurd"/"nonsense". I feel like science fiction is somehow singled out in society as a special kind of crap.