I believe the hardest sci-fi is called "non-fiction". There is a "scale" for this, anyway:
https://tvtropes.org/pmwiki/pmwiki.php/Main/MohsScaleOfScien...
I'd say something like "For all mankind" also lands in the Sci-Fi genre, and there REALLY has been nothing physically impossible (yet), though the nuclear engine is borderline
It's hard to imagine "The Martian" or "For all mankind" to NOT be in the category of "science fiction"; they're certainly not fantasies. Maybe "For all mankind" belongs in "historical fiction" but it's edging towards being too recent for that. I don't think they're "just fiction" because grouping them in with, say "100 years of solitude" or "ulysses", or even "the client" doesn't seem quite right either.