First: no, that reasoning doesn't generalize nearly well enough. Unicode has many quotation marks, and " is just one of them. “ and ” are perfectly acceptable to use to wrap double-quoted text and you can't say they "should" use ordinary " and a proper font, because a font doesn't know which one is the beginning and which is the end of the quoted text. (Hence why those characters even exist).
Second: no, Unicode definitely considers other characters to be apostrophes:
>>> unicodedata.name(chr(700)) # https://en.wikipedia.org/wiki/Modifier_letter_apostrophe
'MODIFIER LETTER APOSTROPHE'
>>> unicodedata.name(chr(1370)) # https://en.wikipedia.org/wiki/Armenian_(Unicode_block)
'ARMENIAN APOSTROPHE'
>>> unicodedata.name(chr(2036)) # https://en.wikipedia.org/wiki/N%27Ko_script
'NKO HIGH TONE APOSTROPHE'
>>> unicodedata.name(chr(2037))
'NKO LOW TONE APOSTROPHE'
>>> unicodedata.name(chr(65287)) # https://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms
'FULLWIDTH APOSTROPHE'
>>> unicodedata.name(chr(917543)) # https://en.wikipedia.org/wiki/Tags_(Unicode_block)
'TAG APOSTROPHE'
There's even a "double apostrophe" which is definitely not any kind of quotation mark, semantically (
https://en.wikipedia.org/wiki/Modifier_letter_double_apostro...) .
For that matter, there's a "Greek question mark" (https://en.wikipedia.org/wiki/Question_mark#Greek_question_m...) which is semantically equivalent to ? for Greek text, but rendered identically to ; in most fonts. And there are some CJK ideographs (https://en.wikipedia.org/wiki/Ghost_characters) which represent completely fake characters not used in any writing (including in any other East Asian language), which were included in an old Japanese character standard by mistake and then copied into Unicode.
"Unicode only has one" is the start of a lot of false claims. It is very much "designed by committee".