You've just described groups making up about 15-20% of the U.S. population; that's not really "tiny".

If you follow the links, one of the large slices you don't account for is "high-risk heterosexual contact", which makes up about 1/3 of new HIV infections. That's people who are not part of one of the high-risk groups, but have had at least one partner who was (which may or may not have been known to them). So the proportion of people who have some rational reason to worry is relatively high, unless you are absolutely sure that none of your sexual partners have themselves ever belonged to one of the high-risk groups. That's a particularly common route of HIV infection among women, since not all bisexual men self-identify as bisexual or disclose that part of their sexual history to female partners.