In my limited experience, both asocial and antisocial would match. People who don’t care about other people can also find it easier to harm them.
Asocial people don’t have time to educate every single person with a grave misunderstanding about them that they encounter so they choose to keep their circles small and valuable.
Furthermore it is all on a spectrum as Aristotle told us ages ago but with an average life span under 90 (while deteriorating) we only merely approach true wisdom. Rarely obtaining it.
Take something like autism, there is NOTHING that stops autistic people from caring about others and being extraverted, but repeated social rejection may eventually make them withdrawn. Or maybe social interaction is just overstimulating and unpleasant. Such a person may do empathetic things like volunteer and donate to charity.
asocial people also suffer from their inept self-promotion abilities often making their virtues go unnoticed and uncelebrated. So if you're relying on personal experience, you're running into inductive reasoning problems right away. Whereas people with antisocial personality disorder might happily tell you all about the great things they do for everybody, even if their claims are total lies.