I don't think merely believing or giving credence to the people who know him, combined with observation, means that his vilification comes not at the hands of those who know him. It only followed from it. You may look for fault in those primary sources, but why separately attack the rest of the world for their opinions on the matter?
Also, for what it's worth, I think defending something somebody said by calling it an "unpopular opinion" is a really easy trope to fall into that makes your tone seem dismissive to the opposing viewpoint (the "popular" viewpoint).