The word we're searching for here is "divisive"
The people I disagree with are divisive.
Ruth Bader Ginsburg was never described as divisive, though she was objectively more divisive than a rich guy who makes cars and rocket ships and flamethrowers. Antonin Scalia was described as divisive, so the argument can't be made that SC justices are somehow different.
It's a rhetorical trick that's been used for many decades. It's obvious, and it's tiresome.
I don't think most people would argue that Musk isn't a visionary. I also don't think most people would argue that Musk isn't divisive.