Compare him to some similar cases. Larry Ellison is a cutthroat executive, huge Republican donor, and is known to make abrasive and belligerent public statements. He was even temporarily the richest man in the world (there was a week in the early 2000s where Microsoft's stock dipped, putting Gates at #2). But he doesn't get the irritation that Musk gets because he stays off Twitter. He doesn't try to stick himself and his brand everywhere. Brian Armstrong banned activism at Coinbase and that also attracted backlash, but he's not seen as a hugely toxic figure. That's because that announcement was in the form of a long dry company policy statement, not by declaring himself to be a free speech absolutist or some sort of protector of not being political in the workplace.
So long story short is, beyond Musk's corporate actions and political stances, he has made himself to be a public irritant. People who don't like his actions and stances are going to be extra-aggrieved by it. I think in some ways there is a certain strategy to it, as we've seen since 2015, to use the negative emotions stirred up in social media to bolster one's aims. Musk has embraced the mantle of a universal celebrity, one who can produce infinite amounts of admiration from his devotees... and infinite amounts of annoyance from those who disagree strongly.
A pretty good video likening that public style to magician's tricks here: https://www.youtube.com/watch?v=wkvvAQxxo_0