If you do that you frequently end up with a position that is "extremist" to someone or another. This effect is so reliable that if you meet someone who isn't extremist about anything it's safe to say that they don't think for themselves, rationally or not.
As a corollary if you agree with all of the mainstream positions of your (aspirational or actual) social group, whether you realize it or not, you've probably arrived there via social pressure rather than an open, honest, rational method.
So I don't hold an accusation of extremism as an insult, but more a prerequisite of someone capable of independent thought.
That doesn't mean such a person won't be wrong as much as anyone else, but they're not necessarily more wrong than conventional wisdom, and they tend to be right or wrong in interesting ways. Which is valuable.