So if Common Joe's views on gender can be summarized as "Women should stay in the kitchen", is it elitist to not take that idea seriously? It has nothing to do with people's relative social or intellectual standing, and no one should have to bring themselves down to anyone else's level. Bad ideas are bad ideas, and should be labelled as such.
It's hard to tell if someone making a controversial statement is doing so to learn and participate in discussion, or if they're just trolling. And with ol' Common Joe the latter can be just as likely as the former. So sometimes people err on the side of caution, and prefer to avoid outright controversial topics that are difficult to discuss and articulate. I don't think it's an issue of anyone looking down on anyone else.