Unpopular ideas may be ahead of their time, but they may equally be just stupid or deliberately proposed in bad faith.
That's where moderation comes in. Honestly, I think good moderation is better for weeding out trolls and terrible ideas than downvotes. All sorts of idiocy makes it to the top of Reddit, for example.