The "factually wrong dilemma" is in no way a red herring. It's actually precisely the point I was responding to in the grandparent post. Down-voting based on opinion is something else entirely. Of course, people will confuse opinion and fact sometimes, and I would encourage people to be skeptical of themselves on delineating the two, but that doesn't invalidate the idea that people can say things that are factually wrong.
Yes, if someone says something that is factually correct (or at least not incorrect) based on a different world-view, then I agree, don't vote them down. If they say something that is factually incorrect but accepted in their worldview, I don't know that that changes what I was saying before.
Edit: you know, it's usually my policy not to comment on how comments have been modded, but I do find it hilarious that in this discussion, someone's down-voted this comment (at the time of this edit) without replying.