Yes, if someone says something that is factually correct (or at least not incorrect) based on a different world-view, then I agree, don't vote them down. If they say something that is factually incorrect but accepted in their worldview, I don't know that that changes what I was saying before.
Edit: you know, it's usually my policy not to comment on how comments have been modded, but I do find it hilarious that in this discussion, someone's down-voted this comment (at the time of this edit) without replying.