You only lose reputation when downvoting answers. Question downvotes are free. Also, question upvotes give 5 points to the poster