HN downvoting replies is a problem

Imagine you make comment, someone replies, and you reply to the reply. Because your position is unpopular, the original comment gets down-voted to -4, which is the minimum. But Now your reply will also get down-voted after your original comment hits the -4 cap. This means you cannot respond to anyone without risking more down-votes. This seems totally unfair. How are you supposed to defend your position and reply without losing dozens of karma? Do you just ignore it? Delete the original comment?