You're missing out. I regularly have posts that go to -2, -3, etc., which are then upvoted back into positive territory. Here's a fairly recent one: https://news.ycombinator.com/item?id=11332081 (I think that one got downvoted initially for the explicit religious imagery.)

The little mental boost I get from an occasional 'yo-yo' completely counteracts the 0 and -1 votes I get all the time. But mostly, I don't care about internet points. The point is to learn and share.

Which isn't to say that I don't occasionally delete a post, or wish that I had. Another word for "self-censorship" is "judgment".