I think the jokes with negative votes should increasingly loose points with time to avoid it losing a chance right after submission. I've been working on a similar upvote/downvote system and think that way will work more efficiently.