The "first" kind of comments are easy to detect, so I think they wouldn't be a big problem
The chances that a post written later in the game will become one of the most voted because it's really a good comment are close to none anyway
Another interesting feature would be highlight and prioritize content written by users you follow, based on some simple rule like how many votes you gave them before or how many interactions you already had in the past
But downvotes are a prize and a status, you have to reach a certain amount of karma points to be able to downvote something so I guess unfortunately they are here to stay