You're not supposed to discuss voting...but I've noticed a couple times that sometimes when I make two related comments on the same thread expressing basically the same opinion, one gets upvoted a lot and one gets downvoted a lot, and that's just weird. Maybe the exact timing determines if the upvoters see it.
Yeah, this is just one reason why I wouldn't be super concerned with your hacker news points.