I think I got it around 500~. There could be a manual component to it?

Your user page shows you how much karma you need to hit the next level. In your case, you need 172 to downvote comments. https://news.ycombinator.com/user?id=exikyut

It's 500 last time I heard, but you don't get a downvote button on every comment.

And there is no downvote button on posts (or if it is, I haven't reached that threshold yet.)

Either they tweak the threshold over time, or there are other factors than karma. I'm sure I gained a downvote button well before 1000. I don't think it's particularly useful -- I only use it when I think a comment isn't worth reading at all. I usually upvote people when I think they're wrong, so we can have an interesting discussion.