Same spot here - I've actually been on HN a couple years longer than my profile here indicates, but it wasn't until several years in that I actually felt the need to create an account to comment.
I only comment when I feel like I can really contribute. I don't know what the threshold is for down voting, but I must still not be at it.
I don't know how much it matters to me though - it definitely makes me feel like more of an outsider to the community, but I don't know that it's a permission I really need.