Totally agree with the first part.
The second part describes a thing I regularily do as "pretty creepy". To clarify: before deciding to flag, downvote or vouch for someone I might have a look at their comment history to see if I am misunderstanding something.
Also if someone posts something nice I look through their history to find more good comments.
I don't think I'm the only one who does this and in fact I'd recommend it instead of blindly thinking "troll" and hit downvote and flag whenever one sees something one doesn't agree with.