People who troll for fun don't adjust their behavior, usually. I mean, they aren't behaving that way on accident. The point of being a troll is to mess with other people. "Oh look how nice this site is, suddenly I'll stop trolling and be nice." If a site becomes prominent enough and doesn't have some kind barrier to entry, it will eventually become a plaything of trolls.
Self-moderation works to a degree, though the only sites I've ever seen maintain a high signal-to-noise ratio are those that either don't appeal to trolls (low profile sites, sites with no game/karma aspect, etc) or prevent you from just signing up for a free account and messing with people (metafilter/pay wall, moderated mailing lists, small communities that reject outsiders by default.)
A site with high-quality discussion and participants typically becomes more visible over time as it is increasingly linked to by others, raising its visibility. Instead of people being attracted to a site in order to participate in high quality discussion, now people are joining because it's visible. The signal-to-noise ratio drops, and if the site doesn't eventually implement some kind of protection, it will also be overrun by trolls/spam.
If you watch the "New" section in HN right now, 4/5s of the links that roll through at some points in the day are either dupes, spam, or trolls. Auto-kill flagging helps to a degree, but I think at some point pg will need to relent and disallow new signees from submitting stories or voting.
It's not like this is a new phenomenon in internet behavior, people write about this all the time. HN is interesting because it remained pretty high quality for a while, and the "broken windows" theory seems to be true to a certain degree. But the internet isn't entirely like moving neighborhoods in real life – you can easily participate in as many sites as you want, and just as easily screw with as many as you like, provided they have little or no barrier to entry.
I think if HN was changed so that accounts can't vote at all (up OR down) or submit links until they reach a certain karma threshold from just commenting, it would go a long way in raising the quality. (The reason you take away upvoting is that the troll/spam/etc accounts will just start slinging useless votes everywhere. Comments accrue points for seemingly no reason, votes become devalued, and illegitimate accounts start gaining karma. Additionally, you need some smart 'voter-ring' detection algorithms to prevent people from making multiple accounts to vote each other up.)
I've helped run a couple of community-driven meta-moderation sites before, and watched more than my share of ones run by others become mired and sink over time. Just my two cents.