Actually, the recent articles on HN's ranking algorithm suggest that PG considers comments to be evidence against an article being a quality article. Presumably that's why he penalizes articles with lots of comments.
I think a better way of doing it would be to get a large number of articles which are known to be controversial or uncontroversial and then use features from them (not just the number of comments) and try to develop a good predictor of how controversial something is. It would probably be a lot more effective than just arbitrary penalties for articles with 40 comments.
Though I kind of object to this stuff being done automatically at all. There aren't that many articles on HN that human moderation is impossible. As cool as automation is.
A better indicator of how trollish a thread is and how much that's getting in the way of the conversation (because those are two different things - users may be willing to put up with vehement discussion in one instance but not another) would be the ratio of downvoted comments to comments. I can even see measuring the length of comments and the depth of a thread, as a deep thread with lots of tiny comments might be an indication of sniping.
I realize i'm probably being incredibly naive and that the whole thing is complex with multiple factors interrelating but I don't see how the ratio of upvotes to comments tells you anything useful, if it doesn't take into account that people may comment without upvoting, and people might upvote bad comments. I make certain to upvote the article now with every comment because i'm aware that not doing so, regardless of the content of my comment, is an implicit downvote.
I think it's also a mistake to correlate "controversial" with "unwanted" but that might just be a matter of semantics.
Read pg's comments in this thread. https://news.ycombinator.com/item?id=6596038
That's not quite right. Anything with >= 40 comments AND more comments than upvotes gets the controversy penalty. Both conditions need to be true. An article with tons of comments will have NO controversy penalty as long as there are more votes than comments.
For most articles, there are more votes than comments, so commenting will not harm the article's rank. (I wrote the recent article analyzing HN ranking.)
I would believe, that PG had analyzed a lot of articles, to find a good ratio, that does penalize these kind of controversial topics, that he does not want to gain too much traction.
What I do not understand though is the fact, that controversial topics are penalized in the first place. But that is just me, one who loves to argue, to find optimal solutions (or to see, that there are really only two believe-systems at work).
The flamewar detector does its intended job with things like the gender discrimination stories but it looks like it throws out the baby with the bathwater and contributes to the echo chamber effect.
Here's a couple of Samsung stories that met the same fate. http://hnweird.tumblr.com/
It's really hard to tell(except if you're pg or HN staff) whether a story disappeared because of the flamewar detector, or flagging or the domain greylist or something else. I guess this is intentionally so.