I've taken to calling it the 'overheated discussion detector' because it's not always about flamewars. And yes, we do turn it off when a story and thread are high-quality. We don't necessarily see every case of that but we do review most of them.
When people see something they don't like, they don't think, "Well, any even-handed system will put up things I don't like sometimes." They think: Bias! Outrage! And some rush to "J'accuse!". In any large group this will happen to quite a few people every day (sometimes the same few!), and it takes 100x as much effort to responsibly respond to the j'accuses as to fire them off in the first place. This would amount to a DoS attack on resources that we could otherwise use to make HN better.
Therefore our goal should not be the impossible one of placating everybody with a litigious bee in their bonnet, but (a) being able to say "no, we don't do that" in good conscience, and (b) retaining the good will of the community as a whole. Actually we like to help users with bees in their bonnets too, but to think it can always be done leads to madness.
We're toying with adding vouch links so users can vouch for a story they don't think should be marked as a flamewar. That's something that might improve the quality of HN.
Makes sense about releasing more stats - how about internally then? Could you take a look at the number of pro- and anti-YC stories that you've turned the system off for and see if it seems unbiased? Often it's easy to judge each story one by one and think you're being unbiased. But we're all biased. Going back and checking those stats may reveal something you didn't think was happening. Or not. Not as any kind of placation, but more as an internal quality check so you know you're on the right track.
Sounds like an odd cough algorithm.