I agree with you up tot eh bit about 'you are being played every time' someone makes an emotional argument. Quantitative methods are very important, but so are qualitative ones.

To take an example of the latter, conservatives frequently cite the case of Kate Steinle, who was allegedly murdered in San Francisco a couple years back by an illegal immigrant who had been repeatedly deported but kept coming back. (I say allegedly because the defendant claims he stole the gun in a simple crime of opportunity and the fatal discharge was accidental. No motivation for the shooting was apparent at the time, so arguably it was a case of manslaughter rather than murder. Anyway...)

Now this case is frequently mentioned to highlight the supposed danger of allowing illegal immigrants to wander about freely. A quantitative analysis reveals that immigrants, even illegal ones, tend to commit fewer crimes on a per-capita basis than their legal neighbors, so hyping up this case to frighten people is misleading.

On the other hand, all that quantitative analysis isn't much comfort to all those who suffer when someone is killed, and ignoring that (because we can't easily quantify how sad a given tragedy is) undermines credibility in the quantitative approach because it runs counter to the fact of human experience. Soviet-style communism is a good example of crushing individual life through an unyielding adherence to the quantitative data preferred by bureaucrats, and we could find many other tragic examples from other political systems without having to look very hard.

tl;dr only looking at the aggregate data throws the baby out with the bathwater because we are emotional animals and emotional experience matters too.