Hacker News Reader

Comment by pjungwir | Hacker News Reader

Any time I see "accuracy" reported as a single number, I assume it's B.S. and "for marketing purposes only". If you're serious, give me the precision and recall.

View on HN

>If you're serious, give me the precision and recall.

The thing is decision-makers have no idea (and no care to know) what those are.

Maybe these people shouldn't be decision-makers then. We need both greater transparency in decision-making processes, and greater education amongst the population about statistics, causal inference, and the scientific method. In order for the citizenry to hold decision-makers to account and obtain a higher standard we need:

1. Citizens to have the time and motivation to seek out and examine primary evidence.

2. Citizens to have the skills to understand and critique that evidence.

3. Robust mechanisms to allow citizens to replace people in power.

This is an inherent problem with the specialization of labor. Those who run police departments have more relevant specializations to their job role than information science. This isn't a problem with statistics or science in particular -- it's an issue with every specialization. This is only going to get worse as popular opinion of the role of higher education is starting to shift from a more liberal-arts focus to a job-placement focus.

Correct and that's unlikely to change, but it should be grounds to throw out the prosecution. Yes, this will make it harder to convict real criminals. Too bad.

Precision is an attribute of the deployed system. The same tech in an affluent suburb will have different precision deployed in an inner city. Ran into this exact problem in biometrics, where NIST publishes tech error rates like FNMR and FMR, and system owners don't even know precision is a thing. They expect the vendor to tell them if it will work, when it depends how many true and false events you throw at the system.

I've never understood why ML folks insist on using precision, which depends on the base rate. Why do they not use probability of false alarm as in traditional detection theory?

It doesn't really depend on the base rate though. The idea is that a high precision model will have all the targets at one end of the distribution, regardless of base rate.

Precision is P(T|R), where R is the event that a thing is reported, and T is the event that the thing is true. By Bayes' rule, P(T|R) = P(T)/(1+Pfa/Pd). So in terms of the traditional statistics Pd and Pfa, it is proportional to the base rate.

Reply on news.ycombinator.com