In 21-st century should it be a mandatory requirements that people in leadership positions in highly technical fields must have at least basic technical qualifications in the areas they are making decisions?
https://abcnews.go.com/US/court-oks-barring-high-iqs-cops/st...
False Positive Identification Rate grows as the enrolled database gets larger, but True Positive Identification Rate does not (it actually goes down a tiny bit, as some true matches get lost due to lower-accuracy initial search passes). Both FPIR and TPIR assume that the entire universe of possible people is enrolled - i.e. that you are in a "closed set." So what they're really telling you is what the error and success rates are given how many people are in the universe that you are trying to distinguish from each other.
But FPIR is a closed-set metric, that tells you how many false positives you will generate per search. If not everyone is enrolled, you have an open-set problem. And the rate users actually care about is Precision (True Matches / (True Matches + False Matches)). They want to know if the matches that come out of the system are correct.
So looking at rates, that's simply TPIR / (TPIR + FPIR), right? And TPIR is largely constant as N grows, right? So FPIR is the effective error rate, right? Pretty much the whole industry agreed, so Aadhaar dynamically changed its threshold during initial enrollment so that FPIR would stay constant and therefore Precision should stay constant.
** Pause here to see if you can spot the issue **
The intuition that should have screamed out at everyone is that comparisons are independent events and it would be really, really weird if the precision of any given returned match depended on how many of these events were independently happening in other computing threads. It shouldn't matter at all how many independent comparisons are happening. Because they're independent.
TPIR doesn't grow because it assumes everyone is enrolled. Which is definitionally not the case during ramp up. If you want to know how many True Matches the system will spit out, you need to multiply TPIR by probability of enrollment! The Precision is defined by how big of a population you are trying to distinguish, not how many of them are enrolled in the database. The database size is basically used to give you a prior that any given comparison is true; the simplest prior is 1/pop_size, which is the same a 1/db_size for a closed set. But not for an open one.
If the database grows by 10x, you get 10x the false matches because FPIR goes up and you get 10x the true matches because they are 10x more likely to be enrolled. In other words, 10x the matches with the same ratio of true to false. Instead, they lowered the threshold early in the lifecycle, making it more error-prone. This system cost more than $1B in a country where that's a big deal, and had a lot of eyes on it. Trust me, the "experts" were involved and a bunch of them had PhDs. Didn't help.