You're right it doesn't. At least not completely. I was thinking about precision (i.e.: if the test is positive, what are the odds that its prediction is true). It turns out, that accuracy is not defined as "true positive / (true pos. + true neg.)", but "correct predictions / all predictions". The whole point of OP's statement: "It's kind o remarkable how even a 99.9% accurate heuristic is insufficient at scale.", which you actually support with your example.

> There is an important difference between scenarios where we care about the relative versus absolute frequency of errors.

The context is chat control without probable cause over the whole population of Europe with a low prevalence. My point, and presumably that of OP, is that even a small relative frequency of errors will yield an unsustainably high absolute frequncy of errors.

> This is merely information provided to a human agent.

It will be in theory. In practice the human agent will just forward the decision. A human agent is not sufficient; you need to test only with probable cause for the kind of scenario we're talking about. The exact opposite of "Chat Control 1.0 and 2.0".

P.S.: The comment I originally replied to choose a very convoluted way of saying that the false discovery rate of the test matters for a proper evaluation. Both you and they explain this by throwing numbers without context in combination with slightly inaccurate definitions. I got the definitions mixed up differently, which led to this follow-up.