A minimal standard for evidence availability in black-box systems

A sign says: "Dogs must be carried on the escalator."

At first glance it seems clear. On a second read, it becomes obvious that what matters is not the dogs, but whether they are being carried.

Grandma calls out: "The chicken is ready to eat."

Many system outputs have the same problem. They look definitive, but they silently hide whether the required conditions were ever met.

When systems consume outputs from black-box algorithms, the usual options are to trust the conclusion or ignore it entirely.

In clinical genomics, the latter is traditional. For example, the British Society for Genetic Medicine advises clinicians not to act on results from external genomic services https://bsgm.org.uk/media/12844/direct-to-consumer-genomic-t...

This post describes a third approach, grounded in computer science. Before any interpretation, systems should record whether verifiable evidence is actually available.

The standard adds a small but strict step. Each rule first reports whether it could be checked at all: yes, no, or not evaluable. Then the evidence is used in reverse, not to confirm the result, but to try to rule it out. If removing or negating that evidence would change the outcome, it counts as real evidence. If not, it does not.

Crucially, this forces a simple question: could the same result have appeared even if the evidence were absent or different? Only when the answer is no does the result actually count as evidence.

The idea comes from genomics, where hospitals, companies, and research groups need to share results without exposing proprietary methods, but it applies anywhere systems reason over incomplete or black-box data.