A review of research trials finds they are rigged to exaggerate AI performance

Libelous editorialized title violates HN guidelines.

Article actually says

-----

There were 81 papers in total, of which only 9 were prospective studies. All of the studies were in the area of medical imaging.

The study design often meant AI performance was exaggerated. For example, the human comparison group averaged only 4 doctors (too small!). Often expert and non-expert performance was combined together — making the expert group perform worse on average.

The reporting quality of the papers was typically pretty low, with low scores on the CONSORT and TRIPOD checklists. Papers often made claims of being better than clinicians in their abstract, without including the appropriate caveats.

Most of the studies didn't have their code or data publicly available. This raises questions around whether the research can be reproduced by other scientists - a key principle of science.