> Consistent across performances and devices

LOL! I think the assumption that an ML system would be robust across these contexts is not well founded. Things as innocuous as the lighting, background, or placement of athletic tape could easily wreck a model like this.

The thing about human judges is that the modes of failure are pretty well known at this point; and human perception and visual processing is pretty robust to perturbation.