AI "progress" is nearly always presented in a way that confuses people, and usually includes some amount of fear mongering. I think its a problem between research and PR in general, but its exacerbated by weak theory in deep learning.

When Duplex is presented, the researchers do not communicate the domains where the research is reproducible, and where it is not. There is not theory that describes what domains can gain utility.

It seems like the obvious outcome of presenting research, without a theory that the audience can understand, is that the research will be misunderstood. Press narratives are required to latch onto this every time there are eyeballs on an AI demo: "Turing Test?", "Ethical Questions?", "Job Loss?", "How advanced is AI?", etc.

In terms of a "goal that might validate advance", I think a theory that helps others to understand the limits of the research. When other researchers, beneficiaries of the technology, and the press understand the real scope of the research it reduces the need for critique like OP.