A moment of reckoning for natural language processing

An obvious instance of Goodhart's Law:

https://en.wikipedia.org/wiki/Goodhart%27s_law

One of the challenges here is that we don't know what intelligence, or even comprehension, truly is, so we have no way of assessing whether a system has it. Many times I've watched people read something perfectly well, and then when discussing it afterwards it becomes apparent that they said all the words, and can recount "things" from it, but actually just don't "get it".

And no, I can't define what I mean by that ... that's part of why all of this is so hard.