I have a hunch I am misunderstanding your argument, but does that mean the only way to build a "true reasoning machine" would be to just create a human.
I guess what I'm really asking, what would you expect to observe to make it not illusory?
I guess what I'm really asking, what would you expect to observe to make it not illusory?
A desert mirage in the distance is an illusion; to the observer, it's indistinguishable from an oasis. You can only tell that it's a mirage by investigating how the appearance was created (e.g. by dragging your thirsty ass through the sand, to the place where the oasis appeared to be).
The illusion happens when, clearly, the alleged reasoning behind how such a system comes to be is based on prior knowledge of the system as a whole. Meaning, its construction/source was within the training data.
My opinion is it isn't binary, rather it's a scale. Your example is a point on the scale higher than what it is now.
But perhaps that's too liberal a definition of "reasoning" , no idea.
We seem to move the goalposts on what constitutes human level intelligence as we discover the various capabilities exhibited in the animal kingdom. I wonder if it is/will be the same with AI