It fails the usual memory tests. Zach W proposes asking it first about what it liked on its trip to Australia and soon after, why it has always avoided visiting Australia or some such thing. It provides plausible answers to both - which is problematic.
Except that there exist people who behave somewhat oddly similarly.
Right, humans also lie and can appear to be arbitrarily inconsistent or forgetful.
Lying is not a counter to my point. Lying is done intentionally; accurately reporting what you had for lunch is not possible if your sentences are generated probabilistically, and neither is lying about it. What if you accidentally told the truth?
Yes, of course, ‘intentionality’, that clears everything up.
Artificial Stupidity (or Idiocy - if we want to keep the acronym)