It's only indistinguishable to an outsider, because one side of the conversation has practiced puppeting the other and is looking for confirmation instead of probing for flaws.
> This is an actual possibility. What you say may be true. The irrationality occurs because this claim is possible for ALL possible conversation threads here on HN.
It's just not worth it to spend much time considering whether every single conversation thread is fake, but sure it's possible. That is not irrational, and it's not a contradiction with dismissing the AI.
Neither kind of conversation proves there are two sentient actors to an observer. So "this isn't good enough" isn't a claim exclusively for the AI chat log.