Why is that any different from the utter mess of a world humans find themselves existing in?
Let's work with that.
In a given context window or conversation, yes, you can have a very human-like conversation and the chatbot will give the feeling of understanding your world and what it's like. But this still isn't a real world, and the chatbot isn't really forming hypotheses that can be disproven. At best, it's a D&D style tabletop roleplaying game with you as the DM. You are the human arbiter of what is true and what is not for this chatbot, and the world it inhabits is the one you provide it. You tell it what you want, you tell it what to do, and it responds purely to you. That isn't a real world, it's just a narrative based on your words.
The question of how to evaluate whether something is conscious is totally different from the question of whether it actually is conscious.
I don't know what you're thinking of, but mine are.
Practice of any kind (sports, coding, puzzles) works like that.
Most of all: interactions with any other conscious entity. I carry at least intuitive expectations of how my wife / kid / co-workers / dog (if you count that) will respond to my behavior, but... Uh. Often wrong, and have to update my model of them or of myself.
I agree with your second paragraph.