Even as an LLM skeptic, I've had chat sessions with ChatGPT that felt unbelievable at the time, where it almost seemed to be generating incredibly cogent ideas and arguments all on its own. Then, after a few days, I went and reread the transcript, only to realize I'd been hinting and nudging it along much more than I consciously thought at the time. I only remembered the answers that were the most impressive and forgot about the ones that missed the mark.
ChatGPT is really good at picking out the key ideas in the prompt and responding to them. It's really easy to inadvertently nudge it to give a certain type of response, which of course makes it feel all the more astounding when it gives you exactly the type of answer you already unconsciously expect.