I have had a similar interaction when I was building an AI agent with tool use. It kept on telling me it was calling the tools, and I went through my code to debug why the output wasn't showing up, and it turns out it was lying and 'hallucinating' the response. But it doesn't feel like 'hallucinating', it feels more like fooling me with responses.

It is a really confronting thing to be tricked by a bot. I am an ML engineer with a master's in machine learning, experience at a research group in gen-ai (pre-chatgpt), and I understand how these systems work from the underlying mathematics all the way through to the text being displayed on the screen. But I spent 30 minutes debugging my system because the bot had built up my trust and then lied to me that it was doing what it said it was doing, and been convincing enough in its hallucination for me to believe it.

I cannot imagine how dangerous this skill could be when deployed against someone who doesn't know how the sausage is made. Think validating conspiracy theories and convincing humans into action.