Surprisingly, they hallucinate more than you might think.

https://x.com/lefthanddraft/status/1777495120910426436?s=46