Working on a follow up article on exactly this (it is mathematically impossible and we have receipts...)
Roger Penrose made an argument like we can know Gödel's theorem is true without being able to prove it but AI can't, but I think you can figure both are guessing in a similar pattern recognising kind of way.