The models are a lot of fun to play with, but yeah, every time I've tried to use them for something "serious" they nearly always invent stuff (and are so convincing in how they write about it!).

Most recently I've been interested in what's happened with the 4-color theorem since the 1976 computer-assisted proof, and decided to use GPTChat instead of google+wikipedia. GPTChat had me convinced and excited that, apparently the computer-assisted part of the proof has been getting steadily smaller and smaller over the years and decades, and we're getting close to a proof that might not need computer assistance at all. It wrote really convincingly about it! And then I went and looked for the papers it had talked about. They didn't exist, and their authors either didn't exist, or worked in completely unrelated fields.