For ex. I tried asking ChatGPT about cartoons from childhood. I wrote "What was that cartoon in the 1980s that was based on some kind of gummy candy?" and it correctly identified "The Adventures of the Gummi Bears". I wrote "Sing the theme song for me" and it produced the song missing the first verse. I wrote "That is missing the first verse!" and it produced the whole correct song.
On the other hand, when I asked it to describe the instrumental 90s X-Men theme song, it tells me:
'...the lyrics are epic and uplifting, with lines like "We're the X-Men, we're the best there is at what we do." The song also has a sense of urgency and danger, with lines like "We're fighting for our lives" and "The mutant race will survive"...'
When I put "The X-Men theme song doesn't have lyrics" it readily accepted the correction but unlike getting the missing first verse I wasn't really getting any verifiable information by making the correction.
And of course it was happy to tell me about a nonexistent Gummi Bears / Rescue Rangers crossover episode.
This does make sense as, beyond what has been typed, there is no memory implemented in most of these models, so revisions are currently the only game in town to get more accurate results.
What surprised me most concerning this entire situation is that the model did insist on being correct. Normally ChatGPT has been set up to be a bit cautious and more likely to admit to having made a mistake, to the point where if you ask in a direct manner like this lawyer has done, the model may claim to have been incorrect, even when the output was actually correct, in my experience. Bings implementation, of the same underlying model, meanwhile can be so forceful in trying to convince users that the output is correct, even when provided with online resources that show the oposite, that it would not be unreasonable to feel gaslit by that LLM.
The rest of this situation was not very surprising and I have do admit, I am happy that this was caught right away. Lawyers actions have a massive impact on countless people every day, if this had not become such a public scandal right away, perhaps a lot of defendants would have suffered under improper representation due to reliance on imperfect models.
That and that it’s just a language model, an approximation of neither the world, nor a body of knowledge, but of English, and not an answering machine at all.
This reminds me of a story, I believe referring to Pascal's demonstration of his (newly invented, entirely mechanical) calculator to the Royal Society. He showed that pressing certain levers in the correct order means you want to operate on certain numbers, and certain other levers mean choosing the operation you want, and then by cranking his calculator you could read off the answer of doing your operation on your numbers. World's first calculator! Someone asked: if you press the levers wrong, do you still get the right answer?
Their mental model is "magic" and they don't understand the details of how magic works, because it's magic.
(the answer of course that the LLM doesn't actually search the internet and/or doesn't have access to a law database it can query)