LLM's will turn out to be nothing more than another glittering gadget. Except this one will obliterate any coding skills of future generations. Not to mention the gigatons of AI generated garbage text, images and music that it's filling the world with. What a waste.
ChatGPT: The larger number is 3.
User: If I have one apple, pick another apple and then get an apple from a friend, and Alice has two apples, who has more apples?
ChatGPT: You have more apples. You started with one apple, picked another, and got one more from a friend, so you have three apples. Alice has two apples.
[edit]:
User: what does good will hunting say to the know-it-all at the bar?
ChatGPT: After Will Hunting outsmarts the know-it-all student in the bar, he spots the student's female friend outside a shop later on. To tease the student further and show his wit, Will taps on the window to get her attention and, holding up his phone number, famously asks, "Do you like apples?" After she nods, Will triumphantly says, "Well, I got her number. How do you like them apples?" This line became quite iconic and is one of the memorable quotes from the movie.
Every time I've tried gpt-4 etc there are such problems. I don't use it for trivial things since then what's the point, and for anything non-trivial it screws up too much.
It is useful for translations and brainstorming, but not much else. Translating from natural language to code is a part of that, but not letting it code itself.
I mean... yeah. ChatGPT doesn't know what numbers or what logic is. It wasn't "confused". It just doesn't do that.
You're using it for something it's not intended to be used for and, unsurprisingly, it's bad at it. If I said I'm giving up on using hammers because they were terrible for screwing screws into the wall, you'd tell me I'm using the wrong tool. This is no different. ChatGPT has its uses, but it isn't magic. Just like any tool it has limitations.
LLMs have fairly well understood failure modes. They have incredible strengths, too. And they differ very much by what you use. If the ChatGPT you were forced to use was GPT3.5 (and the behavior sounds very much like it was), you literally have no idea what modern LLMs can actually do.