Right now, without even being allowed that extra digging, it is darn difficult to be sure about its high-dimensional musings.
What I would like to see most is the actual word/phrase/source distributions for a given prompt, in order to judge the sparsity of the underlying training data and subsequently, 'crowd source' the remaining gaps.
The other day I was struggling to parse a Japanese sentence, a particular grammatical construction made no sense to me. I wrote the sentence in ChatGPT, asked it to break it down for me, and it came up with a plausible-sounding explanation. Problem was, I couldn't find any hit on google when I searched for the thing it was talking about. So I asked ChatGPT to give me more details, tell me what I could search for, and it would insist that its explanation was correct and then gaslight me by telling me that the reason I couldn't find anything on Google was because it was a niche subject not usually taught in grammar books.
After some more searching around and double-checking it turns out that I had misread a kanji and the sentence I typed into ChatGPT was complete gibberish as a result. ChatGPT's explanation, while sounding very plausible, was complete fabrication.
The idea that some inexperienced people are shipping software using this tool is insane to me.
For example I had a massive sql query that was loads of statements unioned together, I said “for each statement remove this filter and add this filter” and it would go “certainly, here are the first four, I have used an example table name feel free to change it” then I’d say “can you do it for all the statements, not just the first four” and it would say “of course here you go” and just give me the first four but also makes them useless by changing the table name!
I’ve got great hopes that one day I can get it to help me shape the code in a way that the jetbrains ide’s can’t today - for those I have to choose from a set of available operations - I want to talk to it and get it to change the code in a set of operations that I choose!
One day maybe :)
Instead the model decides to make stuff up and pretend that it knows. That's vastly worse.
It reminds me of the early days of DuckDuckGo, when if you searched for something obscure with no matches online it would still fuzzy match some garbage like a binary blob in a Chinese PDF while Google helpfully would just tell you that it couldn't find anything.
Does the model know it doesn’t know though? Does “know” even make sense as a concept here? I don’t know if it can really introspect like that, but of course it would be so much better if it could can have some sort of confidence score with each answer.
I really think you're trying to compare apples and oranges, in multiple ways. For one, we can test the software by running it, which is a pretty different problem from asking language questions, with a much slower ability to verify correctness (based on what you describe and what I imagine).
I'm not saying your experience is invalid. In my own adventures, the equivalent of what you did was my writing some incomplete bash in an existing script, wandered off to another part of the code. I then came back to that incomplete snippet, and though it was some unfamiliar syntax someone else had written (vim even highlighted it like it was special!). Naturally I went and asked ChatGPT what that snippet did, and wasted 15 minutes trying to corroborate it before checking the git history or something and realizing my own error.
As long as the tests are not also written by ChatGPT...
Many critical security issues require a deep understanding or the code or some intense fuzzing to discover, it's not enough to ask ChatGPT "write me X" then superficially glance at the output to validate that it looks correct. That's the part that worries me. Completely broken code will be caught immediately, but subtly broken code may linger for a long time and make it to production.
And from my limited experience with ChatGPT, it seems very good at making up broken things that look superficially correct.
I don’t notice nearly as many errors when I’m asking it about things I’m not already an expert in. The most likely explanation is that it’s fooling me.
I pay for gpt4. People using chatgpt to “learn” are absolutely slurping up incorrect information without knowing it.
Imagine trying to teach yourself physics with textbook where 10% of it is completely but convincingly wrong.
This is basically how many people already use the internet, read information on random blogposts, stack overflow and more, then take that it's true for granted. ChatGPT isn't really different than reading those things in the end.
What has to change is how people treat information that they read, no matter if it's from a blog post, ChatGPT or a friend. Verify everything, before you'd bet on that it's true.
I've used GPT4 to understand topics I've had no exposure to before, and it's true, a lot of the things GPT4 writes isn't accurate in the end, but lots of things are accurate too. That you can mold the information in a different way than static browsing, makes a lot of difference.
Overall, even with some false information, ChatGPT personally saves me a ton of times and I find myself only using Google for verifying information now, not for finding information.
TLDR: trust your gut feeling.
Credit to the creators of these systems for getting it even close, but until they can actually correctly cite their sources and provide followup references to real content instead of hallucinating titles and emitting hyperlinks to unrelated medical research papers, I'll stick to using it as a very rough overview of a subject, just to get the right collection of search terms for following up in traditional sources.