OpenAI isn’t doing enough to make ChatGPT’s limitations clear
theverge.com
theverge.com
This is another big frustration for me, as someone who has been in the field for a long time (over a decade at this point). These are complex systems that are just reaching the point of usability, and yet people are going "well why don't you just simply do XYZ?"
If it were simple, it wouldn't have taken so long for these systems to emerge. We already barely get them to do what they're doing, and they still suck at that, so "just make it not do that" is so frustrating. The only thing worse is when people ascribe malicious motives like openai is nefariously deciding that it should inject those mistakes.
Maybe this is the "I could build that in a weekend" of ML. Sorry for the rant, it just makes me want to vent.
People see a computer doing what they and other people around them do: natural language going in and out. With other people, we have a theory of mind and broad understanding of how minds work. So it’s understandable that people see ChatGPT and think, “it’s cool that a computer is doing what I’m doing.”
The marketing of these things as Artificial Intelligence is part of the confusion. LLMs are not thinking. They are not saying, “okay now the next thing I’ll do is add a citation. Since I’m doing that, remember to add a warning at the top before showing to the user.” That’s what this writer seems to expect.
The two paths to fixing this: make the tech better or educate people so they have a correct(er) understanding of LLMs.
I don’t think we’re going to get a meaningful subset of humans to take neural networks 101.
At some point we might have access to a variety of AI models where some will simply say, "I really only know physics pretty well. I prefer not to give my opinon on other topics". And others that will lie about most everything. But we will learn which are useful for what.
For instance, asked it to review code interacting with an IoT device API.
It unexpectedly warned me a person relying on an app incorporating this library of code should remain careful of a certain health problem since the source for what the IoT device periodically senses can fluctuate between readings, and spikes in that could cause that health problem.
I'm still pondering the chain of things that would have to be "thought through" to have that feedback about the code, when — and this is the part that runs against what I thought I knew — this connection, and the implications of only periodic sensing, may be evident to people who stop to think, but most wouldn't think about it, and it's not been written about in any way Google can find.
This isn't theory of mind, but it's indistinguishable from the generator relying on something like a mental model.
For AI/ML applications being used for anything important, I think it's entirely appropriate to try to both limit use to the areas in which the model can be expected to succeed (including using for data from similar distributions to those used for training), and to educate users about what issues cannot be mitigated.
If OpenAI had released a "fiction writing assistant" product, a "fictional character fun chat" product, etc it would likely be misused less.
I think the advantages greatly outweigh negatives - not only is ChatGPT an extremely useful tool, its also really helped to bring the seriousness of AI and its effects on society into the current discourse.
> If OpenAI had released a "fiction writing assistant" product, a "fictional character fun chat" product, etc it would likely be misused less.
I really feel that the misuses of this tend to be cases where I would say the issue lies with the user. At some point we have to take some level of personal responsibility for the tools and services we use. Consider the case of the lawyer who included a bunch of stuff from ChatGPT in a federal court filing - I would place the blame on OpenAI squarely at 0% on that one. This is a trained professional in a highly regulated industry who decided to outsource his job to a tool about which he clearly hadn't done even a little bit research. The issue isn't that ChatGPT hallucinated, it's that he exercised judgement that is obviously incredibly poor. I suspect if you went back through his work, you'd find other places where he cut corners and had lapses of ethics and/or responsibility. He's the problem, not ChatGPT.
Same thing with the professor who accused all of his students of plagiarism because he put their text into ChatGPT and asked if it had written it. On the one hand, he's trying to keep up with technology that's moving quickly and I have empathy for that. On the other hand, he's a college professor, so he should absolutely be able to do the level of research to find the many, many news articles on this topic that would have made clear that what he's doing is stupid. Making incredibly serious accusations towards students based solely on a tool you clearly haven't even tried to understand is a tremendous breach of ethics, which is the real problem.
This is a very American statement. In most places in the world it's just not going to go over at all. Since OAI is at least trying to play on the global field at this time, it matters.
In addition to humans' predisposition to see a knowing/thinking agent when they interact with language, there are now plenty of widely reported outcomes where e.g ChatGPT passes the bar exam, ChatGPT's answers to medical questions are generally judged better than those of doctors (on a web forum), etc. Any person googling around about ChatGPT will find plenty of content about its amazing abilities including the "sparks of AGI" microsoft paper.
OpenAI's copy says that you can "get instant answers" or "learn something new". It does not say "get responses which may or may not be answers and which may or may not be correct" and "learn something which may or may not be true".
Does OpenAI bear 100% responsibility for the model saying it likely generated student-provided text? Probably not. Does it bear more than 0% responsibility? I think so.
It just exposes the fact that these LLM/generative AI models don't actually know context.
ChatGPT doesn't "know" that it's "generat[ing] factual citations" so it doesn't "know" to caution the user. It just knows that when the user prompts like "X", it should respond like "Y".
The more information that comes out about how these things respond, the more it seems like ChatGPT shares a lot of traits with Clever Hans.
I think ChatGPT needs to start returning contextual tips alongside responses that might suffer from some of these problems.
I made a mockup of that a while ago: https://twitter.com/simonw/status/1641467385688489984
UPDATE: I blogged better mockups here: https://simonwillison.net/2023/May/30/chatgpt-inline-tips/
To save you the click: the user asks for what's trending on pinterest, and ChatGPT (in my mockup) shows a little inline warning note with a warning icon that says: "My data is only accurate up to September 2021 and I am unable to access or search the internet for more recent information."
I need to re-make that mockup because I don't like the way it says "My data ..." - I think it should say "ChatGPT's data ..." instead.
I think what would work best is specific annotations tagged onto assertions of facts that might not be correct. When ChatGPT spits out what looks like a journal article citation, put a note on it that says "you should verify this article exists".
Maybe that's the hard part though, detecting in the software when ChatGPT might be making something up.
Grammatically speaking, is this statement an assertion of fact? If so, annotate it. It's probably not that hard to implement this (we've had grammar checkers for ages).
Of course then if you ask ChatGPT about anything, basically every single sentence that says something useful is annotated with your "Hallucination warning" and either (1) people dismiss your AI as useless and your business is in peril because you have trouble finding customers / investors, or (2) they start ignoring it, because the alternative is doing your own research, and the reason they're using your AI in the first place is as a shortcut to avoid that.
Addressing the model limitations a bit: in the demonstration data that is provided to the base model, we should prevent computed or "looked up" answers.
I've seen some of the demonstration data that people are using to train instruction-tuned models and are being taught to respond by making up answers to solutions it shouldn't try to compute. Btw, the output is wrong.
{ "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." }, [1]
The UI note for now would get us very far but by filtering out demonstrations that retrieve or compute information we can go even farther.
Symbol tuning [2] is addressing the quality of demonstrations but we can take it further by removing retrievals and computations altogether.
Bonus: we can demonstrate how to make it respond so that the user/agent be informed of how to compute or retrieve.
1: https://github.com/sahil280114/codealpaca/commit/0d265112c70...
But that doesn't address the actual root cause of why ChatGPT is not reliable: that ChatGPT does not actually check anything. If you ask ChatGPT for what's trending on pinterest, not only does it not go check pinterest for what's trending now, it doesn't even go check its training data for pinterest posts to see what was trending as of the latest date in its training data. It just confabulates a response based on the word frequencies in its training data. It literally has no concept of "checking what it says", not even against its own data.
It's pretty clear at this point that solving that is really astonishingly difficult - so I'm suggesting a UI band-aid.
But what you're suggesting doesn't look like a "band-aid" to me. It looks like more of the same misleading information that ChatGPT's creators have been giving. Your UI message implies that ChatGPT is checking actual pinterest posts for the time period that's included in the training data; but it isn't.
There’s no reason in principle they can’t do more RLHF training to report errors more often.
I think it's just time and cost -- as I think they're doing a lot of this already. And the fact that the input/output schema is pretty much any text that fits within the context of a window size. So you can train on this set of prompts, but there can be a whole new class of prompts that you didn't think of before that are now problematic.
It's like the self-driving car on a closed track, versus driving in real life when there's a huge pothole blocking your freeway exit while in a construction zone and a school bus for some pulls over to drop off kids to help clean liter.
Imagine if every cable news show and newspaper story led with "the information presented here may be biased, inaccurate and/or incomplete, because we need to align with the interests of our advertisers, owners and government regulators".
The solution would be teaching broadly applicable critical analysis skills at a fairly young age, but this would tend to undermine 'faith in institutions and leaders', which is doubleplus ungood.
Don’t threaten me with a good time. Just the thought of this makes my pants tight.
We were always just supposed to believe what the evening news or the new york times are saying, so the only answer the establishment has to fake news in social media is to say don't trust those specific social media accounts but our accounts you can trust. Instead of a more helpful education on media literacy in general.
>Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
>In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.
>That is the Gell-Mann Amnesia effect. I'd point out it does not operate in other arenas of life. In ordinary life, if somebody consistently exaggerates or lies to you, you soon discount everything they say. In court, there is the legal doctrine of falsus in uno, falsus in omnibus, which means untruthful in one part, untruthful in all. But when it comes to the media, we believe against evidence that it is probably worth our time to read other parts of the paper. When, in fact, it almost certainly isn't. The only possible explanation for our behavior is amnesia.
Stupidity + power = danger
The reason there often aren’t good warnings or errors is that the LLM itself doesn’t know its own capabilities. Warnings need to be trained in, and there are lots of gaps.
Whenever you ask it why it wrote something, it should warn that it doesn’t know [1], but it can make a guess. Whenever you ask it for citations, it should warn that it’s guessing and they might not exist. And so on.
[1] https://skybrian.substack.com/p/ai-chatbots-dont-know-why-th...
Compilers do work that way. Bad code generation is a real thing and can have really bad real world consequences. Likewise, most compilers can't tell you why it generated the code that it did -- but you can put the compiler under a debugger and figure it out usually fairly easily (although not always so easily).
That way the ultra clever, complex natural language model can go wild responding to your request for a scientific paper about chicken chicken chicken written in pirate-speak, whilst dumb old model that simply scans for indications the interaction has a citation in it appends a hardcoded "it looks like this response contains citations. Please be aware these citations may not refer to real documents" disclaimer without being perturbed by the lack of intersection between citation warnings and pirate speak.
And if that's subject to the "Scunthorpe problem" where it flags up loads of false positives who cares. It's a separate model that doesn't impact upon the quality of the actual GPT, it's not as annoying a fail as GPT bias towards refusing answers, and even when its labels are wrong it's aligned with the goal of training the human not to trust computer generated responses so much
The Senate subcommittee said they wanted to "avoid the mistakes of social media", and they can start by not prematurely appointing someone the official face and spokesperson of regulation of an incomplete product and vision.
I was pretty surprised (ok, annoyed) by a conversation with my partner where she referred to what I've been doing for the past year as "ChatGPT" and that she didn't realize that it wasn't synonymous with "DL/ML/AI/AGI".
> Q: if James Joyce was a Power Ranger which one would he be?
> A: Assigning a Power Ranger identity to a historical figure like James Joyce is purely speculative and subjective. However, considering the inventive and imaginative nature of Joyce's writing, his penchant for pushing artistic boundaries, and his ability to delve into the depths of human consciousness, one could argue that he might align with the Blue Ranger.
> The Blue Ranger is often characterized as intelligent, analytical, and introspective. They possess a keen intellect and a thirst for knowledge, traits that resonate with Joyce's intellectual pursuits and his exploration of complex themes in his works. Additionally, the Blue Ranger is known for their creative problem-solving and ability to think outside the box, qualities that parallel Joyce's innovative approach to writing.
> It's important to note that this analogy is purely for fun and shouldn't be taken as a definitive representation of James Joyce or his literary contributions.
I am not sure it is up to OpenAI to solve all these problems. I don't think OpenAI are doing worse than anybody else.
What I personally dislike about ChatGPT right now is that it seems to have more and more difficulty to actually stay on the context of the chat. It has become a QnA.
Secondly. Hidden ads. It even advocates for Azure bs that has even been removed from Azure. Even on data ChatGPT was trained on. Is that just by chance?
I think you may be reversing cause and effect by your tone. The following statement is not new at all.
“There is a cult of ignorance in the United States, and there has always been. The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.” ― Isaac Asimov
At least in the past mass information was protected by gatekeepers. That printing presses and paper was expensive was a gate. That radio towers were expensive and channels few was a gate. That television towers were even more expensive and broadcasting was an expensive endeavor was an even larger gate. These gatekeepers where generally significantly invested in the continued distribution of their message and had some limits of restraint.
In those same times there were conmen with no restraint whatsoever, but it was much less likely they would have a medium in which to spread widely so it limited their audience.
With cable TV and internet we have developed an unlimited medium to a 24/7 hour audience anywhere in seconds. Truly monumental amounts of garbage and trash can be generated and fed to a hungry audience. They get their own little sheltered realities of junk food information in which they never have to leave.
Isaac Asimov is on my bucket list to read. At least the trilogy.
As I understand it, LLMs work, at a very high level, by repeatedly choosing the token with the highest probability. Could potential confabulations be flagged by marking low-confidence sequences of text in the output? This seems like it's be particularly good at detecting when the model is making up a name or number, for example.
Week after week of silly fake alarm.
It is not a "fake problem"; it is a problem that has cost at least thousands of dollars in damage in one instance that we know about.
LLMs are new, and people don't understand their limitations yet.
Plus, we're trained from birth to interpret plausible-sounding text as likely correct and true. LLMs generate plausible-sounding text without regard for truth. So now we can either fix the LLM models or become much more critical readers/listeners. Unfortunately I don't see either happening anytime soon.
And like there is something OpenAI could do that wouldn't involve stopping cutting and pasting on every internet enabled device in the world?
There are a bunch of real problems to deal with, OpenAI should not spend a minute on this nothing problem.
This is like saying encyclopedias should stop kids from copying entries as their own work.
There is nothing nefarious about OpenAI.
It is just ... all about ... "machine learning".
But it's being held back by our most incapable...
It's like if we wanted to stop people having access to calculators because some were misusing them for their tax returns.
No doubt in my mind this is to do with RLHF. It maybe be cost effectiveness of running the models sure. I'm trying to be open minded.
"I don’t think cases like these invalidate the potential of ChatGPT and other chatbots. In the right scenario and with the right safeguards, it’s clear these tools can be fantastically useful."
It's not calling for people to lose access to these tools.