ChatGPT provides false information about people, and OpenAI can't correct it
noyb.eu
noyb.eu
https://hachyderm.io/@inthehands/112006855076082650
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.
Yes, there are good cases where "realistically sounding bullshit" is useful! I remember in the early days, before everyone grew kind of tired (well, at least me) of GPT, people used "hehe write me PC manual in style of slam poetry" or whatever, and that is fun.
However, people use it to get facts!
Just a few days ago, I have read somewhere that hospitals are planning to use LLMs to replace nurses. That is horrifying. Absolutely horrifying.
AI probably has a niche where it's useful, but because it smells like a magic money machine that will allow managers to replace employees and create value from essentially nothing, modern capitalism dictates we must optimize our entire economy around it, no holds barred, damn the torpedos and full speed ahead because "money." I just hope the fever breaks before people start getting killed.
Some people thought he must be very smart, and would listen to that guy for hours on end. Most people eventually got annoyed / board and left. But he kept going. The thing was, he knew a lot of stuff. But of course, an audience is a hell of a drug, he had to keep going, and some of the stuff he said ended up being bullshit.
Now LLMs are that guy. We have automated "Cory from the 3rd floor, after a few beers". You wouldn't cite Cory from the 3rd floor on your term paper, why would you cite an LLM?
I’ve had to explain this so many times even to engineers. People keep using it as Google. It is not a mechanism to retrieve facts.
It’s rather a reasoning mechanism, that when used like a search engine generates text that looks like output from a retrieval of facts.
The article talks about OpenAI being unwilling to correct errors. But they just can’t. There just aren’t facts like birthdays for specific people discernible from the weights. Maybe for some.
So what would they correct? The best that can be done is like it is said in the article, apply filtering to refuse to answer.
It's labeled "AI", aka "artificial intelligence".
So clearly, it's intelligent enough to tell fact from fiction, right? It's intelligent enough to read and repeat facts, right? It's artificial intelligence, isn't it?
I'm playing Devil's Advocate here, obviously. I'm aware how this actually works like you, but the way it's billed you can't expect the commons to understand any of this in any other way. There is a very specific concept of what AI is among the masses, regardless if "AI" is it or not.
If you call something a duck, you can't blame someone for saying it's a duck.
The best that can be done is just what Gemini does: a button to (smartly) compare the generated text against google search results.
>> It is not a mechanism to retrieve facts
Then bing AI powered by chatgpt shows on its site
>> [Hello, this is Bing! I’m the new AI-powered chat mode of Microsoft Bing that] can help you quickly get information.
if get is a synonym of retrieve and facts is a synonym of information, I'd argue that maybe it's not people misunderstanding what LLMs are but maybe someone explaining it wrong?
Guess in the world of marketing words don't carry any value anymore, then I agree with you
(I don’t know if that’s how Bing AI works)
After all, if you've trained an LLM on a masses of unchecked data you've scraped from the internet, your training data probably includes "Joe Biden is the president" and "Donald Trump is the president" and "Barrack Obama is the president" and "Emmanuel Macron est le président" and so on. It would be understandable if an LLM was confused about who the president was.
These people think handing an LLM the contents of https://en.wikipedia.org/wiki/President_of_the_United_States then asking who the president is sounds a lot more feasible.
Personally I'm not so sure - I've never seen a RAG implementation that impressed me.
So the gaps are the only areas where the LLM can hallucinate on and if your search query is easily available information on the internet, then hallucinations will be less or none.
Edit: I have used RAG with a project that I am working on and it's quite hard to ascertain if the LLM used the information provided as part of the RAG documents or just made up information on it's own, since even without RAG, we were getting similar responses 7 times out of 10.
So if I ask Bing about me it says "Rory McCune is a Cloud Native Security Advocate at Aqua Security." without any ref.
The problem is, that's not correct, that's a job I had two years ago, but someone reading that could be forgiven for thinking that's a fact, given how it was presented.
In this case that's harmless, but I could easily see cases where it would not be harmless.
Later on, a user asks an AI "when was Foo Bar born?", and the AI then looks up the factoid database and responds with the correct factoid, or an error message.
Stop providing service in the region in which your product is unable to comply with local regulations?
You can't extract millions of euros from EU citizens with anything physical that goes against the EU law, but we like to pretend that because something's digital, it's totally fine if your product intentionally breaks the law, even if it's just for a couple of years until some random NGO sues you and courts react. I think that's nonsense.
I'm not gonna say the EU needs its own equivalent of the Great Firewall, but there should be some cost of being intentionally non-compliant, as in fines, as in the same thing the EU already does to Facebooks and TikToks of the world.
You surely understand how this isn't a compelling argument at all, right?
If you have a bug in your X-Ray software and it irradiates patients but you say you don't have the knowledge or resources to fix it, it doesn't suddenly become a way out of fixing it.
This will be a slam dunk of a case. There are many obvious paths that OpenAI can follow, such as stopping business in the EU or preventing it from giving any fact about any person.
There are actually several algorithms intended to allow fact editing in LLMs: https://github.com/zjunlp/EasyEdit?tab=readme-ov-file#curren...
They don't work perfectly (e.g. "Tim Cook is CEO of Apple" and "The CEO of Apple is Tim Cook" for some reason have to be edited separately) but they can deal with the most egregious cases. And, you know, maybe OpenAI can improve them further, given they're always going on about 'safety' as a reason their competitors should be regulated out of existence.
I wouldn’t call it reasoning. To me that implies using logic and being objective. It’s more wisdom/stupidity of the crowds, with the model designer deciding what crowd to use to create the model and then tweaking things to make the model look like it’s reasoning.
My understanding is that AI models like GPT have been able to convincingly convey the form of language, but not its meaning. It looks and sounds like how a human would communicate, but the AI is unable to imbue meaning into the words and sentences it produces. My knowledge of this comes from Lex Friedman's episode with Edward Gibson [2].
I think that's the fundamental issue with LLMs at the moment. It has managed to mostly exit the uncanny valley because, as far as most people are concerned, the text produced could just as well have been made by a human. I think this lends some credibility to the text produced by the LLM, because more or less all text ever produced has had a human behind it. This is no longer the case and as such there is bound to be a transitional period where we learn how to deal with this new technology.
[0]: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
[1]: https://twitter.com/goodside/status/1666598580319035392
Contrary to popular propaganda, the original Luddites weren't opposed to technology, they were opposed to the effect of technological progress on the working class. They knew that automation and mass production would be used to devalue labor, flood the market with inferior products, and that all of the benefits and profit from the industrial revolution would go to the corporations, at the expense of their quality of life. And they were correct.
And modern day "Luddites" were correct about the centralization and commoditization of the web, social media, crypto and NFTs, Elon Musk and autonomous vehicles, and will be proven right about AI. Tech has nothing left to offer but grift upon grift.
https://www.newyorker.com/books/page-turner/rethinking-the-l...
https://news.ycombinator.com/item?id=37664682
https://librarianshipwreck.wordpress.com/2018/01/18/why-the-...
> While inaccurate information may be tolerable when a student uses ChatGPT to help him with their homework, it is unacceptable when it comes to information about individuals.
I don’t even understand this part. Why would inaccurate information be at all acceptable for homework? You need accurate information there just as much as with people. In fact there’s probably a good deal of overlap.
This article is bad.
This is not “the media”. noyb isn’t reporting on what other people did, they are informing us of what they just did.
> Not that it isn’t important but to imply it is a new revelation seems sensationalist.
The article isn’t about the failings of ChatGPT, it’s about a specific legal complaint being made.
> Why would inaccurate information be at all acceptable for homework?
It’s acceptable in a legal sense, you get a bad grade and that’s that. But falsehoods about a particular individual on a popular resource could be quite damaging and are against EU law.
> This article is bad.
Rather, you completely misunderstood it.
Sorry if you took offense to my saying the article was bad. I regret saying that now, it was unnecessary.
It’s not one blogger, noyb (stands for “None Of Your Business”) is a non-profit focused on protecting privacy rights in the EU.
https://en.wikipedia.org/wiki/NOYB
> I regret saying that now, it was unnecessary.
Thank you for saying that. Especially on the internet where we all have the compulsion to double down, I believe those types of admissions take guts and should be celebrated and normalised.
Humans need to develop humane defense mechanisms for the new reality, tech cannot be stopped and cannot be hermetically insulated against mistakes or bad actors.
Attempts fixing hate speech on social media resulted in similar situation, now hate becoming mainstream. American social media banned keywords for racism or hate speech, only to push them to dog whistle racism and hate speech.
The problem about fixing lie is that its impossible to objectively decide what is a lie and this is true for all kind of stuff about people.
Come again?
If you get branded a conspiracy theorist your Wikipedia entry will denounce you as such and the admins and moderators will lock the article and you can do nothing about it.
Obviously ChatGPT will take information about your persona from Wikipedia if available and will update this information accordingly to the changes in Wikipedia.
And you can’t do anything about it.
It's a crappy usecase. And much better ones are typically being overlooked outside a few smart enterprise integrations.
To put it mildly - if someone wants to use LLMs to build a factual chatbot, they should probably just start mining crypto instead, as they'll waste less money on jumping on a trend. But if they think a bit about how LLMs can be used in nearly any other situation, they'll be miles ahead of the majority chasing this gold rush.
As far as I understand it ChatGPT and all other similar systems are blatantly violating GDPR, they would have to for example publish their related training data to conform.
I guess the EU authorities don't do anything for now because they don't want to admit that their funny law basically bans all state-of-the-art AI.
(Ok, Openai also broke the law in almost all countries by downloading shadow libraries, but here they at least have more plausible deniability.)
Given that you can make LLMs say pretty much whatever you want using the right prompts, this seems impossible. LLMs are not a search engine, and based on conversational context might say Emmanuel Macron is the president of France or a baby giraffe.
Can the LLM provide personal data of an individual who is covered by GDPR? Then the LLM is subject to GDPR.
Can this individual exercise their rights with regards to the data that the LLM returns about them? Arguably they can indeed exercise the right of access by means of the right prompts, but can the individual rectify errors or erase such data? If not, then the provider of the LLM is violating GDPR.
> ELI5 how is France governed? > ...and Macron is the lion, the king of the jungle.
We also know that LLMs don't know the current date, and therefore can make calculation errors (which is made worse by their poor math performance as a language token generator). So on one hand it might say Macron was born December 1st 1977 (which is correct), but if you ask how old he is some LLMs might say 45 years old.
There is an incalculable number of ways for LLMs to output incorrect information. In an effort to comply with strict regulation the preprompt contextual limit is going to be exceeded.
Also this creates a situation where all but the most powerful LLMs (and LLM providers) will be non-complaint
That is not personal data under GDPR.
«So on one hand it might say Macron was born December 1st 1977 (which is correct), but if you ask how old he is some LLMs might say 45 years old.»
Or it might say that Macron was born on 14th July 1977, which is incorrect. The claimed impossibility to correct a date of birth returned by the LLM is the trigger of the GDPR complaint that the article refers to.
«Also this creates a situation where all but the most powerful LLMs (and LLM providers) will be non-complaint»
Only under the premise that it is somehow inevitable to feed personal data of living individuals to an LLM for training, and that the only way to correct mistaken data or to stop an LLM from providing such data is "more power".
I reject the premise, not the least because, firstly, OpenAI (the most "powerful" provider) is claiming it is impossible. All that says is that OpenAI's platform was not originally designed with that problem in mind and that, as that of now, they are unwilling to redesign it from scratch only because some guy complained in Austria. It's basically a speedrun of Microsoft claiming Internet Explorer was an essential component of Windows 98.
Meanwhile, LLMs and other AI models are an active area of research. If OpenAI truly cannot stop their LLMs from returning personal data protected by GDPR, and honestly has no way to allow data holders to exercise their rights of deletion or correction, you can be sure that some startup will disrupt the LLM market by finding a way to do it without needing to out-compete OpenAI neither in hardware nor on training corpus size.
Indeed if a startup can find a way to scrub PII of living people from 20 billion pages of text (and prevent LLMs from ever hallucinating) they would be quite a valuable company, in the LLM dev space and numerous other ventures. Until then the EU might have to go without access to language models.
If I have a random number generator producing arbitrary strings, am I required to ensure that the strings do not contain untrue statements about individuals?
The fact that LLMs hallucinate is certainly no secret, even the linked article says OpenAI openly admits that they can't avoid it right now.
What would constitute enough?
Would it be enough to place a statement placed onscreen at the start of every conversation to say that information may not be accurate and that if the information was significant then it should be independently verified?
It is not. Not even people in tech understand this, let alone non-technical people. These tools are being marketed as a way to get factual information. Don’t let your knowledge of the technology blind you to the fact that people outside your circle don’t know what you do.