Hacker News Reader

Comment by sarchertech | Hacker News Reader

It’s completely unreliable because it sounds confident and plausible even when it’s wrong. Essentially it fails in ways that are very hard to detect.

If you ask it question you have no way to know whether it’s right or not unless you already know the answer.

I’ve asked it many questions about my area of expertise, distributed systems, and it was often flat out wrong, but to a non expert it sounded perfectly plausible. I asked my my wife, a physician, to try it out and she reported the exact same problem. Some of the answers it gave had even veered off from just annoying to actually dangerous.

That doesn’t mean it can’t be useful, but using it to answer important questions or using it to teach yourself something you don’t already know is dangerous.

View on HN

Weirdly enough, people seem to view this only from one side: "ChatGPT is an expert at lying!". Actually, it was explicitly instructed to sound confident, serious and expertly. The important thing here is: to most people, it is enough to sound like that in order to make them believe. This heuristic is trivial to hack, gets exploited not just by ChatBots and is downright dangerous on a societal level.

Acquiring and vetting information is a task of crucial importance situated in between journalism and science. Just trawling the internet and autocompleting the acquired driftwood is a weird approach to begin with. Doing so without transparency in private companies is simply folly.

I know flesh and blood humans who are confident in their knowledge but still manage to make shit up and get facts wrong. I make up my own mind about what's true and what's useful by considering an ensemble of sources.

In my experience with ChatGPT, it gets less wrong and more right than most humans, and it's able to find relationships among facts and concepts that are impossible with Wikipedia and Google.

It's extremely useful and powerful in its current form, in spite of its many limitations.

With other tools, we don't demand perfection to find them useful.

Just because a hammer sometimes misses the nail and puts a hole in the wall, just because Wikipedia sometimes gets vandalized with lies, doesn't mean hammers and Wikipedia aren't essential tools.

A good hammer is designed with an optimal weight distribution that makes it easy to hit the nail. ChatGPT may be better than using your own head at times, people will still sue sooner rather than later over lost digits.

However, it provides an excellent opportunity for us mere humans to reflect upon the way we organize our society in regard to vital information processing. We are awfully bad at it. So bad actually, we can't even tell how it should be in the first place and are disturbed already by some piece of software telling nonsense.

>I know flesh and blood humans who are confident in their knowledge but still manage to make shit up and get facts wrong. I make up my own mind about what's true and what's useful by considering an ensemble of sources.

They're turning the freakin' frogs gay!!!

Yeah, it's kinda funny how we simultaneously expect the ideal AI to have human intelligence, but also not to make human mistakes. Guess what? If you build a perfect replica of a human brain, it will make human errors, because that's what human brains do.

And incidentally, it's the consequences of those errors that teach us not to repeat them... which is the same feedback loop we use to train AI in the first place!

It's basically like political candidate Kari Lake in Arizona. She is an extremely confident speaker, talking about the conspiracies against her, how she really won the election for Az Gov, all of them have no evidence or support. She is such an excellent speaker, she's just chilling to me because people will often just believe someone who is so well spoken. Chatgpt is that. Imagine connecting what chatgtp says with with a "super confident fascist human avatar" that is kind of Kari Lake. It will happen because it's obvious.

We are just going to have ever more persuasive unreliable sources of information.

> It’s completely unreliable because it sounds confident and plausible even when it’s wrong

This describes pretty accurately a non-trivial amount of people I have worked with.

And hopefully after someone shows you that they can be described that way, you stop trusting what they say (at least about the thing they are known to be unreliable about).

A whole lot of people can be trusted within a reasonable identified set of constraint, and we do, while discounting them in others where they're known to spew bullshit. It's very rarely all or nothing.

Obviously. Chatgpt will randomly spew bullshit about nearly any topic though. So you can really only trust it for things you are already an expert in or things that are very easy to verify.

It's not random. It's probabilistic. There is a very big difference. The types of errors are predictable which means they can be improved, and empirically so if you follow the academic literature.

I have read enough of the academic literature to say with some confidence that the majority of the errors I’m talking about are not probabilistic in any meaningful sense. At least not in a way that can be predicted from the amount of readily available training data.

Ironically, this reply is about as useful as if chatGPT wrote it:

- there’s no cited facts

- it boldly proclaims its conclusion

- which you’d only be able to verify as an expert

…so I’m having trouble understanding what you’re complaining about with chatGPT, when that seems to be the standard for discourse.

I don’t trust a random hacker news comment by someone I don’t know or can verify any further than I can throw it, so in that sense they are probably similar.

The comment I was replying to was “the errors you’re talking about are probabilistic if you read the literature” my response is “no they aren’t I have read the literature.”

Note that I’m talking about a specific class of error and proving a negative is difficult enough that I’m not diving through papers to find citations for something n levels deep in a hacker news thread.

Here you go, here's a bunch of papers that you have not read. If you had read them then you would know that the errors are predictable and therefore there are many measurable ways to make improvements.

Toolformer: Language Models Can Teach Themselves to Use Tools: https://arxiv.org/abs/2302.04761

PAL: Program-aided Language Models: https://arxiv.org/abs/2211.10435

TALM: Tool Augmented Language Models: https://arxiv.org/abs/2205.12255

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: https://arxiv.org/abs/2201.11903

Survey of Hallucination in Natural Language Generation: https://arxiv.org/abs/2202.03629

I took the time to read through the only one of those that looked like it was peer reviewed and read the abstracts for the rest.

Survey of Hallucination in Natural Language Generation only provided promising methods for detecting hallucinations in summarization tasks, which are of course much easier to detect. Searching arXiv for a list of non-reviewed papers that sound like they might be related to the topic at hand is fun debate strategy. But no one else is reading this far into an old thread, so I'm not sure who you're trying to convince.

None of these paper prove your claims about hallucinations, and most aren't even trying to. However, even if the errors that I'm saying aren't meaningfully probabilistic aren't hallucinations.

chatGPT (as in the current release of the OpenAI tool that was released 30/Nov/22, so ~3 months ago) has this problem.

This does not mean it is a problem that is impossible to fix.

You shouldn't listen to them either then

It may describe a number of people you have worked with but it does not describe the average medical textbook which will usually describe the best knowledge we have of a condition. If chatgpt produces something that looks like it came from a medical textbook then it is hard to blame many people for believing it. More generally, people started to trust computers more than people as soon as calculators demonstrated their reliability.

I don't know, I get this feeling ChatGPT has also read all the quack books where the condition depends on the alignment of stars when you were born, or how chicken entrails land when the shaman "studies" it. Those books are also written confidently without giving any sign of being completely fabricated.

In the end, why do people believe what they believe? The answer is it connects with what they already believe. That's it. If you had a diet of what we call science, you'll have a foothold in a whole bunch of arenas where you can feel yourself forward, going from one little truth to another. If you are a blank slate with a bit of quackery seeded onto it, you end up believing the stars predict your life and that you can communicate with dead people with a Ouija board.

CGPT doesn't have an "already believe". It just has a "humans on the panel give me reward" mechanism, and all it's doing is reflecting what it got rewarded for. Sometimes that's the scientific truth, sometimes it's crap. All the time it's confident, because that's what's rewarded.

This is not a new phenomena.

Have most people invested time in making sure their web search results are accurate?

It is dangerous for sure, but that's what you got.

What if instead of asking ChatGPT for facts, we instead asked it for plausible methods of determining those facts? So instead of answering "The statue of Liberty is 151 feet tall" it could instead respond with "Look it up yourself on wikipedia."

ChatGPT has made up sources when I've asked for them, so I wouldn't 100% trust it to provide great sources. Also, if ChatGPT just redirects you to search, doesn't that remove most of the value?

> Also, if ChatGPT just redirects you to search, doesn't that remove most of the value?

That's pretty much the conclusion I've already come to. I have to verify everything ChatGPT tells me, so using it is pointless if I already know where/how to look something up.

> It’s completely unreliable because it sounds confident and plausible even when it’s wrong.

That applies to my boss, college professors, Wikipedia, or my neighbor after a couple of beers. It's not designed to give the correct medical steps to diagnosis a brain injury.

The problem is most people do believe these confident facts spoken by a leader, they aren't evaluating them. An easy example is the fairly widespread belief that somehow the us pres election was stolen. People said it, but there mountains of evidence don't exist. But millions of people believe it.

You're not thinking about this probabilistically. This isn't about "correct" and "incorrect". There is an entire universe between those two poles with regards to most kinds of knowledge. Facts are a specific kind of knowledge, and frankly, not that important. It's more important to know why the Civil War was fought than to know the exact dates and locations of each battle.

I've heard people say they wouldn't use ChatGPT even if there was "only a 1 in a billion chance that it made their bank account details public"...

May I introduce you to some very real probabilities:

https://oddsofdying.com

If you are living in America there is a 1 in 615 chance that your cause of death will be the result of an automobile accident.

So yes, it is unlikely that we will ever create a tool that can answer with 100% confidence. It is also unlikely that a manufacturing process will result in a widget that conforms to allowed tolerances 100% of the time.

However, in manufacturing this is understood. A defect of 3.4 per million widgets is considered an incredibly high process capability.

These tools are being made more reliable everyday. Please have a realistic goal in mind.

Edit: Well I've learned this much. Many of you are not Beyesians!

I asked it 3 questions about distributed systems, 2 of the answers were so fundamentally wrong that they would lead to a complete misunderstanding of the subjects, and if you tried to implement systems based on that understanding they would be broken in hard to detect ways.

The problem is that an LLM can only be as reliable as its training material.

For example I wrote a blog post on the 2 generals problem. In the comments there is more text from inexperienced people asserting that a solution exists than there is text from my original article.

An LLM trained on that article and comments will always be wrong.

LLMs can be trained on smaller vetted training sets sure, but they also currently require massive amounts of data and there’s no guarantee that just waiting a few months or years will improve reliability enough to fix these issues.

That's a very small sample size. Also, the scope of that question is rather broad.

There are a number of techniques you could try to improve your results.

But I get the distinct feeling that you are not interested in improving your results as you have a motivation to "prove" something about LLMs.

I have no strong interest either way. I played with chatgpt with no expectations going in, and I’ve played with it for many hours since. I have found it severely lacking for many purposes which people have said it’s suited. That’s it.

Yes, but you have to know to improve the results, and the techniques don't alway work. I asked for a table of sports results from a specific date in the past that's easy to lookup, and it gave me wrong results. But they looked plausible. When I asked it about specific result, it gave me a completely misleading answer, but one that would be plausible to someone who doesn't know the sport very well. After informing it of the wrong answer, it corrected itself by giving a different wrong answer. I tried that a couple times with the same result.

This becomes a problem when people start to treat LLMs as authoritative on facts.

Isn't part of the problem that an LLM will complete a statement with words that form a kind of statistical midpoint of the things that might come next. But when we're dealing with specific things in the real world, the true way to complete a sentence will not be that kind of statistical function, in fact it only reliably will follow from actually knowing the answer.

So, it's sort of like driving a car based not on a complete map of the area, but a partial map where the missing pieces are filled in from statistical averages of things that appear generally in the world-- but not necessarily in that location. We all may be impressed at how close the predictions come to reality, but the only truly reliable thing is that those predictions, measured with precision, will be wrong. And if you rely on them without verifying them yourself, you might have a serious problem.

I’m going to refrain from epistemological instruction but I would recommend you start with the Greeks and at least cover Kant and the analytical philosophers that followed.

The “true” way to answer a proposition is not even remotely as simple as you’ve described.

Well, yes, but ... the fact remains an LLM is not equipped to grapple with truths.

Or we could just deflate "truth" for this particular discussion, since what matters is whether predicting the next word works well enough without knowledge of the world. Pragmatically speaking, that's what matters.

Consider why chatgpt is banned on Stackoverflow despite the median quality already being poor there (because its answers/questions are even worse than that; the more answers from chatgpt, the worse the site)

The irony here is that I get more help from ChatGPT than Stack Overflow.

I suspect I'm better at using the tool than most people. So if most people are not good at using the tool and then submitting their poor results to Stack Overflow, I can see the problem.

Reply on news.ycombinator.com