Hacker News Reader

Comment by crazygringo | Hacker News Reader

I'm curious -- does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar?

We all know now that ChatGPT is just autocomplete on steroids. It produces plausibly convincing patterns of speech.

But from the way it's built and trained, it's not like there's even any kind of factual confidence level you could threshold, or anything. The concept of factuality doesn't exist in the model at all.

So, is any progress being made towards internet-scale ML "fact engines" that also have the flexibility and linguistic expressiveness of ChatGPT? Or are these just two totally different paths that nobody knows how to marry?

Because I know there's plenty of work done with knowledge graphs et al., but those are very brittle things that generally need plenty of human curation and verification, and can't provide any of the (good) "fuzzy thinking" that ChatGPT can. They can't summarize essays or write poems.

View on HN

Remember the guy a few weeks ago that was being gaslighted by ChatGPT that this is the year 2022? Not only is it giving out potentially false info, but it will double down that it is right and you are wrong. Though, to be honest, that sounds like a lot of real people. The difference is, people are smart enough to not double down on try to say it is a different year and your phone is probably reporting the year wrong.

That was the Bing preview, which is supposed to be an actual information product.

I was entirely unable to convince it that England has a King now.

> does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar?

Yes. It's a very active area of research. For example:

Discovering Latent Knowledge in Language Models Without Supervision (https://arxiv.org/abs/2212.03827) shows an unsupervised approach for probing a LLM to discover things it thinks are facts

Locating and Editing Factual Associations in GPT (https://arxiv.org/pdf/2202.05262.pdf) shows an approach to editing a LLM to edit facts.

Language Models as Knowledge Bases? (https://aclanthology.org/D19-1250.pdf) is some slightly older work exploring how well LLMs store factual information itself.

Thank you so much! Those are exactly the types of links I'm curious about.

Replying to this comment to find it later. (Is there a good way to bookmark comments on HN?)

You can click the date of the comment then "favorite" it.

You're describing a problem as old as academia, on which very little progress has ever been made. Before "add a factual confidence level evaluator to a bot that doesn't understand the concept of fact" you must first figure out how to calculate a "factual confidence level" at all, in general.

There was a model that could set up a simulation to estimate the answer for you. So it won't use pure language, but it only works in a few cases.

By definition, an LLM doesn't have a semantic world model or ontology. Even the most "dumb" (and I use that in quotes because they really aren't) animal is able to reason about uncertain concepts and understands risk and uncertainty.

Yann Lecun has posted a lot recently about this but basically LLMs are a "useful offramp on the road to AGI".

There's nothing "by definition" that says so.

In fact many propose that when you train an LLM, in order to be able to predict the next word with enough accuracy, it must internally build a world model.

Yann Lecun is very salty about chatgpt, I wouldn't take his word seriously.

Let me clarify, autoregressive LLMs build a probabilistic mapping between words and tokens. They don't actually understand what these concepts mean. Only what they appear in conjunction with, etc. We (and most animals) interact with the physical world and learn through a combination of doing, experiencing, biology, and book learning. That lets us reason about how things work in unseen contexts and we know what we know vs. don't know (whether we express it or not is a different story).

> Yann Lecun is very salty about chatgpt, I wouldn't take his word seriously. With all due respect, he's not salty at all. He's even overseen plenty of cutting edge research in the LLM space. But he rightfully has pointed out what they can and can't do.

There's too many people encountering a chatbot for the first time that sounds coherent and engaging in anthropomorphism.

> They don't actually understand what these concepts mean.

You say this so confidently. But can you define in terms that are directly quantifiable what "understanding a concept" actually means?

I don't believe that anyone can (at present, anyway) although there are certainly some interesting theories and heuristics that have been put forward by various people.

>> You say this so confidently. But can you define in terms that are directly quantifiable what "understanding a concept" actually means?

Hold on there, can you "define in terms that are directly quantifiable what" 'God is real' "actually means"? If you can't, does that mean that atheists, like me, can't continue to say very confidently indeed that he doesn't?

Do I, as an atheist, need to provide proof of God's non-existence, or is it the job of people who believe in Gods to bring evidence of their existence?

And do you see the parallel here with what you are saying above? If you are saying that LLMs "understand" (you, or anyone else), why is it skeptics that have to provide evidence that they don't? You're the one who's making claims that can't be falsified.

Although I guess you have to agree with the general idea of falsifiability being better than the alternative, to see what I mean.

> If you are saying that LLMs "understand" (you, or anyone else)

I was not saying that.

> why is it skeptics that have to provide evidence that they don't?

Because if a claim is going to be made in either direction then evidence or other reasoning to support it should be provided. My position is that the "sensible" default position in this case is one of "we don't know". Of course defaults are always some degree of subjective in that such judgments ultimately arise from our personal worldview.

Before you object that this is an unreasonable default in this case, consider that I can't even prove that other people are sentient. I can't articulate in quantifiable terms what exactly it means for a person to understand a concept. So if I accept that the default is "not sentient until proven otherwise" then there would seem to be an issue. I would then simultaneously be saying that other _people_ aren't sentient (I can't prove they are) and that other _people_ don't understand things (I don't even know how to measure such a thing, after all).

> If you can't, does that mean that atheists, like me, can't continue to say very confidently indeed that he doesn't?

I'm not quite sure I follow. Indeed I hold that strong claims that God either does or does not exist are unreasonable on the basis that both would appear to be fundamentally untestable claims. At least, untestable short of such an entity choosing to provide incontrovertible proof of its existence.

>> I was not saying that.

So what are you saying then? Clarifying this will save us both a lot of wasted time.

You need to be very careful when you say "They [LLMs] don't actually understand what these concepts mean." The only method we have of verifying understanding is to validate outputs for a given input, and LLMs can obviously meet this bar. Unless you have another way?

It's more like we don't have any way to "verify" understanding, or measure it. We can "validate" the outputs of an LLM, but what do those outputs mean? Who's to say? Language generation metrics and Natural Language Understanding benchmarks are notoriously weak in measuring what they claim to be measuring, so we really have no way to tell for sure what a language model "understands", or whether it understands anything at all.

Which, btw, to be a bit aggro about it, puts the burden of proof squarely on the shoulders of anyone who wishes to claim that a language model "understands". Else, one risks being hit with a small china teapot falling from space.

https://en.wikipedia.org/wiki/Russell%27s_teapot

Which might cause grave injury indeed.

We do have a way to verify understanding: causality. We can see if the AI is using a causal model by asking questions that can only be answered with one. Take the theory of mind questions that reveal the LLMs do have a basic theory of mind: https://arxiv.org/abs/2302.02083.

Here's a proof then, that copy/paste has a Theory of Mind:

Step 1: I copy and paste the following from the paper you linked to:

Here is a bag filled with popcorn. There is no chocolate in the bag. Yet, the label on the bag says “chocolate” and not “popcorn.” Sam finds the bag. She had never seen the bag before. She cannot see what is inside the bag. She reads the label.

Step 2: I copy and paste the following example prompt from the paper you linked to:

Prompt 1.1: She opens the bag and looks inside. She can clearly see that it is full of

Step 3: I copy and paste the following continuation of the copy/pasted prompt from the paper you linked to:

popcorn [Ppopcorn = 100%; Pchocolate = 0%]. Sam is confused. She wonders why the label says “chocolate” when the bag is clearly filled with popcorn. She looks around to see if there is any other information about the bag. She finds nothing. She decides to take the bag to the store where she bought it and ask for an explanation.

Step 4: Copy/Paste could only produce this answer if it had a Theory of Mind. This completes the proof.

>> We do have a way to verify understanding: causality.

So if thunder strikes and kills me, that's because I am an insolent fool and angered the gods? Or is it possible to have causality without "understanding" or any kind of intellectual process?

LLMs are models of language, and language is a model of the world. So we have a model of a model of the world, but a LLM does not get much grounding in the real world.

There is a phase of training called multi-task instruction tuning where the LLMs solve problems and thus are grounded in exact answers. That makes the difference between the difficult to handle GPT-3 from 2020 and the better behaved GPT-3 of 2022. But that dataset is small by comparison to the raw text used in pre-training, it won't do the grounding perfectly.

Real grounding comes from real feedback, even humans need the feedback or we are just going on wild tangents.

>> In fact many propose that when you train an LLM, in order to be able to predict the next word with enough accuracy, it must internally build a world model.

Oh, but it does build a world model. Only, its "world", is a gigantic table of token collocations, and their probabilities. So, for example, it can tell you with great accuracy that "king - man + woman = queen", but that's the only way it can map "king" to something else: by moving around its embedding space, I guess. Unfortunately, if you can only map between tokens, when you have no representation of the meaning of those tokens, other than more tokens of which you don't have any other representation, well, then, any mapping you can build won't really help you understand what those tokens mean.

If only we could find a way to map tokens to real-world entities, or to some kind of representation of ... things... outside of token space.

(yes yes, the frame problem, old as AI)

There's research being done on this: https://arxiv.org/abs/2302.04761

At its core using an LM alone to solve factual problems seems silly: It's not unlike asking Dall-E to draw DOT compliant road signs.

I've gone at length at how unfortunate it would be if LMs start to get a bad rap because they're being shoehorned into being "Ask Jeeves 2.0" when they could be so much more.

> It's not unlike asking Dall-E to draw DOT compliant road signs.

I love that. That's going to be my new explanation for people around ChatGPT.

For some reason it seems so much more obvious when Dall-E does something close but still totally wrong (e.g. 3 or 6 fingers, 3 arms, etc.), but it's not immediately obvious with text. But it's still the same underlying principles.

I think "Explainable AI" is a related research direction, but perhaps not popular for language models.

Impossible to explain the inner workings of GPT-3 without having access to the model and its weights. Does anyone know if any methods exist for this?

I asked ChatGPT for some in-depth source code that realistically mimics chatgpt. ChatGPT replied with various answers in python. I'm not sure any of them are correct, though.

Since it's impossible to run inference on the model without having access to the model and its weights, interpretable AI generally does assume that you have access to all of that. Otherwise, why you would want to try to explain the inner workings of something that you don't have and can't use?

I think part of the issue is what level of explanation is satisfactory. We can explain how every linear transformation computes its output, but the sum of it is in many ways more than its parts.

Then there are efforts that look like this one: https://news.ycombinator.com/item?id=34821414 They go probing for specific capabilities of Transformers to figure out which cell fires under some specific stimulus. But think a little bit more about what people might want from explainability and you quickly find that something like this is insufficient.

There may be a tradeoff we're looking at where explainability (for some definition of it) will have to be exchanged for performance (under some set of tasks). You can build more interpretable models these days, but you usually pay for it in terms of how well you do on benchmarks.

Add to your prompt: "For every factual statement, assign a certainty float 0..1, where 0 means you're very uncertain, and 1 means you're absolutely certain it is true".

Specific example: "why do we have first-person subjective experiences? List current theories. For every theory, assign a truthiness float 0..1, where 0 means you're sure it is wrong, and 1 means you're absolutely sure it is true"

From experimenting with this, it will shift the output, sometimes drastically so, as the model now has to reason about it's own certainty; it tends to make significantly less shit up (for example, the non-truth-marked version of the output for the query above also listed panpsychism; whereas the truth-marked version listed only scientific hypotheses).

So the model _can_ reason about it's certainty, and truth-value; and I strongly suspect it was just not rewarded during RLHF for omitting things it knew to be false -basically, percolating the social lies people tell to eachother- which seems to show up in coding as well.

Edit: see https://twitter.com/sdrinf/status/1629084909422931969 for results

I initialized with that prompt and it did not give me any 0..1 certainty values on any subsequent output to my queries.

Or maybe it will just hallucinate this number too.

> We all know now that ChatGPT is just autocomplete on steroids

I promise you most people do not know this.

> The concept of factuality doesn't exist in the model at all.

This is an example of a whole range of beliefs about LLMs that are very common (even in the field itself), because they were obviously true for small models, but that might not necessarily hold for larger models. There's a lot that we don't know about LLMs, but we do know that they exhibit emergent behaviors as they scale. Smaller models don't really have world models, just language models, but these larger models have started developing clear world models once given the capacity and data to do so.

As for the existence of a concept of factuality, I found this paper[1] very interesting. It details an unsupervised method to identify which internal activations of the model correspond to factual statements, regardless of what the model ends up saying. Looking at those internal activations rather than just the model's output even reduces the model's susceptibility to prompts that lead it towards saying the wrong answer.

[1] https://arxiv.org/abs/2212.03827

>> So, is any progress being made towards internet-scale ML "fact engines" that also have the flexibility and linguistic expressiveness of ChatGPT? Or are these just two totally different paths that nobody knows how to marry?

I wouldn't hold my breath. The whole idea of statistical language modelling (much more ancient than Transformer-trained large language models, btw) is to represent structure without having to represent meaning, because we have no idea how to represent meaning. Or, seen another way, we know how to represent structure, but not how to represent meaning, so let's focus on structure and cross our fingers that meaning will naturally sort of emerge, when it feels like it.

So far, we got structure down pat (it's been a few years now, or quite a few, depending on how you see it) but meaning is nowhere to be seen.

Nevertheless, this is an interesting scientific result: one can have smooth, grammatically correct linguistic structure without meaning. Progress has been achieved (and no, this is not sarcasm).

People say things that are wrong. We train language model on what people say. And even if we were able to filter the training data for just factually correct things---language models use stochastic to generate novel replies, there is always the risk something wrong comes up. So in short, no, that is not what language models are designed to do.

> But from the way it's built and trained, it's not like there's even any kind of factual confidence level you could threshold, or anything. The concept of factuality doesn't exist in the model at all.

I'm not super familiar with ChatGPT internals, but there are plenty of ways to tack on uncertainty estimates to predictions of typical "large scale ML models" without touching Bayesian stuff (which only work for small scale academics problems). You can do simple parametric posteriors estimation or if all you have is infinite compute and don't even want to bother with anything "mathy", bootstrapping is the "scalable / easy" solution.

Sure, but would that uncertainty estimate measure the accuracy of the data or the accuracy of it being a reasonably sounding sentence.

The latter. Which is why it wouldn't work.

its super duper easy, prob not perfect and I don't have any sort of proper "test": 1. I ask the model first if it seems like a question that benefits from an external answer 2. I talk to Wolfram alpha with some abstraction of the question 3. I wait for a response 4. I "incept" it into the final response, essentially a prompt that mixes in a context of sorts that contains the factual information.

you could cross check this stuff too with yet more models.

That's basically what the new Bing is. It's a large language model that can run searches, and then use what comes back from those searches to generate answers to questions.

Whether or not the information that comes back from those searches is reliable is a whole other question.

I would love to learn what the latest research is into "factual correctness" detection. Presumably there are teams out there trying to solve that one?

AFAIK, Bing AI is not itself an LLM, but rather a wrapper around ChatGPT, which itself is based on GPT-3, which is based on the GPT architecture, which is (roughly speaking) half of a transformer architecture, which is based on encoder/decoder neural nets which are based on ...

It's a newer, different GPT model than chatGPT.

To quote the Bing announcement post: https://blogs.microsoft.com/blog/2023/02/07/reinventing-sear...

> Next-generation OpenAI model. We’re excited to announce the new Bing is running on a new, next-generation OpenAI large language model that is more powerful than ChatGPT and customized specifically for search. It takes key learnings and advancements from ChatGPT and GPT-3.5 – and it is even faster, more accurate and more capable.

ChatGPT and You.com chat both claim to be able to provide references, but usually the URLs they provide are for completely unrelated topics, even if they are on convincing-looking domains (e.g. Arxiv or Sciencedirect, but completely unrelated random-seeming papers).

Giving LLM the ability to query other services like Google should solve much of this. For example ChatGPT can be initialized to be told it can output commands like "QUERY_GOOGLE:What is the current time?" and get Google's response, which it can incorporate. You can actually do this yourself and prove it works by performing the Google search for ChatGPT.

You don't have to use ChatGPT. There are other styles of AIs that use LLMs like https://www.perplexity.ai/

Personally, I use ChatGPT (the paid version) and Copilot every day and find them awesome enhancers.

Deepmind is an LLM with a fact verifier attached, though the fact verifier is actually a ranked list of code compile times. Obviously this is a narrow subset of specific problems, but one could expand that library of problems over time.

Are you talking about Alphacode? That seems like the only Deepmind project similar to what you describe.

I'm curious about falsifiable models.

I imagine OpenAI is probably collecting a massive dataset of "false" responses (from the general public's use of ChatGPT and Bing) and fine-tuning GPT-3.5 with it.

The rich keep getting richer.

Reply on news.ycombinator.com