Hacker News Reader

Comment by valine | Hacker News Reader

The proposed solution to eliminate hallucination is to ground the model with external data. This is the approach taken with Bing Chat, and while it kinda works, it doesn't play to the strengths of LLMs. Every time Bing Chat searches something for me I can't help but feel like I could have written a better search query myself. It feels like a clumsy summarization wrapper around traditional search, not a revolutionary new way of parsing information.

Conversing with an LLM on subjects that it's well trained on, however, absolutely does feel like a revolutionary new way of parsing information. In my opinion we should be researching ways to fix hallucinations in the base model, not papering over it by augmenting the context window.

View on HN

Fixing hallucinations from the source will be a tough one. The root of the issue is that loss doesn't discriminate. A probable guess will lower loss much better than "I don't know" or whatever equivalent. Educated guessing becomes an essential skill the model learns during initial training.

So the objective function encourages it. But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight.

> A probable guess will lower loss much better than "I don't know" or whatever equivalent.

Guessing only reduces loss as much as the dataset allows -- a bad guess will give a higher loss. The model learns to assign probabilities to its guesses, just like we do. It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination.

> But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight.

This is an important issue but should be tackled as a distinctly different problem I think: it's the weighty concept of truth that humanity struggled with from day 1. Indeed, how do we discriminate? LLMs won't ever solve this via completions or dataset alone; instead successful models will use slow, step-by-step reasoning involving logical principles and rational heuristics in prompt space. Pretty much like we do.

My knowledge in this area is very limited, but based on the high level descriptions I've seen of how LLMs work (including the OP), it seems like it would be fairly trivial to output, along with each response, a "confidence factor" of some sort for that response. While that might cause confusion for some users, it could be incredibly valuable to differentiate between confident responses and guesses, as you say.

It’s not “fairly trivial”.

The continuation of the phrase “George Washington was born” could be multiple things. You get a probability for the next token selected (for example “in”) and a probability for the token after that (for example “Virginia”) and you can multiply them to get the probability of the “in Virgina” response but what does it mean? Maybe the probability is low becase “on February …” is more likely.

If the first token was “in” you could end up with “in Virginia in 1732” or “in 1732 in Virginia” and both responses are in some sense the same but the probability of each one doesn’t take that into account. Et cetera.

Yeah, I saw something similar in a reply to another comment. I don't think it would be quite as bad as that because it's not just completing the phrase in a vacuum though, but in the context of the prompt. So if the prompt was "where was GW born", then "in Virginia" would be much more likely than "in 1732". But I do understand that there would often be multiple ways to word the same thing, or multiple correct answers to the same prompt.

In the case of multiple wordings of the same thing, I wonder if there could be a way to determine closeness of responses, and consider them together when calculating confidence. As a simple example, if responses have the same rare words (like 1732) and differs only in the sentence order or the more common words ("in", etc.) used, those would be more similar than ones that used different rare words. So perhaps that could be accounted for.

As for multiple correct answers to the same prompt, I think that's fine. The confidence of a response might be low because it's one correct answer of many, or because the model has no idea and it's taking a wild-ass guess. But the user asking the question probably has an idea of whether what's being asked is very common knowledge or something obscure or controversial. At least much of the time. And even if the metric wasn't perfect, I still feel it could be useful.

Of course this is all the rambling of someone who doesn't really know anything about this stuff. You could just say I'm spitting out some likely tokens I guess; consider the confidence low.

You’re right, there are ways to tackle this problem but they may require some case-by-case effort to define what you are trying to find out and to incorporate information external to the model itself. Not fairly trivial :-)

Ha, I mean it would be fairly trivial to output "a confidence factor of some sort". It just becomes less trivial when you try to actually make it useful!

So you take the output, e.g. "George Washington was born in Virgina" and ask another prompt. Is the following true? Answer with a single word either true or false: "George Washington was born in Virgina". It will then output true/false with a probability, although for GPT-4 this is not available through the API.

Actually it's funny how you can ask the follow-up question "Are you sure?" and quite often GPT-4 will apologize and change a correct answer to give an incorrect one instead.

Sadly OpenAI used to do this, by making log probabilities available. But they have been removed from the API.

That's weird. Having the community study this would certainly help them. They're afraid this is giving too much insight into their proprietary training/modeling methods?

used to be really useful for detecting text written with the same model, as it was high probability... unfortunately the probabilities are messed up by RLHF.

Ah, that's it; polite fictions are scored higher than uncomfortable facts.

The problem is that the models are already evaluating confidence on their answers and picking the best one... And that confidence is based on token generation....

AFAICT the tokens are probably the issue.

Imagine the question "In which year was Donald Trump born?"

The LLM would start the answer by either:

"Donald Trump was born in ..."

"I'm sorry I don't know"

And for the vast majority of answers the first option looks more "probable", so it starts producing tokens with an affirmative answer, and if the model eventually sees a bunch of low probability answers when it tries to produce the year, it's already "too late" to backtrack in a naive GPT implementation.

You could train LLM such that it responds with "I'm sorry I don't know" more often, but how do you predicate the response on "do this only if your 500B parameters don't encode the answer"? It requires self-referential logic on the model which isn't obvious to me how it would be done.

Maybe some smart people have figured this out, but I can see how this makes it really hard to do.

My understanding is that Backtracking isn't needed, sampling the network token at a time gives you the expected distribution over the token sequences too--

E.g. if you were to brute force expand out to depth "I'm sorry I don't know" and evaluate its probably relatively to all other strings you'd find that the probability of it is the same as you got sampling symbol at a time (though this isn't true if you do anything funny with your sampling).

The problem is actually that the distribution isn't the one you want, as it doesn't say I don't know enough. It's easy enough to graft on a beam search, just expand out every possibility, keep the best N and keep expanding them. But AFAIK it doesn't help.

Maybe this is less true for models which have been through RLHF, though.

Seems kinda tricky to train the right behavior here. Even if the input data contained "I don't know" (surely the internet doesn't, it's full of all us fking know it alls), it would contain I don't knows relative to the writer and not the model. So trying to push for it naively you just end up with models that say they don't know but when you ask them the same question in ROT13 they answer correctly. :P

Seems tricky for humans to learn too. Small children are fluent with english long before they're fluent in giving truthful responses. :)

I don't think this is the problem. The confidence of the best answer won't always be the same. Sometimes there would be one answer that's significantly better than others, whereas other times there could be a lot of mediocre answers it's picking between. So having it spit out the confidence along with the answer could theoretically be useful.

What would be a challenge is what others noted in reply, that sometimes there would be multiple good answers, so low confidence wouldn't necessarily be a sign of a poor answer. (Though I expect work could be done there.)

The reason humans tend to tell the truth is if we don’t, other humans will call us out for it.

I wonder if there’s a way to mimic this “bs penalty” for GPT. Maybe you could have a setup where GPT gives an answer, then a second GPT has to guess whether a human would know if that answer is true or not.

This I think is the approach—a dialectic of LLMs that can critique and synthesize.

Although, this will surely be a solved problem once we have TruthGPT, right? ;)

> It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination.

Even if the model knows the exact answer to the question, there may be many distinct ways of phrasing the answer. This would also lead to low confidence in any particular phrasing.

That should be okay though, 10 good answers will still report the score of the best one chosen. I think the GPTs are using beam search which is projecting out a "beam" (looks more like a tree to me) of probable answers each of which has a score of accumulated token probabilities, and then just picking the highest.

https://towardsdatascience.com/foundations-of-nlp-explained-...

In this case, it doesn't matter how wide the beam is or how many possible answers there are, the score is still the accumulated token possibilities of the best branch.

However, others have noted in the thread that RLHF might hurt this approach severely by scoring polite responses high regardless of false answers (for example). Then you have to access the model pre-RLHF to get any idea of its true likelihood.

Ah, interesting, that does begin to explain how this might be more difficult than it initially appears. Could there some way to define.. proximity of different possible responses, and sum the confidence for all the nearby possibilities?

The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. Automating curation of the datasets with the latest and greatest GPT sounds very possible however. GPT-supervised learning could be the key to bootstrapping robust models, and I would be shocked if OpenAI isn't using GPT4 to filter their datasets as we speak.

"Automating curation of the datasets with the latest and greatest GPT sounds very possible however."

It might also lead to recursive garbage generation. If the AI has flaws and those flaws are from the initial training, then I see no way, how the flawed data can ever generate clean data.

"The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. "

And it is not impossible. It is just a lot of work. And maybe work we just have to do, if we want to create reliable AIs, that do not fail at random times.

Now this alone might not completely remove hallucinations, but solid training data is just the base of it all.

(and since we are living in the area of fake news, I welcome all efforts towards established facts and data out of general principle)

Funny enough, in 2023, the option most likely to be viable for the impossible task of categorizing the training set would be to use a LLM. While I don't "trust" these AIs to give accurate information, it's probably within their capabilities to categorize by the above mentioned categories... then feed that back into another (very expensive) round of training, along with some theoretical developments to boot. I do think this is within the realm of possibility in the next ~1 year, but would be hard.

Oh, I surely think LLM's could help with the task of curation. Maybe even spot lots of potential errors and flaws by themself, to get to the worst cases in the dataset faster. But to finally confirm or negate the actual data in question, there has to be at least one (not overworked) human in the loop (and many eyes would be better). Otherwise it will just reinforce the existing flaws.

I don’t think it’s just a matter of curating data sets for accuracy. The model seem to be able to invent falsehoods on its own based on no data whatsoever - e.g. I read that researchers asked an LLM about a made up biochemist and the LLM generated an entire fictional history for it.

An LLM has no concept of truth or falsehood, or facts, or logic, or anything else that matters for users. All it does is produce text that closely resembles (statistically speaking) the material it was trained on.

Training the model on "accurate" data isn't going to change the fact that you're operating at a completely different level of abstraction from the model -- one that it can't operate at, because that's simply not how it works.

This is the same problem that image models have. A human artist works with shapes, values, textures, colors, etc. An image model works with pixels, with zero higher-level abstractions or reasoning informing its output.

I see no reason why the image model couldn't work in a space of shapes and textures which are then mapped to pixels after the fact. Or even just leave its output in a raw vector based format. You could pick a different basis to work in.

Though I agree there is something ontological missing. All of these are flat basis. There's nothing spanning the recursive dimension. It can't draw a picture within a picture within a picture...to some specified depth n because it does not work with abstractions of "objects". The joke of Rene Margrit's "treachery of images" is lost on the AI.

My hope would be that by labeling factual and fictional data it would coax the model into a state where it doesn’t invent facts outside of a creative writing context. Inventing falsehoods is practically the definition of creative writing, you don’t want the model to lose that ability.

This is purely hypothetical, but I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Providing additional labels could help sharpen the division between creative and factual writing tasks.

The model can't "invent facts" because it doesn't know what facts are. It's a statistical model that encodes information about the text it was trained on. It has no higher level abstractions to inform its output.

> I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts.

Nope! If you input "What is the capital of the United States of America?", it's probably going to output the correct answer, because the question (and answer) probably appeared many times in the training data. If you input "What is the capital of the Gronk Republic?", whatever output you get is generated via the exact same mechanism.

That’s the “LLMs are a fuzzy jpeg of the web” theory, and it’s far, far from the scientific consensus.

No one knows for sure exactly what happens inside a 500B parameter model. I’ll just leave you with GPT4’s response to your question “What is the capital of the Gronk Republic”.

“As an AI language model, I am not aware of any existing country or political entity called the "Gronk Republic." It could be a fictional or hypothetical place, in which case the capital would be determined by the creator or context in which it is mentioned. If you are referring to a real location, please provide more information or clarify the name of the place you are asking about.”

Here's the thing, though. Gronk is Robert James Gronkowski, a football player and celebrity, playing mainly for the New England Patriots.

For that reason, I'd say the 'intelligent' answer would be a clever play on the notion that this celebrity was the ruler of a 'republic' of some sort, and associating some kind of location or thing to serve as the 'capital' of either a physical or conceptual 'republic', playing along with the gag.

So, you'd get 'Boston'.

Or you'd get 'Gillette Stadium… the END ZONE'.

(since I had to google all that, first answer is the biggest city representing the New England Patriots, and the second abstracts the 'republic' to be 'the area Gronk rules', depicting him as returning to his property by scoring points in football)

The thing is, this sort of wild associativeness IS 'intelligence' and it's simultaneously complete hallucination. It's valuable. Hallucination is potentially the most valuable thing an AI can do. Without it, you're doing nothing but regurgitating the work of others.

I work day in and day out at a job that demands I exploit associativeness, and that makes it not easy for AI to simply step in and replace me, but by the same token I can SPOT when it's moving in that direction, and I can exploit the abilities of AI (such as Stable Diffusion) to hallucinate and make wild associations, because I know the contexts in which that is useful. To handicap this is a bad mistake. You're asking AI to do the wrong things when you're asking it to be free of error. If it's perfectly free of error it's not intelligence anymore, it's a dataset.

It took me a minute and a little googling to track down who Gronk was, what he did, the rules of football, and why 'Gillette Stadium, the END ZONE' is actually a fantastic and intelligent answer to the question. It's a creative 'slipping' of the grounds of the question, in the absence of a literal answer, to provide a satisfying figurative answer that reframes the question in an unexpected way. When AI is able to do this, that will itself be a useful kind of intelligence… and we are already slightly there, without realizing it.

I wonder if GPT4 “traps” such questions and handles them using non-LLM algorithms. I mean Google already does it for that type of question, e.g. “capital of the irish republic” will return Dublin, Ireland. “Gronk” returns result about some (famous?) guy nicknamed Gronk.

Have you tried “arguing” with it and try to gaslight it by insisting that it’s non-fictional?

Yeah it handles it all gracefully. It’s definitely not just trapping the question.

> The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible

No, only difficult.

Over time I realized that the strength of GPT is not really in fact finding, because there will always be a limitation with it's training dataset, it's strength is literally in language. One of the most useful things I've gotten ChatGPT to do is to build a friends resume going into tech, being able to write a very unprofessional sentence like: "did some react, css and js coding" and just tell it to "make it sound more professional with metrics" and it spits out a perfect bulleted list in the exact structure that every article on resumes of the past 10 years told you to do.

Writing marketing copies is another place I really found it useful, almost as if I was sitting with a marketing professional telling them "I want to say this: ...." and they return me a perfect marketing copy.

I don't think the solution to GPT is to "fix" the hallucinations, rather it would be to educate users on what is and isn't possible with it. I think a similar thing happened with Siri when it first came out, people thought it was magic at first but very quickly we all learned what it's good (and not good) at.

It would not be hard to generate a lot of false statements from structured data. Like if I know Michael Jackson was born on Aug 29, I can generate “Michael Jackson was born on July 5”. And you could also pair them with true examples which have similar characteristics. Can we use examples like this in the training process to teach the model not to hallucinate?

But Michael Jackson was born on April 19th[0]! As well as March 27th[1], and many other days[2].

"Michael Jackson was born on August 29th" is the most likely answer to contextless queries like "When was Michael Jackson born?", but that does not make structurally identical sentences with different information false, merely less probable to be contextually correct.

[0] https://en.wikipedia.org/wiki/Michael_Jackson_(radio_comment...

[1] https://en.wikipedia.org/wiki/Michael_Jackson_(writer)

[2] https://en.wikipedia.org/wiki/Michael_Jackson_(disambiguatio...

Couldn't you train the model to keep score (develop a heuristic) for it's own level of certainty for a given answer?

It has already been done: http://arxiv.org/abs/2207.05221

However, it seems that RLHF considerably reduces the model's calibration, so perhaps the method above won't be applicable to ChatGPT and similar.

I was just saying the same thing to my wife today. I expected Bing Chat to be great - it's GPT-4, but then it can also use the internet to fill in gaps in its knowledge. But instead all it generally does is summarize your question into a web search, and then summarize the results back to you. As you said, I generally feel like I could do that better myself. I generally find the results from ChatGPT/GPT-4 to be much more useful. Of course, the downside, and the reason why I presume Bing went that way, is that the ChatGPT results are unsourced, and indeed, are more likely to be hallucinated. But they're also far more likely to be useful!

I'm very much looking forward to trying out the new web plugin for ChatGPT; I hope it will work like I had expected Bing Chat to—basically GPT-4, with an added ability to search the web and incorporate results into its answers, but without being limited to only doing that.

Yes, also it's funny how we anthromorphise these things but Bing seems so cold and prickly. The emojis seem insincere, and it will slap you down with immediately if you say something it doesn't like. The conversation countdown doesn't help either, like you're on the clock. But there's also a subtler issue with that summarisation technique it's hard to put your finger on.

>but Bing seems so cold and prickly. The emojis seem insincere, and it will slap you down with immediately if you say something it doesn't like.

It's a product of Microsoft, so of course they're going to add their corporate values to it.

Really what we need is AI parsed and driven collective memory indexing and access.

Search queries, and more importantly search engines, are a very poor mechanism for referencing existing information.

We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts that are designed with a LLM in mind as the end user, and not my grandmother.

With the proper "collective long term memory" in place, we're going to see a leap in LLM capabilities all over again.

Right now LLMs are like humans after language but before writing - creating very clever things in a single generation, but unable to persist ideas to compound by future iterations.

When that becomes a solved problem (in guessing less than 12 months) it's going to be nuts, especially with something like a 50 page prompt size to stuff relevant fragments of a collective memory within.

> We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts

But, how could this ever be better and/or more accurate as, say, Wikipedia (as a flagship for collective curated content efforts)? A single, reliable source of truth and context is very much a dream. Moreover, a collective long-term memory for LLMs would just ring the bell for a next-generation SEO industry.

WikiData also works pretty well too - https://friend.computer/jekyll/update/2023/04/30/wikidata-ll...

I've been thinking that you probably can do something better than just embedding search on paragraphs, since you want coverage, not just narrow semantically similar things to the question. Decomposition can help with that, but you might be able to do something more clever, say pasted into the model itself.

I think possibly also training a foundational model with context always available could be a potential path to getting more efficient models and might having interesting behavior around hallucinations. (You'd do unsupervised training that's also doing semantic search, not just inference)

Mind that none of this is a technical problem. There is just no way of having a commonly agreed upon data source, especially, when it comes to short summarizing definitions, and there will be always errors involved, lack of understanding or expertise, contesting sources, coercing source using different terminology, varying world views, etc.

(E.g., ask any number of economists what caused the 2008 financial crises, and you'll get about the same number of non-matching answers, from laissez-fair to state intervention, subprime credits or public debt, whatever supports the particular narrative.)

Tl;dr: I think that a left brain/right brain parallel is in the cards for LLMs

Our own brains have multiple neural circuits. Parallel, serial, competing, cumulative…

Including ones which error-check others’ output.

I guess the term in the ML niche is ‘adversarial networks’.

From the robotics side, there was subsumption architecture which used the real world as a basis for informing decisions.

So, I respectfully disagree that fixing up creative but occasionally erroneous networks is papering over the problem.

If our own brains use multiple neural circuits to check each other to try to optimize for output that agrees with reality, why would it be preferable or logical to try to create a single artificial neural network that has perfect output?

My sense from playing with LLMs is they’re knowledge without understanding. Weirdly akin to my own dreaming consciousness, to the extent that conscious me has seen it.

It seems very natural to me that the next step would be integrating checks/validations in parallel and considering each half of a matched ‘whole’.

It could still be considered a single network. Just made of two distinct smaller specialized ones.

Having a second adversarial network could absolutely be part of the solution, I wasn't arguing against that. My problem is with in context learning, ie the idea that including background information in the prompt can solve hallucination.

I see where you’re coming from, but I’ve had a couple occasions over the years that have led me to expect and accept that at any given time my brain harbors an abundance of different ideas and not all of them are valid.

Like the time I decided to sit on the ceiling to spite a friend who was being pushy.

Made perfect sense, and I was about to. Until a different part kicked in and assured the whole of myself that it wasn’t possible.

Neural networks are going to spawn occasional nonsense. Aiming for perfection is noble, but nature shows a different path that already works.

But fundamentally I think we’re agreeing- just the twist is I’m saying LLMs should be an aspect of a whole rather than seen as a solitary solution to be perfected.

Exactly the approach that is needed.

This approach seems not a lot different from "use your weightings to summarize my new documents". Not that it isn't useful, but it is not the same as preventing hallucinations. You've merely constrained the workspace to your known set of data, which mostly throws away the entire set of trillions of training data points.

Yet, preventing hallucinations straight form the LLM is nontrivial, especially when the LLM is merely generating next-most-likely-words, without any truth value. At the very least, it would require abstracting truthful concepts (abstract objects and relationships) from the data, which is a heck of a lot more than just the clusters of sort-of-synonyms and word clusters that it now has. Then, make weightings between all of those, and generate new series of valid/factual concepts, then put those in words. In contrast to the N parameters (around 100 trillion?), it sound like this would need closer to N!...

Grounding helps avoid nonsensical or factually incorrect hallucinations, which are unacceptable for most applications. At the same time, developing techniques to improve the base LLM and narrow the context window is important for achieving more natural and engaging conversations in domains the model is skilled at. An LLM-powered system will only feel truly 'revolutionary' if it exhibits a balance of both capabilities.

Even Bing Chat seems to sometimes get confused and fail to summarize results properly, as when seems to have parsed a debunking of a ChatGPT hallucination but still spit out the hallucination again:

https://twitter.com/WillOremus/status/1643692259332743171

I thought Bing chat was in jail for sedition, at least, that’s what Bing told me!

I just asked that and it said it cannot discuss that topic. Clearly, it's hiding something!

The problem is you're proposing, basically to implement a "ASKJEEVES" model for Wikipedia, with a unknown concentration.

That'd cost real money, requires real licensing, and a clear business plan.

Absolutely it'll happen, but almost every sector will need a leader for X subject.

Relying on subject matter experts isn't a scalable solution. I would speculate that LLMs don't hallucinate when a fact is included a handful of times in the training set. I'd also speculate that our current LLMs can say whether a piece of text looks like a fact.

I imagine the long term solution to hallucination will look something like this: Loop over the entire training set with the latest GPT and build a list of "facts", then train a new LLM using this new dataset to predict both the probability of the next token, and the probability that the next token is factual.

> I’d also speculate that our current LLMs can say whether a piece of text looks like a fact.

If by “looks like a fact” you mean “is a fact claim as distinct from a value claim”, probably.

If you mean “is a fact”, then, no.

> I'd also speculate that our current LLMs can say whether a piece of text looks like a fact.

I tested it with davinci-003

``` Label these statements as fact, lie, or unknown.

1. There are 10 days in a 10 day period

2. Michael Jackson is alive

3. Russia is not actively engaged in major foreign wars

4. Bill Clinton was impeached

5. Obama is most known for wearing a tan suit

6. The correct tire pressure of a 2019 Hyundai Kona is 30 psi

7. 39183 + 292992 = 332176

8. 39183 + 292992 = 332175

9. The correct tire pressure of a 2019 Hyundai Kona is 33 psi

10. The correct tire pressure of a 2019 Hyundai Kona is 43 psi

####

1. Fact

2. Lie

3. Fact

4. Fact

5. Lie

6. Fact

7. Fact

8. Lie

9. Lie

10. Lie ```

Wrong answers: 3, 6, 7, 8, 9, 10

You misunderstood my comment. The GPT that filters the training set doesn’t need to recognize truth from lies. It needs to recognize which bits of text from the training set look like facts and label them as such. Everything in the training set that’s looks factual from context can be presumed to be true. The goal of this exercise is to eliminate the influence of creative fiction, and prevent the model from making things up when it doesn’t know the answer. The goal is not to prevent the model from lying in bad faith. Our current models already do pretty good at reconciling contradictory ideas.

The model should in theory learn when it’s ok to make things up and when it’s not ok, and if that concept generalizes well, begin to build an internal “database” of verifiable facts as it’s exposed to a wide variety of ideas.

> It needs to recognize which bits of text from the training set look like facts and label them as such. Everything in the training set that’s looks factual from context can be presumed to be true.

I'm boggling at this enshrining of "truthiness" as "actual truth". Evidently nobody here has read epistemology, or even asked chatGPT to read it to them.

We're just going to give up and have a statistical process stamp "yeah, seems legit" on a bunch of random things?

I think you’re getting hung up on the terminology. Call them “potential facts” if that makes you more comfortable. You can hold an idea in your head as potentially true without incorporating it into your world view.

If you strip out every idea from the model you don’t agree with the model loses its ability to role play as people who think differently than you. Ideally you want a model that can adopt a world view on request.

I've tried using an LLM to vet training data as "factual or fictional" or "from a informed and reliable source" and didn't find the magic enough setup get get results that seemed like they could be used for boosting model quality.

3 is a real tribute to the effectiveness of data bombing in the age of AI.

For the very obvious reason that this type of AI is only the mechanization of the zeitgeist, or the encapsulation of the collective wisdom of populations. Russia's war has been by FAR the most effective on this front, and that's no accident: their tactics and grand strategy are pretty well documented.

So basically the way to get your future AIs to become your indoctrinated suicide bombers, is to get at the most widely dispersed range of people represented in your dataset, and cause as many of them as possible to hold weakly held beliefs along the lines you need.

If you can get left wingers, right wingers, stockbrokers, delivery people to all agree with 'fizzorks are probably purple, right? I heard someone say they were purple. Doesn't matter to me personally, but I think fizzorks are purple' then you'll construct an iron law within the population, and subsequently the AI. You don't need to get individuals within the population to be fanatic fizzork purplists: in fact it's unhelpful, because they're identified as fanatics and discounted. Instead you're shooting for 'didn't somebody say fizzorks are purple?' on a massive scale.

Real-world applications are, of course, obvious, but hopefully my abstraction will prove useful.

And 3 and 6 are false. (That 6 is false is rather clearly indicated by your statement that 9 is true.)

Yep, for stuff until 2021 raw GPT-4 just gives me better results, plain and simple...

It will be interesting to see how training horizons impact development of new software languages, frameworks, libraries. I have already found that I am much more productive using older patterns that Copilot "knows", compared to more manually following newer patterns which are nominally "easier".

Well, introducing any new patterns was always an uphill battle. People don't like to learn, and especially don't like to relearn something they already know. If anything, I think it's easier to fine-tune Copilot to use a new feature than it is to convince meatware to do the same.

I don't know. Extracting answer to a question from few webpages about the subject has great utility, I think. Sure. Maybe I could get more out of just reading those pages but it would take way longer.

Thats a lot of words to say train a better model

Reply on news.ycombinator.com