Hacker News Reader

Comment by indigochill | Hacker News Reader

> LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question".

Isn't the hint in the name? It's a language model, not a knowledge model. LLMs are exceptional at generating stuff that passes as coherent language (like, the parts of speech are all where one would expect them to be in the sentences). The trouble is people think it goes deeper when the knowledge modeling part only happens incidentally because language and knowledge modeling are so closely related.

View on HN

No, you're overindexing on the name. It doesn't mean that these LLMs are about linguistics and grammar specifically. Clearly, they aren't as likely to generate "Colorless green ideas sleep furiously" as "It's important to feed your dog".

Obviously, they're better at grammar than many other things, but it's clear that they contain a vast amount of knowledge. They contain the information that the sky is blue and firetrucks are red, which isn't a matter of grammar or coherence, but is simply information about the world.

> It doesn't mean that these LLMs are about linguistics and grammar specifically.

Correct. But it does mean they're about "language" rather than knowledge. It's meant to read, parse and generate language, no matter if it includes prose, facts, knowledge or whatever.

> Clearly, they aren't as likely to generate "Colorless green ideas sleep furiously" as "It's important to feed your dog".

Clearly, as the training set probably contains much more instances of the latter, than the first example. And if it's more probable to generate, that's the text it'll generate. It doesn't understand why or how, just that it's more likely, so that's the way it's going.

> it does mean they're about "language" rather than knowledge

Not really. It was invented because they were trained on natural language corpuses. It wasn't a deep, philosophical concept.

It continues to be used for models that are doing completely non-languagy stuff, such as robotics, although I expect the growing use of 'foundation model' or some other less-confusing term will likely replace it.

Language is about knowledge.

If it wasn't, what would we be doing here?

> But it does mean they're about "language" rather than knowledge.

To some extent, those are kind of the same thing.

> To some extent, those are kind of the same thing.

They are not the same thing. They are not even "kind of" the same thing, for any reasonable definition of "kind of." That should be obvious, because a lie is language, which is pretty close to the opposite of knowledge.

I mean no offense, but your hedging words are doing so much work that I think your statement counts as bullshit.

Edit: thinking about it more, I think your comment shows the particular kind of bullshit that often characterizes tech people's thinking and is so useful for hyping shit (e.g. take something hard, lossily reformulate it into something easier but vaguely similar, solve the easy thing, then declare you solved the hard thing).

A language is a particular way of serializing concepts and their relationships.

Knowledge is... concepts and their relationships, typically with the implication of being grounded in objective reality (although one can be knowledgeable about things like, say, Tolkien, or Star Wars, or fantasy tropes more generally).

Language is (a particular way of representing) knowledge, in about the same way that a particular .cpp file is (a particular way of representing) a linked list or whatever.

> Language is (a particular way of representing) knowledge, in about the same way that a particular .cpp file is (a particular way of representing) a linked list or whatever.

So, in your view, are .cpp files and linked lists "kind of" the same thing?

Language can be used to express some kinds of knowledge, but they're not even "kind of" synonymous. Language can express a lot of things that aren't knowledge, and there's knowledge that can't be expressed with language (qualia, at least).

Of course they are the same thing. Language as humans use it isn't a god-given perfect formal grammar system. It's people grunting and gulping about what they experience, for many thousand of years, each generation picking up the patterns from the generation before, and adding to them. Language was, is, and always has been about knowledge, because you literally build the language skill up from correlations.

All the more and less formal models we invent for describing "language"? This is just fitting curves to a snapshot of reality. So sure, "a lie is language, which is pretty close to the opposite of knowledge" - a lie can be expressed in the formal model. But the truth (as subjectively understood) is much more likely to be spoken or written down in practice.

Communication isn't random. Language isn't random. Learning it is picking up the patterns.

I'm having trouble understanding why you'd respond with that. Are you interpreting knowledge to mean knowledge-of-language or something?

>>>> But it does mean they're about "language" rather than knowledge.

>>> To some extent, those are kind of the same thing.

>> They are not the same thing.

> Of course they are the same thing.

Language and knowledge (both broadly construed) are obviously not the same thing. Language can encode non-knowledge like nonsense or lies, and there is knowledge (e.g. of the experience of qualia) that can't be expressed in language.

I think the point up-thread is true: something that knows only about how language is unreliable source of knowledge. Even if all its input is true knowledge, it can still blindly combine that input in linguistically plausible but false ways.

they really are not.

you can use language as a proxy for knowledge in some cases, but doing so guarantees generating bullshit, which is the entire point of the OP.

    it does mean they're about "language" rather than knowledge

Language in part encodes knowledge.

It also encodes opinions, fantasy, bad-faith arguments and lies without discrimination. Given that LLMs aren't trained to separate their sources, I think it's fair to say today's AI focuses more on language than the integrity of it's answer.

Agreed, discriminating what language contains is the issue.

I would argue they do not contain knowledge using the classic definition of knowledge as "justified true belief."

They can emit statements that happen to be true because the probabilities fall that way (as a sibling comment points out.) But it's not "justified", because models don't have any ability to remember or cite sources, or even internally classify tokens as true or false (it's all just weights.)

(And it's not a "belief", either, because there's no intentionality, but that's a more slippery concept.)

Therefore, philosophically speaking, it is fundamentally impossible to acquire knowledge from a LLM unless you verify ("justify") a statement via some secondary, more trustworthy channel.

I tried to mention containing knowledge to avoid a discussion like this: not 'knowing' or 'having knowledge' or something that could lead to a discussion like this, which I don't think aligns with the point I was trying to make, which is that top LLMs are not merely good at things like grammar, but that a key part of what they do has to do about information about the world.

FWIW, although 'justified true belief' is classic in the sense of being an old definition of knowledge, I don't think any significant number of modern philosophers would use it.

Well, they're concerned with Gettier problems etc too.

But yeah the problem what you mean by "knowing." For example, I am perfectly happy to say that a LLM "knows" words just like I "know" words, or that it "knows about" very popular topics in a loose sense.

But that's a very different sense of "knows" from the kind of specific propositional knowledge we're usually interested in when using a LLM as a search engine, fact checker or source of information!

By that definition of knowledge, I don’t contain knowledge either.

I’m clearly keeping track of how strongly I believe a variety of things. Not how I started believing them to begin with, however, so I lack the justification. And in a lot of cases, the lost justification would have been: “Because someone else said it, so now I’m aping them.”

I mean, this is why philosophical skepticism is a thing. There's hundreds of books written on epistemology.

But I think almost none of them would ascribe "knowledge" to a statistical engine that output propositions of random truth values weighted by how many examples of that proposition it's seen in it's training data.

You can view the logprobs in the API and literally watch it choose to say different things based on it's internal dice rolls.

What we have that an LLM doesn’t have is a mobile body and other sensory inputs so we can verify knowledge through practice.

Knowledge without a feedback mechanism for verification is prone to veer off course.

Right now the feedback mechanism is based on human reinforcement.

What if we put an LLM in an android’s body and let the machine interact with the world through the same senses that we do?

A body may be a necessary condition for epistemological knowledge, but it's certainly not sufficient... most animals are embodied but do not possess propositional knowledge (IMO anyway.)

Ultimately though this is all about the definition of words. Humans find the distinction between "true" and "false" to be important and meaningful; animals don't care. AIs don't care (being incapable of caring), but humans care very much about the propositional veracity of statements that AI makes.

The important thing is that we're on the same page about what's actually happening with these systems and how they work and don't slip into magical thinking or assume that just because a system is facile with language it automatically has a sense of truth or right or wrong.

This is only going to get more complicated as models get more sophisticated.

> most animals are embodied but do not possess propositional knowledge

https://www.youtube.com/watch?v=yhfl7kasjZc

There are a quite a few videos with the same reaction. There does seem to be something going on.

> but it's clear that they contain a vast amount of knowledge.

Is it though?

There's nuance in what we call knowledge, but to me, knowledge is not uni-dimensional. What I mean by that is, I know when I _know_ something, and I have learned that that is not a trivial skill.

How do I know I _know_ something? Usually because I have searched for evidence from multiple angles / dimensions / senses. When multiple independent observations of quality agree, and my conclusion matches the result, it is safe to say that I know that fact.

An LLM may have multiple 'dimensions' in a linear algebra sense, but does it have independent information?

For what it's worth, I mentioned they _contain_ the knowledge to try to avoid discussions about whether they really "know" anything -- if there's a better term like 'information about the world' or something, I think that's all I was trying to tell the original person I replied to. The point wasn't that they do what humans do, but that they are not just grammar machines.

> How do I know I _know_ something? Usually because I have searched for evidence from multiple angles / dimensions / senses. When multiple independent observations of quality agree, and my conclusion matches the result, it is safe to say that I know that fact.

I wanted to thank you for laying out this perspective. In my experience, in discussions about LLMs, a lot of time people get caught up on words like 'know' which work well for talking about humans, but don't cash them out in definitions I know what to do with. This is a really content-ful of sharing what you mean by your claim.

This is technically true, but no one actually does much of it consciously. But this is how you picked up every skill and understanding, including language, since you've been born.

And this - the unconscious fusion of correlated patterns - is pretty much what LLMs undergo in training.

EDIT:

Also, a lot of confusion is created by using terms like "a fact" or "a conclusion" or "a result", as if they were fixed points in space, fully defined mathematical symbols. But there exist no such thing. There is no binary "know/don't know", there is no binary "result match/does't match the conclusion". Those are all continuous quantities, that get rounded off for convenience.

>Clearly, they aren't as likely to generate "Colorless green ideas sleep furiously" as "It's important to feed your dog".

There is a setting right there called temperature, no?

You would think.

However, stating that LLMs predict text, not facts - and that LLMs don’t “think”, is enough to start a debate.

Yeah the issue is anthropormphization and the “cool” factor of demos, vs actual experience.

> Isn't the hint in the name? It's a language model, not a knowledge model.

True. But most of the training corpus was not linguistically-sound gibberish; it was actual text. There was knowledge encoded in the words. LLMs are large language models - large enough to have encoded some of the knowledge with the words.

Some of. And they only encoded it. They didn't learn it, and they don't know it. It's just encoded in the words. It comes out sometimes in response to a prompt. Not always, not often enough to be relied on, but often enough to give users hope.

What we really need is a language model coupled to a knowledge model.

> And they only encoded it. They didn't learn it, and they don't know it.

People say stuff like this a lot, and I never know what to do with it. I'm not sure what specifically I'm supposed to get out of these distinctions.

I'm not sure either. An alien could make a similar argument about our brains made of meat and electricity.

Well, without bothering to get all epistemological about it, it's clear that LLMs doesn't know, say, the law in a way that a lawyer knows it, or physics in the way that a physicist knows it.

And that's enough to be a problem, given how people are trying to use LLMs. They're trying to use it like a human who "knows" (whatever that means). And whatever an LLM actually does, it doesn't do that.

My uncle doesn't know the law the way a lawyer knows it either, but that doesn't stop him from trying to use it like someone who does ;)

If the claim is only "LLMs don't work like humans" and "LLMs make a lot of factual and logical errors", then that seems to be correct as far as it goes, but it doesn't go very far.

> But most of the training corpus was not linguistically-sound gibberish; it was actual text. There was knowledge encoded in the words. LLMs are large language models - large enough to have encoded some of the knowledge with the words.

>Some of. And they only encoded it. They didn't learn it, and they don't know it. It's just encoded in the words. It comes out sometimes in response to a prompt. Not always, not often enough to be relied on, but often enough to give users hope.

And some of the pieces of "knowledge" in the training corpora were wrong, lies, or bullshit themselves.

Garbage in, garbage out.

> The trouble is people think it goes deeper when the knowledge modeling part only happens incidentally because language and knowledge modeling are so closely related.

I think there's a strong argument to be made (that has been made) that much of our knowledge capability is built on language - that language did not simply allow us to communicate our knowledge, but actually allowed individuals to think in a new way.

I certainly agree that current LLMs have no regard for truth and logic at their core, and there are clearly areas where their ability to read and write does not translate to producing true statements. LLMs can't build things out of language like humans can.

I am less sure that LLMs don't have a long way to go before we reach the limits of what modeling language can do.

I can invent a grammar, explain it to an LLM, and it can apply it correctly in ambiguous situations. If its just generating output based on statistical probability based on its training corpus, how would that work? It feels like there's something emergent there that isn't just a very good probabilistic calculator.

> Isn't the hint in the name?

Actually the hint is not in the name, that only tells a portion of the truth. If LLM were trained on textbooks teaching foreign languages from every other language, the name would suffice. But it does not, LLMs are trained on wide variety of texts, many of them social media, blog, and general web posts. The larger web is an ocean of bullshit, and that is where the bullshit within LLMs comes from.

'Textbooks are all you need' ought to be how LLMs are trained so people can use them for their "knowledge work" without bullshit ramifications.

> 'Textbooks are all you need' ought to be how LLMs are trained so people can use them for their "knowledge work" without bullshit ramifications.

But in many spheres of knowledge there are still a lot of contradictory opinions in textbooks. Why did Rome fall?, What's the best X? In science there have been many schisms: quantum theory, tectonic plates, GMO/Organic etc.

Writing code is probably one of the few areas where just about all content is 'good' - there may be a lot of bad or poor performing code in books and online but the vast majority of it 'works'.

Very good points. That gives me the idea of specifically training LLMs on each side of the active scientific controversies and then host/publish an event where they debate. Might be very interesting to geeks, or they will hate it. Probably both.

Textbooks are a rather bad source of info. Consider that history textbooks in many countries look at everything through the lens of a particular national mythology, since one of the goals of a country’s public education is to forge a cohesive society, and that is done by downplaying past controversies.

For many other fields, introductory textbooks even at the university level may abound in oversimplifications, and it is only in later years of study that one begins, by reading specialized papers and monographs, to grasp how complicated the facts really are.

> The larger web is an ocean of bullshit, and that is where the bullshit within LLMs comes from.

Not only that: there's also the fact that LLMs do not "fact check" their output against any data source. They just generate text based on relative frequencies of different kinds of text in their training data. They don't even have the concept of text being related to other things in the world; in fact they don't even have the concept of "other things in the world".

Any text generator built that way will output bullshit, no matter how accurate its training data is. "Bullshit" does not mean the output is necessarily wrong; it means, just as Frankfurt said, that the thing generating the output doesn't care about whether it's true or not. LLMs meet that criterion: they don't care about anything other than relative frequencies in their training data.

> Isn't the hint in the name? It's a language model, not a knowledge model.

Well yes, but how is this insight helpful? People aren't just looking at the name. Have your missed the last 10 months of ridiculous breathless coverage of LLMs?

> The trouble is people think it goes deeper when the knowledge modeling part

People are literally calling this shitty text prediction algorithm "AI". The discourse is completely out of hand.

It is "AI", using the definition of that term that was established when the term was coined in 1956.

https://en.m.wikipedia.org/wiki/Dartmouth_workshop

> An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves.

LLMs fit that definition well. They're not AGI, but I think it's fair to call them AI.

There is no widely accepted definition of "AI" and even if I accepted this one (I absolutely do not) no LLM meets this bar in the general sense.

Game tree heuristic searches, like the minimax algorithm with alpha-beta pruning, has also been called AI, which is why programs that play chess are called AI.

What I think is the problem is that when people hear "artificial intelligence," they don't think of chess programs anymore; they think of something like Skynet from the Terminator. Or, maybe now they think of ChatGPT.

That 1956 workshop is widely credited as the place where the term AI was coined.

If they don't get to define "AI", who does?

What some people now call AGI, the average person thinks of when they hear AI.

Reply on news.ycombinator.com