Hacker News Reader

Comment by random3 | Hacker News Reader

Just like they "know" English. "know" is quite an anthropomorphization. As long as an LLM will be able to describe what an evaluation is (why wouldn't it?) there's a reasonable expectation to distinguish/recognize/match patterns for evaluations. But to say they "know" is plenty of (unnecessary) steps ahead.

View on HN

This was my thought as well when I read this. Using the word 'know' implies an LLM has cognition, which is a pretty huge claim just on its own.

Does it though? I feel like there's a whole epistemological debate to be had, but if someone says "My toaster knows when the bread is burning", I don't think it's implying that there's cognition there.

Or as a more direct comparison, with the VW emissions scandal, saying "Cars know when they're being tested" was part of the discussion, but didn't imply intelligence or anything.

I think "know" is just a shorthand term here (though admittedly the fact that we're discussing AI does leave a lot more room for reading into it.)

The toaster thing is more as admission that the speaker doesn't know what the toaster does to limit charring the bread. Toasters with timers, thermometers and light sensors all exist. None of them "know" anything.

Yeah, I agree, but I think that's true all the way up the chain -- just like everything's magic until you know how it works, we may say things "know" information until we understand the deterministic machinery they're using behind the scenes.

I'm in the same camp, with the addition that I believe it applies to us as well since we're part of the system too, and to societies and ecologies further up the scale.

I agree with your point except for scientific papers. Let's push ourselves to use precise, non-shorthand or hand waving in technical papers and publications, yes? If not there, of all places, then where?

"Know" doesn't have any rigorous precisely-defined senses to be used! Asking for it not to be used colloquially is the same as asking for it never to be used at all.

I mean - people have been saying stuff like "grep knows whether it's writing to stdout" for decades. In the context of talking about computer programs, that usage for "know" is the established/only usage, so it's hard to imagine any typical HN reader seeing TFA's title and interpreting it as an epistemological claim. Rather, it seems to me that the people suggesting "know" mustn't be used about LLMs because epistemology are the ones departing from standard usage.

colloquial use of "know" implies anthropomorphisation. Arguing that usign "knowing" in the title and "awarness" and "superhuman" in the abstract is just colloquial for "matching" is splitting hairs to an absurd degree.

You missed the substance of my comment. Certainly the title is anthropomorphism - and anthropomorphism is a rhetorical device, not a scientific claim. The reader can understand that TFA means it non-rigorously, because there is no rigorous thing for it to mean.

As such, to me the complaint behind this thread falls into the category of "I know exactly what TFA meant but I want to argue about how it was phrased", which is definitely not my favorite part of the HN comment taxonomy.

I see. Thanks for clarifying. I did want to argue about how it was phrased and what is alluding to. Implying increased risk from "knowing" the eval regime is roughly as weak as the definition of "knowing". It can be equaly a measure of general detection capability, as it can about evaluation incapability - i.e. unlikely news worthy, unless it reached top HN because of the "know" in the title.

Thanks for replying - I kind of follow you but I only skimmed the paper. To be clear I was more responding to the replies about cognition, than to what you said about the eval regime.

Incidentally I think you might be misreading the paper's use of "superhuman"? I assume it's being used to mean "at a higher rate than the human control group", not (ironically) in the colloquial "amazing!" sense.

I really do agree with your point overall, but in a technical paper I do think even word choice can be implicitly a claim. Scientists present what they know or are claiming and thus word it carefully.

My background is neuroscience, where anthropomorphising is particularly discouraged, because it assumes knowledge or certainty of an unknowable internal state, so the language is carefully constructed e.g. when explaining animal behavior, and it's for good reason.

I think the same is true here for a model "knowing" somethig, both in isolation within this paper, and come on, consider the broader context of AI and AGI as a whole. Thus it's the responsibility of the authors to write accordingly. If it were a blog I wouldn't care, but it's not. I hold technical papers to a higher standard.

If we simply disagree that's fine, but we do disagree.

I think you should be more precise and avoid anthropomorphism when talking about gen AI, as anthropomorphism leads to a lot of shaky epistemological assumptions. Your car example didn't imply intelligence, but we're talking about a technology that people misguidedly treat as though it is real intelligence.

What does "real intelligence" mean? I fear that any discussion that starts with the assumption such a thing exists will only end up as "oh only carbon based humans (or animals if you happen to be generous) have it".

Any intelligence that can synthesize knowledge with or without direct experience.

So ChatGPT? Or maybe that can't "really synthesize"?

How would ChatGPT come up with something truly novel, not related to anything it's ever seen before?

Has a human ever done that?

We obviously can, otherwise where do our myriad of complex concepts, many of which aren't empirical, come from? How could we have modern mathematics unless some thinker had devised the various ways of conceptualizing and manipulating numbers? This is a very old question [1] with a number of good answers as to how a human can [2].

1: https://plato.stanford.edu/entries/hume/#CopyPrin

2: https://en.wikipedia.org/wiki/Analytic%E2%80%93synthetic_dis...

As you link to The Copy Principle: it, or at least that summary of it, appears to be very much what AI do.

As a priori knowledge is all based on axioms, I do not accept that it is an example of "something truly novel, not related to anything it's ever seen before". Knowledge, yes, but not of the kind you describe. And this would still be the case even if LLMs couldn't approximate logical theorem provers, which they can: https://chatgpt.com/share/685528af-4270-8011-ba75-e601211a02...

You'd have to pick something that fits:

> come up with something truly novel, not related to anything it's ever seen before?

I've never heard of a human coming up with something that's not related to anything they've ever seen before. There is no concept in science that I know of that just popped into existence in somebody's head. Everyone credits those who came before.

Here you say:

> with or without

But in the other reply, you're asking for:

> something truly novel, not related to anything it's ever seen before

So, assuming the former was a typo, you only believe in a priori knowledge, e.g. maths and logic?

https://en.wikipedia.org/wiki/A_priori_and_a_posteriori

I mean, LLMs can and do help with this even though it's not their strength; that's more of a Lean-type-problem: https://en.wikipedia.org/wiki/Lean_(proof_assistant)

Yeah I was specifically asking for synthetic a priori knowledge, which AI by definition can't provide. It can only estimate the joint distribution over tokens, so anything generated from it is by definition a posteriori. It can generate novel statements, but I don't think there's any compelling definition of "knowledge" (including the common JTB one) that could apply to what it actually is (it's just the highest probability semiotic result). And in fact, going by the JTB definition of knowledge, AI models making correct novel statements would just be an elaborate example of a Gettier problem.

I think LLMs as a symbolic layer (effective, as a "sense organ") with some kind of logical reasoning engine like everyone loved decades ago could accomplish something closer to "intelligence" or "thinking", which I assume is what you were implying with Lean.

My example with Lean is that it's specifically a thing that does a priori knowledge: given "A implies B" and "A", therefore "B". Or all of maths from the chosen axioms.

So, just to be clear, you were asked:

> What does "real intelligence" mean?

And your answer is that it must be a priori knowledge, and are fine with Lean being one. But you don't accept that LLMs can weakly approximate theorem provers?

FWIW, I agree that the "Justified True Belief" definition of knowledge leads to such conclusions as you draw, but I would say that this is also the case with humans — if you do this, then the Gettier problems show that even humans only have belief, not knowledge: when you "see a sheep in a field", you may be later embarrassed to learn that what you saw was a white coated Puli and there was a real sheep hiding behind a bush, but in the moment the subjective experience of your state of "knowledge" is exactly the same as if you had, in fact, seen a sheep.

Just, be careful with what is meant by the word "belief", there's more than one way I can also contradict Wittgenstein's quote on belief:

> If there were a verb meaning "to believe falsely," it would not have any significant first person, present indicative.

Depending on what I mean by "believe", and indeed "I" given that different parts of my mind can disagree with each other (which is why motion sickness happens).

> And your answer is that it must be a priori knowledge, and are fine with Lean being one. But you don't accept that LLMs can weakly approximate theorem provers?

I said that a hypothetical system that used gen AI to interact with the world (get text, images, etc.) and then a system like Lean to synthesize judgments about those things could potentially resemble "intelligence" like humans possess.

>but I would say that this is also the case with humans

Most of the "solutions" to Gettier problems that I find compelling rely on expanding the "justified" aspect of it, and that wouldn't really work with gen AI, as it's not really possible to make logical statements about its justification, only probabilistic ones.

Wittgenstein's quote is funny, as it reminds me a bit of Kant's refutation of Cartesian duality, in which he points out that the "I" in "I think therefore I am" equivocates between subject and object.

> I said that a hypothetical system that used gen AI to interact with the world (get text, images, etc.) and then a system like Lean to synthesize judgments about those things could potentially resemble "intelligence" like humans possess.

What logically follows from this, given that LLMs demonstrate having internalised a system *like* Lean as part of their training?

That said, even in logic and maths, you have to pick the axioms. Thanks to Gödel’s incompleteness theorems, we're still stuck with the Münchhausen trilemma even in this case.

> Most of the "solutions" to Gettier problems that I find compelling rely on expanding the "justified" aspect of it, and that wouldn't really work with gen AI, as it's not really possible to make logical statements about its justification, only probabilistic ones.

Even with humans, the only meaning I can attach to the word "justified" in this sense, is directly equivalent to a probability update — e.g. "You say you saw a sheep. How do you justify that?" "It looked like a sheep" "But it could have been a model" "It was moving, and I heard a baaing" "The animatronics in Disney also move and play sounds" "This was in Wales. I have no reason to expect a random field in Wales to contain animatronics, and I do expect them to contain sheep." etc.

The only room for manoeuvre seems to be if the probability updates are Bayesian or not. This is why I reject the concept of "absolute knowledge" in favour of "the word 'knowledge' is just shorthand for having a very strong belief, and belief can never be 100%".

Descartes' "I think therefore I am" was his attempt at reduction to that which can be verified even if all else that you think you know is the result of delusion or illusion. And then we also get A. J. Ayer saying nope, you can't even manage that much, all you can say is "there is a thought now", which is also a problem for physicists viz. Boltzmann brains, but also relevant to LLMs: if, hypothetically, LLMs were to have any kind of conscious experiences while running, it would be of exactly that kind — "there is a thought now", not a continuous experience in which it is possible to be bored due to input not arriving.

(If only I'd been able to write like this during my philosophy A-level exams, I wouldn't have a grade D in that subject :P)

But do you know what it means to know?

I'm only being slightly sarcastic. Sentience is a scale. A worm has less than a mouse, a mouse has less than a dog, and a dog less than a human.

Sure, we can reset LLMs at will, but give them memory and continuity, and they definitely do not score zero on the sentience scale.

If I set an LLM in a room by itself what does it do?

Yes, that's my fall back as well. If it receives zero instructions, will it take any action?

Helen Keller famously said that before she had language (the first word of which was “water”) she had nothing, a void, and the minute she had language, “the whole world came rushing in.”

Perhaps we are not so very different?

All LLMs have seen more words than any human will ever experience.

Yet they cannot take action themselves.

That’s a safety thing that we have placed upon some LLM’s. If we designed them to have an infinite for loop, the ability to learn and improve, access to mobility and a bunch of sensors, and crypto, what do you think would happen?

Yes, anyone can do it already. E.g. I am sure people have built simple robots with wheels in their home that LLM is controlling by reciving camera, microphone, lidar etc input and then putting output like commands where to turn, what to put in the speakers etc next and could theoretically go indefinitely if there is electricity.

> Yet they cannot take action themselves.

Neither could Hawking, once the motor neurone disease got far enough.

I like the sentiment, but reality says otherwise - just watch a newborn baby make it's demands widely known, well before language is a factor.

Ummm. Maybe you should look up Helen Keller.

Helen Keller did in fact make her demands they just couldn’t be known. In contrast the LLM does nothing of its own volition.

If you put the LLM in a never ending loop, it would definitely be doing something.

A something defined by someone else, yes.

Additionally, thinking organisms don’t get stuck in never ending loops because they can CHOOSE to exit the loop. LLMs don’t have that ability

My analogy of being in loop means being in a live state. So we as humans are in the loop continuously, we do have a way to exit the loop, but in that comparison it means taking our own life. We are in loops of getting input and producing output. You can also give LLM a tool to shut itself down, or you can give it tools to build on its knowledge base, so it would always be outputting new tokens that are based on new input and are producing different output.

E.g. it could have access to camera and microphone feed, which is automatically given to it in interval as part of the loop, it could call tools or functions to store specific bits and pieces of information, to store in its RAG or whatever based knowledge base. It is not going to be in the loop of producing the same token over and over, it would be new tokens because the context and environment is constantly evolving.

We put the LLM in a loop with no instructions with whatever tools you want. Now what?

We will observe what it would do. We could write a script to try it out.

It just gets into an endless loop. Human brains are ridiculously good at avoiding those somehow, you almost never see a biological brain stop functioning without being physically damaged. The error handling is so very robust.

Have you tried it already? What is the endless loop it gets into?

Sure, so I just tried it with visual and audio input.

It does nothing. Because there is not impetus for it to do anything by itself.

What do you mean by nothing? How did you put the visual and audio input, which model, how did you loop it etc?

It’s preferred method of text.

Maintain context and trigger at 1 second intervals.

It has no desires of its own. Nothing that motivates it. It’s not conscious.

It produced no tokens at all?

> It just gets into an endless loop. Human brains are ridiculously good at avoiding those somehow, you almost never see a biological brain stop functioning without being physically damaged. The error handling is so very robust.

We get constantly changing input. And yet, look at this thread, where the same points are being echoed without anyone changing their mind.

> Yes, that's my fall back as well. If it receives zero instructions, will it take any action?

By design, no.

But, importantly, that's because the closest it has to an experience of time is an ongoing input of tokens. Humans constantly get new input, so for this to be a fair comparison, the LLM would also have to get constant new input.

Humans in solitary confinement become mentally ill (both immediately and long-term), and hallucinate stuff (at least short term, I don't know about long term).

Is the LLM allowed to do anything without prompting? Or is it effectively disabled? This is more a question of the setup than of sentience.

Does this have anything to do with intelligence or awareness?

Absolutely.

What tools do you give it? E.g. would you put a GPU there that has LLM loaded into it and it is triggering itself in a loop?

It probably scores about the same as a calculator, which I’d say is zero.

>Sure, we can reset LLMs at will, but give them memory and continuity, and they definitely do not score zero on the sentience scale.

You've recreated a religious belief known as Animism and phrased it in a faux objective way. ("not score zero on the sentience scale.")

Communication is to vibration as knowledge is to resonance (?). From the sound of one hand clapping to the secret name of Ra.

I resonate with this vibe

I think people are overpromorphazing humans. What's does it mean for a human to "know" they are seeing "Halle Berry". Well it's just a single neuron being active.

"Single-Cell Recognition: A Halle Berry Brain Cell" https://www.caltech.edu/about/news/single-cell-recognition-h...

It seems like people are giving attributes and powers to humans that just don't exist.

overpomorphization sounds slightly better than I used to say: "anthropomorphizing humans". The act of ascribing magical faculties that are reserved for imagined humans to real humans.

s/knows/detects/

and s/superhuman//

(sees FSV UI on computer screen)

"It's a UNIX system! I know this!"

The app knows your name. Not sure why people who see llms as just yet another app suddenly get antsy about colloquialism.

If it talks like duck and walks like duck...

thinks like a duck, thinks that it is being thought of like a duck…

Digests like a duck? https://en.wikipedia.org/wiki/Digesting_Duck If the woman weighs the same as a duck, then she is a witch. https://en.wikipedia.org/wiki/Celestial_Emporium_of_Benevole...

If you know enough cognitive science, you have a choice. You either say that they "know" or that humans don't.

It's like the critique "it's only matching patterns." Wait until you realize how the brain works.

Reply on news.ycombinator.com