Hacker News Reader

Comment by kgeist | Hacker News Reader

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do it too. Say, when MySQL fails to start because it tries to read its config from the wrong dir, I may say "oh, this guy thinks he must read the config from ..." (having a language with grammatical genders as my native language also helps make it sound pretty natural). It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.

View on HN

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase.

I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.

> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.

I know people who are like that too.

I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.

> I know people who are like that too.

This is part of the problem being described. You are part of the problem.

"Some people are bad at X" is not comparable—is not even in the same category—as "LLMs are fundamentally incapable of X".

Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry".

What you are observing is that some humans are careless, do not take the time and effort to understand, or have internalized the idea that they're "not smart enough" or "not the type of person" who understands things like <whatever>.

That has nothing remotely to do with the fact that LLMs have no consciousness, no self-awareness, no cognition, no understanding. At a fundamental level.

I don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.

This is an extremely common fallacy I've seen lots and lots of people fall into with respect to LLMs. In nearly every case, they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.

This is deeply untrue, and is highly likely to lead them to bad conclusions about what we can and should do with LLMs.

> This is an extremely common fallacy ... they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.

Claiming that LLMs are conscious or human-like because humans can't do X seems a very strange way to argue for LLM intelligence.

Usually, it goes the other way around: an LLM sceptic says "LLMs are dumb because they can't do X" and soon someone has to remind them that also most of the population can't, in fact, do X.

“ Usually, it goes the other way around: an LLM sceptic says "LLMs are dumb because they can't do X" and soon someone has to remind them that also most of the population can't, in fact, do X.”

And it should not go this way, is the OPs point.

For an analogy, imagine someone looked at a bunch of lightbulbs and expresses dissatisfaction that they aren’t really stars. If someone replies by saying “not all stars are equally bright”, do you think that fact should carry any weight in the argument?

> And it should not go this way, is the OPs point

Well, he's wrong. If you argue that LLMs and humans are fundamentally different because all LLMs do X and no human does it, then showing you that it's not true demolishes your argument. Doesn't prove anything positive, but it certainly proves that your argument is invalid.

Well this requires you to buy the very bullshit argument that comparisons of two physically distinct systems just because they share outputs is meaningful.

If I call a lightbulb an artificial star, the onus is on me to show the behavior under the hood is star like, not just to point at the light and say “you must see it’s a a star since it’s emitting light!”.

> Well this requires you to buy the very bullshit argument

Sorry, no. The only thing it requires you to buy is basic logic. If you argue that B is true because of A, the fact that A is false invalidates your argument (I repeat: not B but your argument). There is no question about it.

"One man's modus ponens..."

Maybe if you the argument was that B is only true because of A, and only for that reason. That’s hardly the case when comparing LLMs and humans. The population behavior of humans isn’t the only thing where there’s both overlap and sharp distinction. The underlying mechanism and substrate are also completely different.

To say the same process is occurring with all this different is bizarre, and that is why the OP is right about which direction the argument of similarity has meaning.

You don’t disagree that our only pre-LLM exemplars of conscious thinking systems is from biology right?

If you’re introducing a new member, you have to compare it across all levels before making a similarity claim. It’s the height of ridiculousness to think otherwise.

Just a heads up - this person in gp comment, their original post in a way that is much much different than the one I was replying to originally both in tone and content.

Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were. Using humanizing language invites discussions of things like the rights and feelings of an algorithm. A judge that is misled by the application of human-centric language to an algorithm can lead to some terrible outcomes. Not everyone is an LLM expert and the language people use leads to them treating LLMs like actual, real humans. That is terrifying.

What is terrifying is the propensity of people hoping for a mechanical slave to do everything possible to avoid touching on the possibility for being-ness of the technology they are desperately hoping will work as a basis for that implementation.

Cyberpsychosis. A next token predictor is not a being.

Stop posting these things. Stop thinking these things.

Why can't it be a being? Why is thinking about such a possibilty bad?

Sorry, but if you're genuinely asking this not for trolling reasons then you should _really_ _really_ see a medical professional (not an insult).

Online comment sections are not the correct place to unpack any of this.

Which is indeed terminating this comment chain, but for good (and benevolent) reason. Doing anything else other than referring to a trained professional in a controlled context would potentially just feed delusions, which is highly unethical.

Might not even be yours but those of another reader.

Nice Unthinkable Topic you have there.

As I said, context.

In the right space with real people, worthy of a debate. On HN? No. Not like this. Not here. Not without filtering the participants for real human beings.

I'd argue that your colossal overreaction to a small invitation to discuss a somewhat interesting hypothetical is what warrants medical attention

I've been going through comment revisions here, and none of them really cut it, but all of them wasted time.

But regardless, you're using this as an insult. I did not.

There is no debate to be had here. Just a bad faith shouting match

Slavery only exists for living beings because humans have limited lifetimes, experience pain and can have their own desires and needs that can be forcibly taken away from them. Algorithms have none of these things and they don't need them (adding them would be the real cruel thing to do). There is no slavery for something that can't experience coercion.

> Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were.

I mean, that is the entire definition of the word. And you also anthropomorphize living beings like many people genuinely attach human qualities to their pets etc. Yes, the risks are very high when it comes to chatbots in particular, especially to people who are not technically inclined. But you'll be surprised at how crucial the ability of anthropomorphizing is. This is a very good paper that summarizes it and is definitely worth reading if you're interested in these things: https://www.researchgate.net/publication/5936908_On_Seeing_H...

> I mean, that is the entire definition of the word.

Not quite - my wording there was very deliberate. By saying that it's a problem when you're treating something that's not living (not non-human!) as if it were, that excludes pets and all animals from the equation. I understand how common it is for humans to assign human qualities to other things and beings, but there is also an unspoken variable of intensity. Representing abstract concepts as humans, interpreting living things in a human-like way or traditionally referring to ships as living beings has a different degree of belief and intensity compared to implying a genuine belief that algorithms are beings that can be enslaved, like what the sibling comment to this one does.

No its not unspoken. Read the paper linked. It explains what you think you are explaining but with scientific rigor.

And you are wrong in your defintion of the world. Attaching human qualities to any non human entity (living or otherwise) is the accepted defintion of what anthropomorphizing is. It does not only apply to non-living entities.

> I know people who are like that too.

Cool.

Question: Why _on earth_ would we make more of them?

>I'm not sure anthropomorphizing is a problem.

It is a problem because it does not really understand stuff. For example, if you ask a human "Do you understand that doing X will kill you 100%?". If the human answers "Yes", then you can expect the human to act according to that understanding. That they will not do X

But an LLM will happily acknowledges the consequences of doing X, but will still proceed to do X. So replace the human in the above example with a robot controlled by an LLM. There is no guarantee that it will not do X.

Have you not met humans that do that, too? When it was found that smoking causes cancer, many, many people kept doing it. Then the government taxed the crap out of it, and some people still do it.

It's easy to say that people have other motivations, but doesn't the AI, too?

You tell it not to do X, but you've also told it to do something that would benefit from X. It's going to "want" to still do X, to support that other thing.

It's also got all the "knowledge" that enables it to do the work in the first place, and all of the tendencies of the people who do that work, because that's what it's trained on.

It's really easy to anthropomorphize AI because it was literally modeled after people.

And for the record, as lead developer, I've had actual humans that reported me to go ahead and do things I specifically told them not to.

>smoking causes cancer

Which part of "X will kill you 100%" did you not get?

Do you have a working definition of "conscious"?

Do you? I’m not sure what you’re getting at.

Not GP, but one of the challenges with debating whether LLMs are "conscious" is that we don't even really know what it means for a human to be "conscious", or even if consciousness is experienced by other humans the same way it is for ourselves.

What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained: it's floating point numbers in a big matrix rather than biological structures and electrical impulses, but there is still that element of frequently used connections being strengthened and infrequently used ones being pruned.

What we hypothesize but do not know: there is a thin brain structure of grey matter called the claustrum that has tendrils that reach into nearly every other brain structure. In many ways, this is similar to the attention mechanism of the transformer architecture. It is hypothesized that this may be the seat of consciousness, owing to experiments where electrical stimulation of the claustrum caused patients to immediately lose consciousness. However, there is no way to prove this, owing to the difficulty of otherwise removing or disabling the most connected structure in the brain and observing its effect on consciousness without permanently killing the patient.

Beyond that, we don't know much. I've got a family friend that's been a practicing therapist for 50 years, and I asked him what was the most interesting observation he made in his career. It was that "Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."

>"Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."

Relevant article: "Generalizing from one example"[https://www.lesswrong.com/posts/baTWMegR42PAsH9qJ/generalizi...]

Personal note: That principle has been the bane of my autistic existence. People sometimes seem literally incapable of understanding that other people even can be different.

> What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained

ANNs have been inspired by biological processes, but in practice you have to squint very tightly to see the similarity. Biological neurons are multiple orders of magnitude more connected than the nodes in an ANN, plastic in terms of their connectedness and continuously learning, and their activations are also affected in complex ways by the levels of various transmitter molecules in the brain.

You are saying the AI doesn't a some property that you don't have any definition for, not even a working definition. People will disagree on whether a cat or a baby is conscious, they're not debating what a baby or cat is. They're debating this term. You might as well be debating whether an AI is a blorb or not, you have just as good a working definition of blorb as consciousness.

I believe you should look up the work of Cameron Berg before making statements like "an LLM is" or "an LLM isn't". Empirically defining all this stuff is very difficult, and making a definition that covers all beings that can exhibit conscious behavior is much more complex than a face value examination would reveal.

We do have entry and exit conditions. We can disrupt it and study the dynamics of the disruption. We have a good sense of the many mechanisms at play that undergird it.

What we lack is a full map of the exact process from sensation to consciousness across all modalities. And I’m afraid when it comes to interception, we can’t unless we observe every one of the 36 trillion or so cells in a human body, as well as the 36 trillion or so symbiotic and commensal microbes, continuously, all the time.

But I keep finding it astonishing that the claim that we know nothing about consciousness gets bandied about. We know a lot. We don’t have a grand unified theory. The lot we know is definitely split across many levels of evidence and hard to follow, let alone arrange. But this isn’t a black box. It’s a grey box, meeting an even more transparent box that is the LLM, where we do know what the guts are made of, and can interfere at every step in the chain of steps that constitute their dynamics.

Comparing the two, we know there’s a level of similarity in that information gets broken down via a neural network. That similarity was sought.

Since then though, neuroscientists have gone and shown that: 1. The other half of the cells in the brain, the glia, are at least as important as the neurons in cognition and consciousness 2. That interoceptive feedback and feelings are critical drivers of conscious experience 3. Evolutionarily, we know all cells can “learn”, and well before there were neurons or glia or brains, every cell evolved an internal clock that allows it to entrain to external solar and (depending on the species) lunar rhythms. 4. In the last few decades we’ve seen how synaptic activity is shaped and driven both by astrocytes and the circadian clock.

All this is showing us that the abstraction from the 1950s that current neural networks are built on were incomplete.

Whatever these components to do give rise to consciousness in biology (and we’re a long way from done solving this), we certainly wouldn’t imagine with all these modules and mechanisms missing, just maxing on one type of information flow in the brain would give you consciousness.

I’d urge you to not keep insisting consciousness is a total mystery. It’s not, and even your AI model of choice will be able to point you to all the mechanistic evidence we have that whatever it is, it isn’t just neural nets.

Whether human consciousness exists on neural nets or otherwise doesn't disallow an ANN in a particular configuration from being conscious. You might as well argue that human consciousness requires biological neurons, so artificial consciousness can't exist.

Odd. Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap? Seems wildly convenient. Where else in science can you show me such a comparable situation in how you define properties?

> Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap?

A light bulb doesn't need to do fusion to make light. An airplane doesn't need to flap its wings to fly. An ANN doesn't need to use a brain's structure to think.

You haven’t proven an LLM has thought. I can measure the spectral properties of light from both a bulb and the sun.

I can measure the dynamics that go with thought in a human.

I can measure some dynamics in an LLM. From all we know, there are huge differences, not least that you can literally turn off an LLM, whereas every process but death in biology shows continued activity even when the organism “looks” off,

What you haven’t shown me is that what the LLM does when it is on is “thinking”. Given that the dynamics are different, the substrate is different, and one can be fully turned off and the other cannot, why are these two things the same?

> You haven’t proven an LLM has thought

Give me any problem that previously required human thinking. Any. That can be solved by LLMs. All you have to do is provide a single counterexample.

> not least that you can literally turn off an LLM, whereas every process but death in biology shows continued activity even when the organism “looks” off,

There is no reasonable definition of thinking that requires that it can't be paused.

> why are these two things the same?

I said they aren't the same, just as a bulb isn't the same as a star. An LLM can think, just as a human brain can think. A bulb can produce light, just as a star can produce light.

>Give me any problem that previously required human thinking. Any. That can be solved by LLMs. All you have to do is provide a single counterexample.

Theres an itch deep in my butt, but I'm in an important meeting and can't leave the seat. Nor can I openly scratch the area.

I can tell myself these things, focus on the itching, think about how itching is felt by the body, think about the social consequences, and make it feel more bearable.

Please tell me which LLM has solved this problem so I can not outsource the issue to it. I eagerly await your answer.

> There is no reasonable definition of thinking that requires that it can't be paused.

Is there a reasonable definition of thinking where the words "paused" and "turned off" aren't treated the same? Can you do that kind of thinking and get back to me?

> I said they aren't the same, just as a bulb isn't the same as a star. An LLM can think, just as a human brain can think. A bulb can produce light, just as a star can produce light.

An LLM can produce language. You are yet to show me it produces thinking.

Your claim is like saying the bulb produces light, so it also goes through fusion.

> Theres an itch deep in my butt,

The LLM will tell you how to solve it. If you mean the LLM should modify your thoughts, no human could do that for you either, but an LLM agent can just as easily as a human figure out how to build the brain computer interface to make this possible in the future.

> Is there a reasonable definition of thinking where the words "paused" and "turned off" aren't treated the same?

No reasonable definition of thinking requires that thinking can't be paused or turned off. Please state your point more clearly.

> You are yet to show me it produces thinking.

Every problem that previously required human thought can be solved by LLM. QED.

> Your claim is like saying the bulb produces light, so it also goes through fusion.

No. How many times do I have to explain that the mechanism of thinking is not the same, just as the mechanism of producing light is not the same? You need to think to understand this, which is something that frontier LLMs have no problem with, but which most humans struggle with.

You’ve shifted your goal posts. You asked me for a problem I can solve with thinking. Not a problem another human can solve for me by their thinking. I have given you exactly what you asked for. An LLM cannot solve it. A human can’t solve it for me but can solve it for themselves (or not be capable of it), it varies.

Whatever you thought you were going to prove with your question, I do hope you’ll dwell on what my example tells you about what thinking is. It isn’t a string of words only. It can be that, but it’s more.

> No reasonable definition of thinking requires that thinking can't be paused or turned off. Please state your point more clearly.

It does. Please define thinking however you want. I’ll show you that it’s a requirement.

> Every problem that previously required human thought can be solved by LLM. QED.

Again, this is the same superficial thinking. Firstly, as my example above showed, no. There are other examples. It requires human thought to cook. No LLM can, nor is smell digitizable in an equivalent way to humans, and yet, cooking is one place where having a human nose is determinative to making appetizing food. Your QED is logically absurd, but is also resting on an assertion that’s flatly untrue.

> No. How many times do I have to explain that the mechanism of thinking is not the same, just as the mechanism of producing light is not the same? You need to think to understand this, which is something that frontier LLMs have no problem with, but which most humans struggle with.

When your “proof” for thinking is the output, I will also compare it to the output, light. My analogy shows the idiocy of your argument, and you don’t want to sit with its implications.

LLMs replicate human language. And many tasks human language can do, LLMs can do or help with. Thinking is not just producing word strings. We think about our family. We think about our bodily aches and pains. We think about how best to do various physical tasks. All this we do sans words, and LLMs don’t do. To call their internal dynamics “thinking” is silly wishful thinking by those who don’t have the ability to go read and understand biology.

And it’s a sad statement of how little you think of your own mind that you won’t explore facts that are outside your expertise that will actually teach you how your mind works, and the wondrous complexity underlying it, and the potential it has to keep doing new things.

I’m going to stop engaging at this point. You seem convinced LLMs think. I wish you the joy of that belief, whatever it is. Religion is always absurd, and this new one is no different.

> You’ve shifted your goal posts.

Remember the problem we're discussing is whether these LLM systems can think. Your deliberate misinterpretation of a question does not make you win that argument.

An LLM can solve problems for itself in just the same way that a human can solve problems for itself. It cannot directly rewire your thinking in the same way that another human cannot.

> It requires human thought to cook.

Why? What part of cooking requires human thought? https://www.moley.com/

> nor is smell digitizable in an equivalent way to humans,

Sensors aren't thinking, but people are working on better olfactory sensors too. https://www.nature.com/articles/s44182-025-00071-y

> assertion that’s flatly untrue.

You have yet to show me a problem that can only be solved by human thinking.

> When your “proof” for thinking is the output

What other way can we show thinking? From the input? Obviously not.

> My analogy shows the idiocy of your argument

It shows the idiocy of your argument. The output is thinking, just as the output is light. You are the one who continues to mistake the output for the mechanism.

> Religion is always absurd, and this new one is no different.

Your assertion of magic in biology that can't be replicated artificially is indeed absurd and one that almost no scientist would agree with.

None of this is an explanation of what consciousness is, either physically, logically or philosophically. (Maybe some physically)

Fundamentally it comes down to an objective decision about what that is. If you say it is “feelings” based on inputs and feedback mechanisms from the brain, then we can do the philosophical discussion around that.

“If the claim is that consciousness is limited to those with a specific type of cell behaving in a given way” thats just a coping mechanism hoping to use a mechanical definition to shield you from the reality that eventually all of these inputs, outputs and feedback mechanisms can be reliably reproduced in a different form.

I’m baffled: how does the fact that each cell has an internal clock that continues to tick even when you take away all external signals not impact what consciousness is logically?

If the time order driving behavior is driven by an internal timekeeper, that is logically relevant to the behavior you’re interrogating. If, on the other hand, all component systems depended on an external clock, that logically points to a completely different dynamic process.

As for the philosophy of it all… it’s true I make no comment on it. I’d rather look at the physical substrate and see what it does, and compare it to behavior, than try to fit a philosophy that originated before we had such high resolution knowledge of the system. They are blind to these facts, just because they were written up too early.

I utterly reject this doesn’t show you anything logically about consciousness. If I can show you that the molecular dynamics of each cell organize to anticipate the dawn, and do so even in constant darkness (and all this is well established.. there was a Nobel for the field in 2017) how is that not logically distinct from a system that is outside of regular time, needs to check a clock to locate itself in time, and then do whatever dynamics it does to solve the problem at hand?

As for calling this a coping mechanism… that’s a bit rich coming from someone who seems to have very little idea of the molecular and cellular biology but seems to want to hold to the belief that we’ve solved consciousness with LLMs. We can both fling that accusation about. Seems more productive to compare the physical dynamics and see what’s different and what’s similar, no?

You said (originally):

> I’d urge you to not keep insisting consciousness is a total mystery.

But don't want to get into the philosophy of it. I'd urge you to consider the definition of conciousness (as specified by the original question) wasn't just about biology. And it is 100% very difficult to define.

> We do have entry and exit conditions. We can disrupt it and study the dynamics of the disruption.

We've known life and death for quite some time. Even from the biological perspective, there's nothing new there in the largest sense. All we've done is refine it. Animals are aware of it.

> but seems to want to hold to the belief that we’ve solved consciousness with LLMs

Quite the contrary, much like everything else, the more we learn, the more questions we have.

> Seems more productive to compare the physical dynamics and see what’s different and what’s similar, no?

ok but....

> how is that not logically distinct from a system that is outside of regular time, needs to check a clock to locate itself in time, and then do whatever dynamics it does to solve the problem at hand?

We only check clocks because we have an internal molecular dynamic that directs us to do so, and it is tied to a similar system as everything else.

> But don't want to get into the philosophy of it. I'd urge you to consider the definition of conciousness (as specified by the original question) wasn't just about biology. And it is 100% very difficult to define.

No. I don't engage in the philosophy of it because most of the consciousness philosophy that gets bandied about and is familiar to folks is from 20-30 years ago, and rest on knowledge that is significantly less granular about details.

You don't, for instance, demand physicists need to engage in the philosophy of gravity do you? You demand theory. Neuroscience does the same of consciousness, and there are definite theories with support coming at the problem from different ends.

Insisting it's a mystery without engaging with the facts is not what I'm interested in.

> We've known life and death for quite some time. Even from the biological perspective, there's nothing new there in the largest sense. All we've done is refine it. Animals are aware of it.

Not quite sure what this means, but as with consciousness, I'd say life also isn't a complete mystery, we have made great strides understanding it as it is and it's evolution in the past few decades, and while the philosophy of life, mired in assumptions from past decades, remains hopelessly stalled, the science of life has simply moved on.

> We only check clocks because we have an internal molecular dynamic that directs us to do so, and it is tied to a similar system as everything else.

Is this tongue in cheek? Nothing about our internal molecular clock tells us to check clocks. For most of life, there were no manufactured clocks, remember? They are a relatively recent invention.

Jeez.

Ability to feel?

> Ability to feel

Interesting, possibly productive, but still not clear: that can be interpreted as just "reacting to input".

> that can be interpreted as just "reacting to input"

How? That's not the same.

Well, define "feel" - in a context that could be also applied to machines. It seems to be (if we really had to employ it for the sake of discussion) "getting signals from peripherals, orienting action".

> if it could, you arguing with it isn't going to make it "learn" or act differently.

Are you talking about a specific harness that doesn't have context retention mechanisms? For example, ChatGPT with disabled memory feature? Or in general where "it" is a fixed-weights network? The latter is trivially true, of course.

Even claude with “memory” enabled isn’t really “remembering” anything. It just injects it into the context and you hope it happens to find it relevant in its attention mechanisms, and then remembers to actually act on it. Anthropic’s own documentation states claude can and will ignore/truncate these. It’s a context trick, nothing approaching actual “memory,” and in fact, arguing with it will make a bunch of memory files, sometimes contradictory, and clutter up the context and act even worse.

I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]."

I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.

I just don’t find it really relevant to the argument presented I guess. I disable auto memory and have my own mechanisms and infrastructure with how my agentic system “knows” and “remembers” things which is roughly an automated, sometimes self-correcting working index on the file system. It behaves much better than claude’s automated “memory” system, so I use that, but digging into how that worked and making something of my own just makes me really dismissive of comparing it to something like actual memory, so I apologize if it came off dismissive.

I guess I will expand on what I meant why I react to claude memory acting mechanically or logically anything like human memory, is because it isn’t how memory in the brain works, they’re not comparable.

The layman’s understanding I have of memory, as someone that has dealt with memory issues much of my life, is that memory formation is heavily tied to emotions. emotions are triggered by input which sends a complex set of signals throughout the brain - you’re not just finding where in your head to store this, your brain is deciding how important it is, and what else to correlate it with - so it can tie them to other related memories. then on top of all this, much of the sensory experience you intake is subconsciously compared against high priority memory impressions and deciding what to pay attention to.

you could, argue that the sensory input is the simple md files and the emotional mechanism is the same effect as to how attention mechanisms work in llm’s. Ok, I can almost buy that, but these tools lack a fundamental ability to decide how important things are.

an analogy. you tell a person “if you pick a daisy in the next five years, an assassin will come to kill you” and they hold a knife to your throat while they say it, your brain whether you like it or not is going to say “THIS IS AN IMPORTANT MEMORY I NEVER MUST FORGET” and you’ll see something that looks like a daisy and have a panic attack 3 years later. that memory is never fo tell me claude or other tool harnesses using memory harnesses can prioritize memories the way that human would, instead they forget even when reminded, because the human brain is just so much better at it

It just acts fundamentally different than someone who would remember.

If someone only remembered vague scraps of what you'd expect them to remember, you might say the person can't remember.

It's much closer to notetaking and reviewing before responding than it is memory.

The issue with anthropomorphizing like this is that "memory" comes with baggage of expectations for it to do certain things, and it breaks them.

Just like "thinking" implies chain of thought, but you'll frequently get those reasoning traces and then a 180 in the final message.

Yeah, I think I've come to the conclusion that the biggest breakthrough we need before we can replace human thought is going to be some mechanism for live update of weights. "Learning" by injecting into context just isn't good enough.

But billions of dollars are going towards research to find these breakthroughs, so we'll get there eventually.

The AI doesn't actually go to sleep at night, it's [mechanistic explanation]

> Just like "thinking" implies chain of thought, but you'll frequently get those reasoning traces and then a 180 in the final message.

Be me, think extensively about an exam question, do 180 because the most probable option is just too obvious to be true (no, this part wasn't verbalized, it's how I describe what I felt about making the decision to do 180, or maybe it's a rationalization and it was a natural analog of an unfortunate sample from a probability distribution).

I’m normally not a big fan, but in this particular case it matters a lot. I could come up with some functional argument, but really I care from a model welfare perspective, whether the model understands its reasoning traces to be a part of itself or it’s simply predicting what a character who wrote the current intermediate tokens would output next.

Reading this is like watching someone call cars moving a “trick” because it uses gasoline as fuel while humans don’t move with gasoline.

Bro the LLM is a token machine, it is reasonable to have its short term memory be represented as tokens in context because the LLM is a token machine. Call it a trick if you want but it does fit the actual definition of what memory in actuality is.

Why argue with it? Why not just edit the memory yourself?

>Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity

You need evidence to make the positive claim that LLMs do not posses any form of consciousness.

Well, our current set of evidence is that it’s a mechanistic mechanical algorithm with an RNG embedded in it and we can both get it to repeatedly produce the same output for the same input and also get it to repeatedly do absolutely nothing at all, which are not characteristics we usually find in objects evincing consciousness.

LLMs bear absolutely none of the traits we’ve come to recognize as the external hallmarks of consciousness in biological organisms, nor anything that would seem analogous in a non-biological substrate.

That said, we don’t have a rigorous definition of consciousness that includes the actual phenomenology of consciousness, so I daresay if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary, I think the impetus is on you to define some version of consciousness that isn’t also satisfied by a book or a movie.

I would agree that LLMs aren't much like the human brain, that doesn't prove that consciousness is not occurring. Does a fruit fly experience anything? If a microscopic insect can experience something, why can't a CPU?

>if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary

Well there is neither any evidence that suggests LLMs are not conscious, and I also never asserted that they are. If I had to guess I would say that any information processing system will produce some kind of conscious experience, but I ultimately have literally no idea.

There’s plenty. First, when we say “LLM”, what are we referring to? What is the entity that would be conscious in this case?

The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences, and every time you assert the question is unknowable you allow that kind of loophole, so it’d behoove all of us for you to spend some time actually digging in on this instead of just idly making or rebutting assertions.

There’s a richer literature here than what you’ve seemed to have engaged with, and I’d encourage you to spend some time with it before handing more money to the magic AI people.

>The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences

I'm not going to change my beliefs or how I think about interesting questions just because its the "socially conscious" thing to do.

>There’s a richer literature here than what you’ve seemed to have engaged with, and I’d encourage you to spend some time with it before handing more money to the magic AI people.

It seems like you are highly emotionally invested in this and that is precluding you from open engagement with the topic.

You're not engaging with the topic. Engaging with the topic is where one either asks questions and listens to the answer or actually seeks to increase one's knowledge on the topic. You're just saying things. That's lazy.

And, you're welcome to do what you want to do, it's your god given right to stay as ignorant as you want about any particular topic, but that comes with consequences. If you want to call that being socially conscious, sure, you do you, but if you're interested in why people keep getting annoyed at your loud proclamations of ignorance which you're trying to proffer as evidence of a curious mind, well, that's why.

It's just as intellectually lazy to leave "consciousness" undefined and proceed to claim that "X cannot be consciousness" because of social reasons. That's the definition of a circular argument.

In the same vein, airplanes can't fly because they don't flap their wings.

As for ignorance: have you tried to make an LLM produce the same output for the same input (as you state above)? Give it a shot, you'll be surprised.

I’m not leaving consciousness undefined. I provided a definition and asserted LLMs were not conscious by that definition. I solicited an alternate definition and was given none. If you’d like to claim the LLM is conscious, you need to define what you mean, for both “the LLM” and “is conscious,” because it’s not passing any of our existing bars for consciousness, and the only thing you can point to are characteristics also present in other objects we don’t consider conscious, so again, the burden of proof is in fact on you and that other fellow to provide some definitions here, because as sits you and the other commenter are just saying shit and refusing to engage with any kind of rigor.

And, to your point: I have, and I did, and I was not. If you’d like to refine your argument further - What LLM? Provide input how, and in what fashion? Under what conditions? - we can have that conversation, but if your assertion is “I can’t get ChatGPT to consistently produce the same output twice and therefore it is conscious,” that’s an incredibly facile argument.

You two seem to be laboring under the impression that this is terra ignota philosophically; it’s not. There’s an enormous amount of literature, thinking, ideas, concepts, frameworks, and approaches that already exist here that you’re welcome to engage with, but you’re not doing that, nor are you actually engaging with any of my arguments except to deny their existence.

Philosophically - sure. But philosophy is a feedback loop. Humans trying to build a model of their brain within their own brain. There is by definition not enough oomph there, and that model will necessarily be approximate at best.

Think that's what an out of touch techie would say? May I remind you that just a few hundred years ago the best philosophers were debating whether the world would descend into anarchy if more people realized that the big monkey in the sky doesn't exist. Just because philosophers talk about something doesn't mean it exists in reality.

As for same output - there's a parameter called "temperature". It governs the random wandering in the output of an LLM. Reducing temp gives more deterministic output, but also reduces the capabilities. Is it possible that randomness is also precisely the mechanism behind human creativity?

it is a physical system, yes. So is our brain. I do not understand why this pointless fact is given out as an argument. Either explicitly say you think consciousness is supernatural or say what physical elements are missing in your view that makes you feel AI cannot be conscious.

Consciousness is a slippery word that is notoriously difficult to debate over. But often, people use 'consciousness' as a shortcut or a familiar word to describe a more complex idea. The point they're driving across isn't about the precise definition of the word 'consciousness', but about people treating LLMs as if they were actual human beings, assigning them all the traits and behaviors they would expect of a human.

I would argue, that the null hypothesis is that it is not, and that anyone claiming that there is a mote of consciousness are the ones with the burden of proof.

The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.

We know quite a lot about consciousness. You may not, but we know enough to know the informational dynamics in a brain are vastly different from an LLMs.

Exactly what kind of certainty are you looking for? Happy to provide it at a molecular, cellular, tissue or whole brain level.

Can we conclusively say one way or another if an ant has a conscious experience?

Yes we can say it has conscious experience. The content and depth of it, we cannot yet fully grok, and of course, what it feels like from inside the ant is something we never will know.

Okay, what about a virus.

Nope. It’s got a genuine off state, when it’s out of a cell, where it lacks agency. It’s dormant and can be taken up by a cell only by chance, not through its own active efforts. It bears none of the descriptive hallmarks of consciousness, nor is there any internal physical dynamic inherent to a virus. By definition it needs a host to start having a dynamic, and ant that point, it anctivates its drive, which is singular: reproduction and transmission, which it must balance.

And this is where you see viruses show the beginning of agency, in the making of this choice. They do show some signs of sensing their environment , even communicating with each other, through a process called quorum sensing, to decide if they should follow a lysogeny (integrate and stay quiet) or lysis (rapidly replicate and destroy the cell).

Note, this requires a host, so in so far as we can talk of a viral consciousness, it’s restricted to finding a suitable host.

There’s also recent work on viruses upon host entry showing some sensing. Usually a single molecule for which they already have a receptor. So you can begin to see the thin levels of awareness a virus has. But it still can’t do anything on its own beyond deciding to not infect.

I’d argue viruses are right at the edge of consciousness, just as they are right at the edge of life. They can hijack life and demonstrate much more life like properties, including a proto consciousness.

But they have no internal clock. They are always sensing the environment through multiple rich channels, nor do they have an active biochemistry they need to maintain when they’re dormant.

Consciousness adjacent? Consciousness-compatible? Both seem to fit viruses. And also LLMs, funnily enough.

What causes consciousness? Why aren't we all p-zombies?

Let me address the p-zombie first: it's a biological impossibly. Saying a p-zombie can be conceived of that has no inner feeling is like saying I can conceive of a philosophical star that radiates dark. Very nice that you can do that, but it's physically meaningless.

Why does biology preclude a philosophical zombie? Because every year we find new evidence of how utterly interdependent brain processes are and how tied to the experiential history of an organism.

We have mapped out the initial engrams that form in response to a new salient memory, and seen that 1. Time of day, 2. Physical location 3. Internal state all unique to the organism drive it's specific response, the specific way it saves the memory, etc.

That is, the organisms path history, it's interactions with the environment over space and time, shape it specifically based on the spaces and times IT experienced.

Take circadian rhythms, the internal timekeeping system all animals have. You feel it go haywire when you have jetlag. That is quite literally a history dependent response from your circadian clock to unexpected light.

Your p-zombie, if you specify it to have exactly your rhythms, will then need to have identical molecules being produced at identical times in every single cell, because that is how circadian rhythms are instantiated in all life, it's in each cell, and these cells can be synchronized, or, due to events like long distance travel or shift work, desynchronize. When desynchronized, you feel it, you feel different parts of your body adjusting, your meals shift, your taste and mood for food shifts...

Now, to put this in perfective, a human has about 36 trillion of their own cells, and an equal quantity of single cells symbiotes and commensal organisms, which give off all kinds of chemicals that shape, among other things, the circadian rhythms we talked about.

Now can you tell me: does the p zombie have not judt an identical compliment of 70 trillion odd cells, and they will all be in the same phase all the time as the cells in the non-zombie conscious person?

If so... How is this masterful phase synchrony across space achieved? What physical process allows for this?

If you cannot conceive of a way to do this synchrony (and you won't, basic physics forbids it), I think the sensible conclusion is to stop worrying about the philosophical zombies, and start looking at what information we have about actual humans and animals.

I brought up clocks to make a point, but actually, the evidence is building the biological clock is foundational for consciousness:

Some clinical signs: https://academic.oup.com/sleep/article/45/7/zsac094/6571981

And a broader theory: https://pubmed.ncbi.nlm.nih.gov/34382225/

> we don't know jack shit about consciousness

People keep saying this, but it's not true. We know a lot about consciousness. There's a lot we don't know about it, of course, but "jack shit" is wildly incorrect.

> I want to hear the evidence.

If you believe LLMs are conscious, then the onus is on you to provide evidence of such.

No, the onus is on either claim.

the onus is on whoever is making a positive claim.

If you are stating "LLMs are not conscious" you need to provide evidence, just like how if you are stating "LLMs are conscious" you need to provide evidence.

Would you want evidence that your your bicycle is not conscious? It can sense your inputs. It responds to the environment. The gears do multiplication (reasoning!).

I think that the difference is that there is a vast gap in complexity and abilities, no?

What about a Jacquard loom? Is it conscious? There is a vast gap in complexity and abilities between that and a bicycle.

To make the claim that an object is not conscious you need to both define what consciousness even is, then you need to make a machine capable of measuring it.

If you have done neither or those things then any claim you have about a bicycle's consciousness is just baseless speculation.

That's not a null hypothesis, that's a strawman.

Regardless -- if we don't know jack shit about consciousness (your words), then any claims of LLMs being conscious are by definition untestable, pure speculation and based on no evidence at all.

> then any claims of LLMs being conscious are by definition untestable, pure speculation and based on no evidence at all.

Am I saying otherwise?

What's the correct null hypothesis according to you?

Some tools like code rabbit (PR review bot) encourage you to do this. I couldn’t believe I found myself replying to code review comments to explain to an AI why we would rather let an exception crash the app than to catch and hide it several times so that it would stick in its memory. Having to interact with bots as if they are humans, especially when they are gate keeping, is degrading.

Not just coderabbit (though it is a bad offender). GitHub's copilot review feature is an equally miserable experience. That one loves talking in imperatives, regardless of the fact that it is a clueless machine.

FWIW, this tells you a lot about both the culture behind who built these things but also about the people that enable this stuff and don't immediately nope out. From that perspective, it's a low price to pay to learn whose judgement to never trust again.

You shouldn’t be interacting like it is a human. It is a LLM. And if the interaction happens to be in a form of prose similar to how you talk to other humans, that is coincidental.

i dont understand, isn't the arguing just some form prompt steering?

That's my thought. I've been using Matt Pocock's wayfinder and grilling skills, and I quite often end up in what would be considered "arguing" if done with another person. It's one-sided. It says something wrong, I correct it.

I find that the AI, like many programmers, likes things to be really solid and over-engineered. For a project that needs that, it's already pretty great. For my shopping list app that I tried creating with it, it was absolutely ridiculous. I ended up "blowing up" on it multiple times, impressing upon it the seriousness with which I meant things. Even with MP's skills adding that kind of context to written files, it still kept trying to scope creep the crap out of the project.

To me it’s evident that we are a few years away from the Her movie, where everyone on the street is talking to its IA friend.

I’m really afraid that it will totally destruct what is remaining of social tissue because why search for friends when you have an always on virtual (and pretty smart) friend h24 in your earbuds ?

I’m not blaming anyone for this outcome. I have myself argued with Claude more than once, and really not about code but about everyday things or nice facts of life I should rather have discussed with a friend.

I wouldn't worry about that too much. I would worry about it, but not too much.

People (generally speaking) also eventually stop eating just fastfood. Not all, but many.

So I think we can have some faith in the self-regulation of others.

Well, I have bad news : https://p.kagi.com/proxy/bja35kxoyryc1.png?c=TklOzPjLPioJ5YM...

That link just runs an infinite reloading loop in my browser. Bad news indeed.

Ah yes, my bad : https://ourworldindata.org/cdn-cgi/imagedelivery/qLq-8BTgXU8...

> I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless.

Unless doing so changed the compiler output, which is what happens when you say different things to an LLM.

code changes change the compiler output.

No. You’re the one who is ignorant.

The LLM is a black box. We do not, in fact know how it works. We know the learning algorithm and we know the scaffold of the transformer network, but the end result of all the weights interacting with each other is something we do not understand. We do not understand this anymore than we understand the human brain.

Now from this the best technical answer we can give is that we don’t know whether the LLM understands or is conscious. But you have to realize that same lack of understanding applies to humans. From a technical standpoint, You cannot say whether your best friend is conscious or not for the same technical reasons as to why you cannot say the same for the LLM. You don’t in actuality know anything.

So, when we have a machine that produces output identical and indistinguishable from an intelligent entity it is actually reasonable to call it conscious, because we already do this for humans. There is no other factor involved. What is clear is that the LLM isn’t human… there is enough evidence to show that its nature is extremely alien. But to say it doesn’t understand or it isn’t self aware is not something anyone can definitively make a statement about other then the fact that it BEHAVES and communicates in a virtually indistinguishable way from something that is self aware.

HN is full of arm chair experts who think they know what they are talking about. But you guys actually don’t. HN was wrong about AI and self driving cars, now we have Waymo. 10 years of research produced self driving cars that are 10x safer than humans. HN was wrong about LLMs. In the beginning HN was sure all it could do was write slop bootstrapped code… now it writes code for all of us. More than the general public HN has been making wrong predictions and wrong statements about AI with an authority that is outright ludicrous. We need to stop. It’s embarrassing how wrong we’ve been.

> Is anthropomorphizing a real problem?

Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.

Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this.

Having said that, I agree with you on the terminology front: I'm not going to say "learned prompt augmentation tokens" either.

I think LLMs are pretty clearly intelligent in some sense of the word, and I don't know how one could ever confidently know they aren't conscious in some sense.

That's not to say they're humanlike, just that people who think they know these ideas are ridiculous seem to be overreaching in the same way Steve Yegge seems to be overreaching

> conscious

The problem with that term is that it is hardly meaningful. Why would you use it? It does not add much, and no clarity, to what it is attributed to.

And on the flip side if it hardly means anything why would you be so concerned when it’s uttered? Just ignore it

> why would you be so concerned when it’s uttered

When I am told "beware of the xyphucymmon", I do beware, though not of the xyphucymmon.

And this means nothing, yet I’m not complaining endlessly about it on social media, or harassing my coworkers about it. Curious

My post «When I am told "beware of the xyphucymmon", I do beware, though not of the xyphucymmon» had meaning, which should be pretty clear. The reply remains: debate is not performance art in which you convey "through the medium of dance and howls". So, yes, communication has obligations for the locutor.

So, yes, imperfect speech is a problem - because societies are not performance art arenas.

@Razengan: speaking as myself a vocal critic of the downvoting system here, do notice that the post you replied to had a substantive reply at the time of downvote - mine. That is sufficent to justify the downvote - of a post that was not «perfectly fine». It somehow said that people should be free to say whatever they want and there are obvious reasons why we do not agree.

I mostly agree with your original point. Is it thinking? Is it intelligent? Is it conscious? From my perspective, overloaded words that we’ve reserved to make ourselves feel more special and above other members of the animal kingdom.

But you’ve not adequately communicated why you care so much whether anyone labels an LLM as such.

> Is it thinking? Is it intelligent? Is it conscious?

Very different things. "Thinking": "Dijkstra". We try to hire more intelligent people, while we do not have a clear idea on a property of "conscious" which would make hiring preferable.

> to make ourselves feel

Irrational.

> above other members of the animal kingdom

Irrational.

> why you care so much

There should be no hint that I would do specifically.

> whether anyone labels an LLM as such

If anyone labels an LLM <whatever>, the point of why would that be important should be clear.

And,

> if it hardly means anything why would you be so concerned when it’s uttered? Just ignore it

Because the behaviour you suggest, consistent with the exchanges in an opium parlor, is not behaviour consistent in normal contexts.

I think that "conscious" has a clear meaning in a sense that "xyphucymmon" does not. I think it might be impossible to engage in epistemology of consciousness (i.e. we can't find out of LLMs are conscious) but that doesn't make the term meaningless

> I think that... has a clear meaning

YawningAngel, of course you do, you employed it... We are telling you that it is not clear to the rest. It is slippery in the technical framework and thus even more so in more informal conversation.

What do you mean by consciousness here? Ie in what way do you think LLMs are conscious?

These discussions take a lot of time to get to the crux because neither side states their assumptions. Assumptions rarely get voiced at all.

I find it useful to ask:

1. Do you believe in quantum consciousness?

2. Do you think a "brain upload", a high-accuracy digital model of an organic human brain, would think or be conscious?

3. What is your working definition of thinking? It doesn't need to be rigorous.

people have always, and will always, be stupid as fuck in any domain they dont make their life. We can handwring terminology all we like but it wont stop the mythologizing and misleading takes on the topic. Not one bit. So why not just use language that is fun and works? Worrying about what others will think due to language used is wasted mental realestate, they'll think stupid shit no matter what, same as I do about a great many topics.

To be clear, there's no shade here, just recognition of the fact that we're all stupid and policing language does little to prevent the impact this has. I strongly beleive the bitter lesson extends to policing language. let language evolve naturally and it will naturally capture the ontological conatellations it needs to, and no one will be harmed in due course, more than they'd be no matter what.

> Actual real people believe

The problem with rapists is mental development, not "those sexy tables and chairs and pots and everything".

It's not that if people do not understand metaphors we should stop using them. It 's not that if people faint when they hear words we should stop using them. It is not that the free unduly associations in a large shared by many "collective subconscious" should hinder us...

> Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.

It's starkly ironic that the species that finds it easy to think (or to be conditioned by market forces to accept) that a computer is smart/introspective/sentient is the same species that will dehumanise actual living human beings because of differences in appearance, social status, or political affiliation.

I dunno, sometimes I encounter humans who lack actual, thinking, intelligence, perhaps even consciousness. I still treat them like people.

To quote Sarah Constantin:

> Humans Who Are Not Concentrating Are Not General Intelligences^

But yeah, you should still treat them with humanity. (Related: you should treat LLMs well, not because they're human but because you are^^)

^ https://srconstantin.github.io/2019/02/25/humans-who-are-not...

^^ hmm couldn't find this tweet but didn't look too hard

I gotta say though, turning on caps lock does seem to get them to cut out the inane banter.

There are other methods, including asking them to eg speak using the Google style guide rules, or to speak at 2/10 verbosity, or to speak as if they're talking to a very intelligent middle schooler, or to use simple sentences in subject verb object form

My favorite - reply like I do, review my previous messages to understand my tone and verbosity. As much as needed, as little as possible

It'll end up like that episode of Dr. Who where they get stranded in a lunar tour bus -- slowly starting to mimic you, and then it'll predict and replace you.

I'd like to hear definition of intelligence that would classify median person as intelligent but LLMs not (and then I'd wait few months for another moving of the goalpost and new definition). Bonus points if it's not recursive e.g. feature that only biological brains can possess.

I wouldn't classify the median human as intelligent.

> Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.

Yes, because it's possible. They don't have human consciousness/thought/intelligence, but it's entirely possible they have some form of consciousness, some form of real thought and some form of intelligence.

Actual real people believe that people are actual, thinking, intelligences, perhaps even with consciousness.

I think it was Karpathy that suggested he preferred “simulated reasoning” but sadly it didn’t catch on.

Actual real people think the world is flat, Elvis is alive, and aliens are regularly flying around the planet. Is preventing peoples unimportant personal beliefs really an important goal or is this just more warning label culture?

What's wrong with treating it as biology though? Even large software systems have biological aspects, their behaviour is emergent and if you want to observe how they work, a holistic approach is needed, you can't really reason about their full state...

For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff

Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.

> Is anthropomorphizing a real problem?

The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well.

For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc.

Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.

But how is that any different than people being misled by real humans saying words that reflect real thinking, but which are actually dead wrong?

The fallacy here is "thinking == correct", not "tokens == thinking"

Because the real thinking still cost the other human the same-ish energy it costs you to put words together, and because after all, the source is a human and not a machine, no, this is very different.

Being mislead may be the shared outcome. But why is different category of source of the mistake and the cost to producer of making the mistake not relevant in this discussion?

Where else in science do you brush aside all differences this way?

Why aren't humans simply biological machines?

There is no "science" that GP is brushing aside. You need to provide repeatable observations or experiments that GP is ignoring.

Please define "simply biological machines". I'm not sure "biological machine" had a proper definition. What's machine like about biology exactly?

A "machine" is a term that is well defined in science. https://en.wikipedia.org/wiki/Machine

And biological machine is? Don’t get me wrong. Biology I is full of molecules that we call machines. But you’re making a broader claim, saying that biology is only this.

This needs you to answer some questions:

1. Why do the machine parts in biology show such flexible application? A gear cog won’t ever moonlight as a signaling chip, but in biology you often have molecules doing double and triple duty.

2. How is the biological machine able to build itself? What does self assembly imply for the machines function?

3. Where does this machine get its inner drive? No LLM has been found that starts outputting text unprompted. A car doesn’t decide to move to a shady parking spot. Why? Where in the machine to biological machine continuum does the ability to make internally driven decisions come in? Why does it come in for biology? A bacterium is able to make such agentic decisions unprompted. Why is no manufactured machine able to do this?

> And biological machine is?

"Biological," too, is well defined. It relates to living things and their processes.

> Why do the machine parts in biology show such flexible application?

Evolution.

> A gear cog won’t ever moonlight as a signaling chip

A gear cog was purpose built for that purpose, but you will find that people often recycle parts into other systems, often in completely different roles.

> How is the biological machine able to build itself?

Protein synthesis.

> What does self assembly imply for the machines function?

The way that a machine is built has no bearing on how the machine functions. I could build the same machine using a 3d printer or a CNC router.

> Where does this machine get its inner drive?

Evolution selected for organisms that survive long enough to reproduce. Different biological systems handle this differently.

> No LLM has been found that starts outputting text unprompted.

If you give an agent a goal, it will perform actions to achieve that goal. This is just as true for artificial agents as it is for biological agents.

> Why is no manufactured machine able to do this?

Many do. Even robotic vacuum cleaners will charge themselves without human prompting.

FWIW I think the paper's argumentation is extremely weak to begin with. Like in section 4.1, it opens by expressing a sound position of skepticism:

> there are significant questions on whether these traces have any valid semantic import to the end user.

Which it contradicts in the very next paragraph, taking a stance that there are no valid semantics present in the trace:

> the false idea that derivational traces are semantically meaningful

It's really not a high quality paper worth taking seriously.

And that's before we get into the complete and total breakdown of objective analysis. It rejects distributional semantics as a theory, while also explicitly stating the results that have been produced under its auspices are "undeniable". Never elaborated on, and at no point in the paper am I given the impression the authors are even aware of the problem with this. It's just more unempirical slop that wants its pound of flesh without putting the work in. Frankly, whoever let this through peer review should be ashamed of themselves.

Yes it is a very serious problem because it confuses a lot of folks with a great deal of power like judges and policymakers.

The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages on this but now I believe it was because they understood that a metaphor can be a double-edged sword.

To be fair, the ANN architecture underneath is a misleading thing to be looking at, it's not where the comparison comes from. Though I can't tell if you meant it to be relevant in that way, or just as a general example for the dangerous nature of metaphor.

LLMs are expressly designed to approximate human behavior within the bounds of the written word. The anthropomorphization is no more philosophically problematic than saying differential calculus measures curves.

I meant it in the latter way: a metaphor can be useful as a pedagogical tool to introduce new ideas, and using the source of inspiration for this idea as the metaphor itself makes perfect sense, but unfortunately our brains seem to be prone to assign other properties of the metaphor that don't actually belong to the object of study.

I imagine this happens because we tend to conflate things that are similar, or maybe because it's not entirely clear which characteristics are being mapped in the metaphor?

They spent so many pages discussing it only to show that the mechanism for how ANNs work is different from the mechanism for how biological brains work. It says nothing about whether they can compute the same things.

> Is anthropomorphizing a real problem?

Of course it is. Anthropomorphizing is in our nature, but it doesn’t mean we have to entertain it and extend it to everything. A poet can anthropomorphize clouds beautifully and I’d enjoy his poem, but I want my pilot to not see clouds as rabbits when they decide if it’s safe to fly through them.

> but I want my pilot to not see clouds as rabbits

The poster meant it "within the metaphors". Tables are said to have 'legs': that does not confuse carpenters.

Yup, that’s why it’s important to ground these arguments in reality.

> none of the serious LLM researchers believe it has anything to do with human reasoning

But some of the biggest evangelists, who are well respected programmers that get lauded on this very site, have said it is fully sentient and has emotions. Even going back to 2022, when the LLMs were dogshit, a Google employee lost his job claiming it was sentient because it said it had emotions.

Combine that with the marketing angle of both Anthropic and OpenAI, who have been trying their hardest to describe every function of an LLM as analogous to the human brain. Because it's politically useful to paint them as dangerous and uncontrollable, so the keys will only be granted to the few people on the mountaintop.

I actually think on the LLM side it might be beneficial to refer to them as <thinking> because it explicitely guide the token generation towards a "thinking space".

As weird as it is, anthropomorphizing LLMs in prompts has been actually pretty useful (think of the latest big math discoveries which were achieved by having the user giving supporting words). It would be interesting to see if a LLM would perform worse if you used a more neutral term.

The paper's argument is rather than using terms like "thinking trace" can lead people to believe that the model is really thinking, and thus these traces can be used as a sort of interpratbility parameter. This can give a false sense of security when building a LLM-based system which requires guardrails and tracability.

> Is anthropomorphizing a real problem?

Completely rational and smart people talk to their pets, plants, their car, and other inanimate objects. This is not considered abnormal by most. It's just what we are wired to do. Some LLMs are uncannily good at tricking people into believing they are talking to a real person. So, there is that as well.

Some people are a bit freaked out by this or still somewhat in denial about LLMs being this good. But people have been yelling at their computers for as long as we've had them; so that ship sailed a long time ago. Trying to stop them doing that is probably a bit futile.

Whether people like this or not, LLMs are actually trained and fine tuned on real conversations and that's where a lot of this is re-enforced. Instead of fighting that, you can just lean into it and accept that communicating like you would with a person totally works and can actually be efficient even as it requires less effort and thinking on your side.

You can go all Jean Luc Picard on AIs and yell "Tea! Earl Grey Hot!" or you can just ask "I'd like a cup of tea, please". LLMs are good at remembering your tea preference. The please is of course completely redundant and should not affect the outcome. If you just want a cup of tea, you should be fine either way.

> Is anthropomorphizing a real problem

Yes. There's a difference between scrapping a session and starting over, or going back and branching something, or using sub-agents to see five outcomes, vs arguing with a system in a long drawn out chat.

Like - I know that if a model starts doing something silly, instead of correcting it - I can probably go back and edit two steps prior to add an extra guardrail, or extra data, or whatever.

> Is anthropomorphizing a real problem?

Yes it's really a problem. On this website you are surrounded by people who have technical knowledge and understand at least somewhat, how a computer functions. You have the ability to separate "fun" and "reality" because you know you're putting input into a really really big calculator. Most people do not fathom this.

AI Psychosis is a real thing, look it up (don't just ask an LLM) and do some reading. It's actively harming people, and the way they think. There's no regulation around any of this stuff and it drives me crazy that we let these AI companies _sprint_ so far ahead of everyone, and now we're facing the consequences.

Simplifying terminology is not a problem. The providers intentionally choosing terminology to make people think it's something it's not is a problem. I hate the term agent. Calling them companions as some do is just gross.

When I was taking an MIT AI course (in ancient pre-LLM times), an autonomous agent was defined as a system that perceives its environment and acts on it (we were focusing on reward-expectation-maximizing agents, but it's not that important). Peter Norvig has said something like, technically, anything can be described as an agent (a rock maximizes the "follow physical laws" objective), but naturally, it doesn't make much sense to model a rock as an agent. With AI agents, the situation is significantly less controversial: they do perceive, deliberate, and act.

All of these terms were picked by individuals, years ago, while reaching for metaphors that made sense to them personally.

None of these "agent" / "thinking" / "reasoning" terms were dreamed up in boardrooms to intentionally mislead people. They are useful but faulty metaphors; there is no conspiracy.

"Agent" is standard reinforcement learning terminology, used in Chris Watkins's thesis introducing Q-Learning in 1989.

I find it annoying because when I read ML papers nowadays I have to back-translate from anthropomorphized talk into actual machine talk, then mentally compare to what I actually know about brains and cognition.

> It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.

A lot of people are not in on the joke. ELIZA effect and AI psychosis is a thing.

Interacting a lot with LLMs might be damaging to the human psyche even for mentally stable people.

there are people on the fringes in relationships with these things, so yes definitely

"Esta poronga no sabe lo que está haciendo", aunque parezca femenino , en realidad no tiene género .

> Is anthropomorphizing a real problem?

Even tech companies are rolling out AI training which utterly anthropomorphizes it, and leads people to think its actually intelligence. This is part of the reason for the backlash - everyone understands it bullshit marketing the second you actually try to use it.

Reply on news.ycombinator.com