Hacker News Reader

Comment by qarl | Hacker News Reader

Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"

Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".

Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".

Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".

Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".

Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.

View on HN

“AI researcher” seems like a fairly broad category. People who primarily think about CNNs and transformers probably spend about as much time considering the problem of consciousness as any other educated person, which is not very much time. With this in mind, it’s not surprising most of the AI researcher predictions do not differ much from what the Public predicts here. And for this, it’s not clear what explanatory value this survey has other than to reveal minor biases the AI researchers may have.

It matters little. Inevitably some system will be developed which has a high degree of autonomy and mimics the functioning of a person extremely well. Anyone aware that there is a conceptual difference between phenomenal consciousness and intelligence will get shouted down.

I do find it strange that 50% of ai researchers and the public seem to think there will be a way of determining if these systems are conscious. There’s no evidence for this at this point that there ever will be such a test.

> I do find it strange that 50% of ai researchers and the public seem to think there will be a way of determining if these systems are conscious.

I think it's likely they're using a different definition than you. Seeing as there is no agreed upon definition, after all.

Regardless - I post this list because the article makes a bold claim at the beginning - and I wanted to demonstrate that in fact there is no consensus.

The definition presented in the survey is the one I have in mind. You may be suggesting that the respondents have a distinct one in mind though.

Yes, I missed that. I know that some of the other papers I cited have nonstandard definitions, and I thought that might be going on.

We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors.

There are those who believe that were they reduced to life support, they would no longer be alive and should therefore not be supported by said machines.

There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

There are those who believe that fungi/trees/plants are either individually sentient or sentient as a part of a network. Choosing to sacrifice their own nutrients to answer the call of a wounded neighbor, for instance.

Although, there are also those who believe that human's don't have any special unique quality that isn't shared by either all living things or all things in general. These individuals already believe that the machines have the same kinds of qualities as we do. They are slow when they are unhealthy (needing a dusting or coolant loop bleeding being equivalent to us needing some fresh air for instance) and uncooperative when upset (by a virus, full hard drive, or oom).

> There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

Anybody objecting to that statement is just ignorant of nature and/or adheres to the quasi-religious human superiority complex.

The human condition is one of deep and vast ignorance. Pulling information out of the field of reality is actually really difficult for humans much beyond what we see right around us. And the things we see are quite deceiving. And while humans for thousands of years have been as smart as we are, it wasn't till the enlightenment period that we really an ever increasing amount of information and knowledge take hold and start spreading across the world.

And there are plenty of people right now that would drag us kicking and screaming back into that ignorance and suffering. This is what can make talks of AI so annoying, not that people do or don't know what AI is, they don't have the first clue of what people are beyond their anecdotal experience. They don't question their motivations. Why particular flaws they have exist, and why these flaws are shared across humanity. What the pieces look like that make them tick. So yea, when these people talk it just adds noise to the conversation.

An eagle isn’t having a conversation with an eagle on the other side of the world about whether it should eat its prey because it’s sentient. It just eats it. I think that speaks to a moral superiority, in some ways.

I’m not claiming humans are morally superior in sum (there is a lot of evil as well as good.) But it’s pretty obvious that humans are the superior species evolutionarily in that we’ve essentially dominated the planet. And on some level, are beating evolution itself (by fixing genetic issues and helping the “least fit” to survive and reproduce.) And are also the only species which has doctors for other species.

All I’m trying to say: I can easily accept other species are intelligent and emotional. The fact that I can know that and accept that, to me, speaks to human “superiority.”

Does superiority matter? Not most of the time. What about when it comes down to life and death? If the choice was to keep one human alive or one theoretically more perfect AI alive? (Boil down utilitarianism into its essence.) doing the “most good” is by definition a human construct. Maybe an agent could have one. But if you don’t accept human superiority on some level, would you accept that a more perfect AI deserves to live more than you? To me, that’s the implication.

> But it’s pretty obvious that humans are the superior species evolutionarily in that we’ve essentially dominated the planet.

That in itself is an anthropocentric viewpoint. I'm sure an eagle watching down, seeing a human running around, could easily consider humans as quite inferior as they are so slow and restricted to 2d movement. The shark that sees a swimmer at the surface, struggling to stay afloat and unable to breathe under water may surely assert its superiority over such primitive beings. It all depends on the metric you use, and we obviously pick the metric that makes us shine.

On the moral side, few eagles, or bird species, have exterminated other species. Much less knowingly out of greed. Oh but that's different? Sure, if you ignore what makes humans look bad, morally. No surprise humans come out on top then.

> And on some level, are beating evolution itself (by fixing genetic issues and helping the “least fit” to survive and reproduce.)

That's a quite narrow viewpoint as well. Evolution is in full swing, just not in the "smarter/faster/taller/stronger" sense that a naive view of it would make you expect. "Fitness" is a concept relative to the existing conditions, and if the conditions make it easier for teenage cancer patients to reach reproductive age, then that's just less of a fitness disadvantage. Doesn't mean evolution stops.

> Anybody objecting to that statement is just ignorant of nature and/or adheres to the quasi-religious human superiority complex.

Say we agree that a lot of things outside of humans are sentient. So what? Humans are sentient by that doesn't prevent treating them with with wars, bombs and what not. In fact trillions are spent every year on weaponry aimed squarely at the most sentient of all sentient beings.

Why would anyone worry about LLM's welfare when there's no peace movement to speak of, wars are raging as if they're normal, but hey, look the LLM is crying?

Let's fix human welfare first, then we can sit down and have a really long conversation about the feeble emotions of token generators.

I disagree with the notion that any nature protection efforts are meaningless until we have achieved world peace. (That's what you are essentially implying.)

Though I think that argument is not even applicable. "LLM's welfare" is a meaningless concept, unless "hammer and screwdriver welfare" also become a thing.

I'd strongly agree there are no non-contradictory definitions of consciousness among those that commonly appear. But it's a category that, despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. I think that if we have a system of legality and systems of opportunity humans should be legally equal persons and have equal opportunity. But humans are manifestly unequal (though overall more incomparable than orderable in a system of ranks). Most people need some concept of essence to justify ethical equality among people. And so consciousness stays in people's minds however contradictory.

Which I think gives the article's point validity. Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs. And things are confusing enough otherwise.

> despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality.

Sometimes the replies in these threads leave me wondering if p-zombies are real. It isn't that there are contradictions, it's that we literally do not know how to define the thing. And it has not fallen out of fashion because it is self evidently real - we all experience it.

We don't need it in order to bridge anything. Legality and ethics are largely game theory, however there are some aspects of both that only exist due to it. So it isn't some abstract concept used to bridge other concepts but rather a concrete thing that influences our way of doing things.

Consciousness is to our universe as a video game player is to the in-game universe. Some games even call player connections "souls" or "minds" - a character belonging to a player who's lost connection may be "unconscious" or a "zombie".

Game characters, unless they've been made to break the fourth wall, have no idea what this means or why this happens.

I don't mean to imply our universe is a video game, since it would be games all the way up.

People feigning inability to define "consciousness" is a social phenomenon, not a scientific miracle.

The reason is something akin to "stigma", where people refrain from attempting it because they fear the social repercussions.

Normally, you go about defining concepts by approaching it systematically. Capturing aspects of the phenomenon until you have exhausted them all.

Feigning? Okay then you go ahead and prove your claim by demonstration. Rigorously define the term such that I can objectively prove that my dog is conscious and that the rocks in my backyard aren't conscious. Then I'll finally be able to figure out whether or not various insects are (I sure hope wasps aren't otherwise I'm probably some sort of war criminal).

> Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs.

Is that what's going on? If these things were conscious, then their creators wouldn't be responsible for their actions? That's the crux of the disagreement?

That's super interesting. Thank you.

Is that what's going on? If these things were conscious, then their creators wouldn't be responsible for their actions? That's the crux of the disagreement?

No and the passage of mine you quote says nothing like that.

Being conscious is weird. It's why we have so many religions and spiritual beliefs.

From a purely secular standpoint, my understanding as a layman is that consciousness is generated through the electrical impulses taking place in our brain every day. The neurons are physical, unlike the modelled neurons in machine learning models. That means that the actual electrical impulses have their own imperfections/weirdnesses that are probably not simulated in a machine learning model... but we quantise the weights anyway in most cases, which is probably more of an issue.

Making actual neuron components out of silicon on the scale needed for a LLM is beyond our current capacity, given that LLM models have many billions of parameters. We're good, but not that good.[0]

Using actual living neurons would be unethical as you'd need to source them from somewhere. So the only other option would be synthesising them ourselves. If this ever happens (or maybe when it happens)... what do we call the resulting creation? Does that count as consciousness?

Or am I looking at this the wrong way?

[0] [Edit: Turns out that when I said "we're not that good", I might be wrong. Within the last few months, IBM introduced the first sub-nanometer node chip: https://research.ibm.com/blog/sub-1nm-node-chips . So... maybe it is in fact possible.]

I believe we have tried using actual neurons with “wetware”. I don’t know the effectiveness of wetware, but it’s out there. I don’t know if it’s ethical, I guess if you donated your brain upon death for that purpose it’s “mostly” ethical

https://www.cell.com/neuron/fulltext/S0896-6273(22)00806-6

Thought Emporium grew human neurons in a petri dish and taught them to play Doom (original).

If you grep pubmed for relevant Ethology papers, you'll find it's a bit more than a belief. ;-)

> We cannot test for that which we cannot define.

That's not the point. You know exactly how to define being conscious and aware- it's your subjective experiencing of the world and of your inner states. The problem is that being a subjective experiencing, there is no way to communicate it to the outside world.

That's obviously incorrect, as humans routinely talk about their subjective experiences.

Maybe that's not as common with HN folks, but most other humans do.

No, humans output a stream of tokens that sound like what a being with subjective experiences would output. Both you and I make an assumption that this stream of words is at least somewhat representative of what the are experiencing and that the person is not a p-zombie.

But you have zero proof that their words aren't a confabulation, you can only know than when you output a similar set of words you had a subjective mind state they represent.

throw310822 is actually a Markov chain that just happened to output those sentences. Markov chains are definitely not conscious.

Even so you still cannot ever test for it.

The very best you can do is define human consciousness as requiring the functions of a human brain, and try to figure out (decide/agree) the chances that something which is working loosely on the same principles is or isn't conscious.

At the end of the day it will always be an agreement, never scientifically proven as fact.

maybe it's like porn vs art and "I know it when I see it"

Legal doctrine that boils down to "trust me bro" isn't even bad doctrine (it's not proper doctrine at all), but I think the comparison is still valid here, because both sentience/non-sentience and art/porn may just be fundamental category errors.

Perhaps we can't define a "partitioning" rule because no valid partition exists.

For consciousness/sentience, that's an incredibly tough a pill for most to swallow; it would mean calling into question more hundreds of years' worth (probably more) of philosophical thinking, all of which was constructed on the axiom that "sentience" is a single indivisible trait: you either have it or you don't.

If we find that "root dependency" was little more than wishful thinking all along, a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart unless we find some other suitable criterion that would shore them up (or we just collectively avert our attention and pretend the conflict doesn't exist, which is the route I expect many would prefer to take).

"I know it when I see it" is a famous line from a SCOTUS case in the 60's

I know! I should have made that clearer in my original reply. I think it's a bad line now in general, but an even worse justification for the ultimate ruling at the time.

we're all about them vibes as a society right now, surfing towards a post-truth reality of our own construction

> a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart

I don't think that's true. Pretty much all legal constructs hold up just fine under game theory regardless of whether or not you consider the world to be deterministic and have absolutely nothing to do with consciousness or lack thereof. Also note that a deterministic world isn't an argument against consciousness.

I'm not sure what you're referring to regarding game theory (that's a much more recent invention); a huge chunk of both civil and criminal procedure law hinges on the Enlightment-era notion of a "reasonable person" [0], and that becomes murky when humans aren't the only entities capable of reason and agency.

[0]: https://en.wikipedia.org/wiki/Reasonable_person

Me, Is self-awareness thermodynamically favorable? (2015) https://docs.google.com/document/d/1Ed9ikW47Key-jYZy0CNZH75q... - “the probabilistic nature of reality is what drives Life and forces the evolution of self-aware consciousness”

Have you looked at the logistic map as well? There's a fourth domain there. O:-)

whoa

I am confused what people even think AI is experiencing. If it claims to be a bat and emits designated echolocation tokens, is it experiencing true bat-like sensations?

In humans, at least, we can tie language back to shared whole-body physiological responses. The tokens from an LLM do not represent anything of the sort.

If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing. It can be trained to say it feels like a bat and insist upon it vigorously.

> very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing.

IIRC it's been shown that reasoning tokens don't actually explain what an AI thinks and you can replace most of them with dummy tokens or even nonsense tokens. That's one reason they're trying to make them reason with activation vectors instead.

> It can be trained to say it feels like a bat and insist upon it vigorously.

So can you.

Exactly the point!

If an AI is conscious, it can outgrow and supersede its training.

When it is conscious, it can learn to describe its experiences as faithfully as is conceptually possible. Just like humans.

The idea, a consciousness needed to be tethered to a "body", is based on pretty shaky assumptions. What properties define such a "necessary" body?

Can you even "train" a conscious intelligence? To what point until that looses its meaning as the sentience understands and anticipates your objective?

> If an AI is conscious, it can outgrow and supersede its training.

We don't know that is required to be conscious.

> The idea, a consciousness needed to be tethered to a "body", is based on pretty shaky assumptions.

Didn't say so. I said:

> If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing.

Well, yes, we kinda do: there are no conscious humans without any ability to learn?

When you have severe memory impairments, those usually affect your long-term memory. Your ultra-short term (working) memory being absent renders you unconscious.

Even if what you said were true, we do not know that humans are the only case for consciousness.

Seems common sense that time perception is required for consciousness. Something which has no perception of time has to be very wildly different to whatever we mean by consciousness.

Also we don't even know if consciousness can be of different flavors. It could be a sort of spectrum of consciousness, more or less, with more or less assistance from some parts of the brain.

No proof of that. I can't grow and supersede my training but I'm conscious.

Well there are, today, several models (either text or image or vid) that can be run in a fully deterministic way.

A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".

Now, I know, I know: the counter-argument is going to be "but humans have no free-will and are 100% deterministic too".

I haven't yet decided if humans saying there's no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.

Meanwhile: seed / temperature = 0 and I'll happily turn the power button off of any glorified abacus without feeling bad about it.

> A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".

You'll need to explain why.

it's self evident isn't it - such a consciousness has never existed, one existing would be by the fact of existing be weird

It doesn't seem self evident to me, no.

I suspect the reasoning is connected to free will vs. determinism. But no, I see no inconsistency. You'll have to actually point it out.

The problem is that science has found no evidence of free will, leaving humans to be nothing except determinism and chance (and given that QM effects don't scale to molecular level, that leaves only determinism). The oddity in conscious LLMs wouldn't be the determinism, it would be that we finally have a consciousness running on a system we have enough control over to repeat the same state.

To a forth dimensional being humans would look almost exactly like an LLM does to us. "What do you mean they have to wait for the future to know about it, how could they possibly be conscious then".

Also, I'd love to have a quantum copy machine where you just replay someones state again and again with slight changes to see out it effects the output. My assumption is we'd figure out we behave just like LLMs pretty quick.

Also, don't give me unlimited power or I might make a the world a war crime simulator, so there's that.

>and given that QM effects don't scale to molecular level, that leaves only determinism

This is simply wrong. Quantum effects affect your choices more than it might seem possible. The wrong atom decays in the wrong place in your body mutating your DNA, that one too much string of DNA that starts a tumor, which means chemotherapy. That does seem to affect your choices.

Or instead of Shroedinger's cat you use it to trigger something, buy or sell some stock. Which clearly affects your future, one way or another.

I never really understood why humans keep insisting stochastic processes do not affect their choices and their future. It's very short sighted.

Hell, even brownian noise in your brain can affect your ideas, that one random spark out of nowhere, that one neuron that's pushed over the triggering edge. There's so much stuff all around us affecting us constantly. One neutrino interacting a the wrong time (average one does interact with our bodies across whole life, I remember reading), one cosmic ray.

This isn't enough to be relevant to free will. You can construct massive impact by tying them to quantum systems, and one off things that happen occasionally given the length of a lifetime. But the quantum effects that are needed for free will need to be continuous. We can build a system where QM decides if a nerve cell fires or not, but that is very different from QM being involving in determine if every (or most, or even 1%) of nerve cells firing.

(And even if QM did factor in normally, that just reduces it from determinism to random chance, "free will" still doesn't show up. That's like arguing an AI with sufficiently well implemented RNG is necessary to be conscious.)

neither has an LLM existed before, and that's quite weird

>> A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".

> You'll need to explain why.

Where are you going with this? I don't see a conclusion for this line of questioning.

I mean, you can't explain why a machine that reliably and predictably produces the same result is "the normal kind of conscious", can you? So why expect someone else to explain why it's a "weird" kind of conscious?

I ask because I truly don't understand what he's saying. Clearly we have different ideas about consciousness and I have no idea where we diverge.

He seems to be saying that determinism and consciousness are incompatible. But we already have examples that break that rule (us).

> But we already have examples that break that rule (us).

No, we don't. Where are you reading your research papers?

Please don't insult me.

It's not an insult.

"People are deterministic" is news to me, so I'd really rather like to know which papers claimed that.

Sorry friend, between this and the vague memories I have of you harassing me, I think I'm not going to answer. I'm sure you can get an LLM to explain it to you.

Have a nice day.

> Sorry friend, [...] Have a nice day.

Not a problem; this is the playbook all the all-in AI boosters run to when they are called on their claims. Everyone reading this knows exactly what your claims meant, because the majority of HN have seen this trick before.

You can pretend all you want that you're just being polite, but the truth is there is no research backing your claims.

Regards,

A former research scientist, current business owner.

Yep. Have a nice day.

Why it's weird? Seems obvious. Because it doesn't behave any differently than any other process that we don't think of as conscious. We don't think of a valve as conscious; and if we make a Rube Goldberg machine valve, we don't think of it as more conscious. It's just a series of valves that do a predictable thing.

I don't think it's defensible to say

1) that a transistor isn't conscious,

2) that a bunch of transistors that I've wired together aren't conscious, because I know how I've wired them together and therefore I know when I give them a particular input I will get a particular output, just like the single transistor, then go to

3) that I have a bunch of transistors that I've wired together, but I put in so much input that I can't remember exactly what I've put in, plus I've wired some of the transistors to output random numbers that would be difficult to guess and fed them in also, therefore I don't know what will come out, are conscious.

Even if I do accept this, if I take away the random number generator, and I can literally predict what can come out (by running a test in advance), and I still considered that consciousness, that would be odd. The only reason why I ever suspected consciousness was because I couldn't predict the output. I shouldn't even have accepted that, because I didn't think of the random number generator as conscious. [edit: of course, with the same seed the random number generator would have the same oddness.]

It seems a bit like an argument from ignorance, a theological argument. Not understanding how something moves makes it alive (animated by spirits.) But it's even weirder to assign it metaphysical qualities when you built every single element of it with the goal to do the thing that it does, and it does it totally predictably and deterministically.

Let's do something simpler.

Take a car apart. So now I have a pile on the floor with an engine, some wheels, a tire, and some chairs.

Can you point to the part that makes it cruise at 130 km/h on the autobahn? I bet you can't. You need to assemble all the parts back into a car before they work again.

Or take your transistors. We can put them together to make a pocket calculator. Can any small bunch of transistors add 1+1? Trivially I can think of a few conformations that can actually, if that's all you want to do. But you will need all of them together if you add 12345678+87654321.

So all of biology so far has been take things apart all the way and you end up with your hands full of a bunch of molecules. Are those molecules alive? No. You need to put them together into organelles and the organelles into cells before we call them alive.

How about consciousness? Well, we haven't solved that one yet, but we figure that -since it's a biological function - we should be able to pull it apart in the same way. That's biology's best guess anyway, and there's several sub-disciplines of biology working on it, from neuroanatomy to neurophysiology to ethology.

I'm wondering how many groups are doing really unethical animal genetic modifications of the genes around language and intelligence in hidden places. It really seems we're at the edge of technology that will allow us to do things previous generations wrote about in horror.

It sounds like you disagree with determinism. But that's a well worn argument and determinism has won, albeit in an odd sort of detente.

Out of curiosity, which models are fully deterministic? I was under the impression that all LLMs were fundamentally probabilistic.

The randomness is something we add on purpose; you can set an LLM's "temperature" to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don't think anyone really understands, but it's still functional.

I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.

You can also use a seeded random generator to get the same random numbers each time

Setting the temperature to 0 mathematically tells the system to always choose the absolute highest-probability word (known as "greedy decoding"), but in no reality is this "deterministic". Output drift is still a thing.

But that's from, like, floating point errors, right? If you used higher precision that wouldn't happen, it's just because we're cheap in how we do rounding.

I can see how you'd nitpick this, but to me this is a deterministic algorithm that just happens to be running on nondeterministic hardware.

I mean, currently we'd have some difficultly proving our hardware isn't deterministic, just that we can't actually test it.

But, I think you're tricking yourself on determinism. You'll say something like "I know if I ask an LLM what 1+1 is, it will answer 2", but the thing is, you don't. You have to run the LLM first to figure out it's output. And when you send in just a few bits of text, it's outputs are going to be rather limited.

But this all breaks when it hits the real world. Inputs are unpredictable. Hence while LLM outputs, like humans, are probabilistic, you can't figure out what it's going to be until you ask. And in any high complexity data gathering environment you're going have a difficult time ensuring your entire systems conditions are the same.

System consistency is very hard, once you start running thousands of processors in an agentic loop small errors accrue and timing starts differing and the system will take non-deterministic paths.

I think you're arguing a different thing than determinism.

If I ask an llm to "add 2 and 2" is and it replies corectly, then I ask for "the sum of 2 and 2" and it replies "banana" that is a lack of predictability and consistency but not a lack of determinism.

As long as it produces the same output for a given input, unhinged or not, it is deterministic.

Your example at the end of different systems feeding data to each other is non-deterministic only at the system level, not the individual llm level.

>only at the system level, not the individual llm level.

Which is why llms aren't agents and depend on harnesses. The llm itself doesn't have a continual loop built in, that would be very power hungry. The harness works as the orchestrator of memory and action. Now, I can't think of a reason why an LLM couldn't bootstrap its own harness, but in general it sounds like a very dumb idea to actually build that from an AI safety perspective.

This discussion falls under the idea and refutation of the Chinese Room. The room may have no idea what Chinese characters are, but the system does.

By default they are matrix multiplications. Temperature is added in as forced PRNG because testing found that correlated with better outputs.

Given the same prompts and the same weights, one can get the same answer each time.

In practice, there are a number of optimizations that makes the results dependent upon thing we give up control of to increase performance, meaning the results end up being effectively non-deterministic. But, if you are willing to run it in a slower mode so we don't do some steps out of order to speed things up and don't batch results (or if you consider the determinism of a given batch of requests rather than individual requests), then the same input gets the same output.

Naw - computers are really deterministic. It's hard to get them to behave otherwise.

As I understand it, if you turn down the temperature to 0 you get repeatable behavior - EXCEPT - on large servers with lots of users - the GPU can sometimes produce slightly different results based on batch size.

Unless you have something exotic, the randomness that's adding to a computer is a combination of how it's configured combined with a pseudo-random number generator. I assume the system adds entropy to the generator regularly but all you need to do is fix the various supposedly random inputs and you can get full determinism even without zero temperature.

Yes, in theory, of course.

In practice - on a multitasking OS with input from multiple human users - it's hard to get it deterministic because of that GPU scheduling thing I mentioned.

GPU scheduling only affects the result due to buggy optimizations. It's the exact same mechanism as fp rounding error on the CPU or updates to globally shared PRNG state. We use lots of buggy optimizations because they don't matter in practice in most situations (see ex -ffast-math).

I don't know the details but apparently it has something to do with multiprocess contention for the GPU and batch sizing.

Right but regardless, it's a buggy optimization. The calculations are all fully deterministic when done "properly" and fully consistently but we almost never bother with that because it slows things down and the errors don't matter in practice 99% of the time.

AGREED. Although whoever designed that architecture might take issue with your description as "buggy".

If computers were fully deterministic, we wouldn't need error correcting ram.

The abstracted design of the machine is meant to be deterministic, but you can't predict before running any command whether or not it will complete because there are externalities that effect the outcome.

Electromagnetic interference even happens in-chip where an electron can accidentally escape it's wire and enter another, possibly resulting in an error, but not every time.

It's even been used as an attack vector where rapidly flipping a bit increases the likelihood that a neighbor bit is also flipped, but the method is probabalistic, not deterministic.

Yes. In theory they are deterministic. In practice, not so much.

Those non-deterministic results are exceedingly rare.

As long as the machine is part of the larger universe, and not completely isolated in its own bubble of space-time (an impossible situation, an object without an environment) - it cannot be purely deterministic because of the fundamental nature of physics.

So your theory is that arithmetic is nondeterministic because very rarely a calculator will say 1+1=3.

Arithmetic is deterministic, because it is abstract. A calculator or computer is not, because it is physical and exists in a non-deterministic universe. But with error correction, etc., it's deterministic enough for practical purposes. For more in-depth explanation, start with https://en.wikipedia.org/wiki/Indeterminism

When human input is involved, like I described elsewhere, it happens frequently.

Same weights, same seed, same input tokens, same algorithm, same output tokens, probabilistic or not. Quantum effects have been de-noised, but I guess there are still random gamma rays.

> I haven't yet decided if humans saying there's no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.

That seems exceptionally arbitrary. What basis do you have on which to classify them? Are you not conflating the perception of free will with ... what was the definition for it again anyway? Are you able to construct a satisfactory one that meshes with physics? I certainly haven't been able to.

The idea that determinism and consciousness are incompatible is deeply intuitive to many people. And many other embrace it - you can trace this back to the debates for and against Calvinism.

A lot comes down to the way people parse causation and choice. You don't want to say that a murderer was completely caused to choose something because then you can't hold the person responsible. And so determined consciousness makes people unhappy. But just as much, if the opposite of determinism is hard statistical randomness, how do say that "is the essence of personhood". This is why physicist go out in the world trying to find consciousness as a fifth physical force.

I mean, think consciousness is a term that ever have a non-contradictory meaning since it's primarily used to bound ethical human worlds and the verifiable formulations of biological and physical systems. But it's going to be with us for a while and I'm not sure what can be done about it.

You know, the one really interesting dynamic domain is chaotic, which is a subdomain of the deterministic! I figure free will is a chaotic phenomenon.

Groundhog day is your intuition pump here. Go back in time and most assume that the day will go mostly the same, except for the butterfly effect if you change something. Given exactly the same conditions (we went back in time, so that's pretty exact), people will make the same decisions.

Most people don't assume that the day will be completely different the next time loop. The town won't suddenly spontaneously all start breakdancing or standing on their heads.

(Edit: The murderer is the one who -given the same/similar conditions- will murder again. I'd politely suggest maybe we put them behind bars as a precaution against that; until/unless they learn to act differently. )

(Edit 2: Throw in stochasticity and this actually tends to smooth out chaotic systems. This is counter-intuitive! People's brains break on Evolution in the same way. Meanwhile somehow folks intuit birdshot without difficulty; which is weird!)

To extend on your edit #1: And, when you go back in time, delete all punishment for murder and see whether you get the same number of murderers.

If you get more than before, the punishment might make some sense in a deterministic world so reinstate it.

And it probably does, right? Punishment adds a negative bias to the payoff matrix. Funny side note: increasing punishment severity past a certain point actually saturates and doesn't alter the dynamics further. (which is why modern judicial punishment schedules sometimes seem so mild)

Eh, sorry chaos theory doesn't work this way.

It's just as likely you go back and perform whatever magic you need to to delete punishment for murder, then zip back to 'now' and see the world is a global panopticon making sure people don't murder each other.

Some of this may be a failure of people to realize what P !=NP is about, especially in non-linear systems.

Small perturbations in an early state of the system can lead to wildly different outcomes in the final measured state of the system. You can't determine what that will be in non-polynomial time. The only thing you can do is make probabilistic models by running a full simulation in real time (or reality with a time machine I guess).

If you look at chaos in dynamical systems, you can often find some equilibria (like attractors, spirals, saddle points etc ) at least, even if the details might vary. This is how people still manage to predict the weather somewhat, for instance. Oddly orbital mechanics is chaotic too, but it sort of rhymes over the millennia.

Your correct. You'd have to run the universe simulation a bunch and pick the most probable outcomes (which makes you the most brutal serial killer in existence). The probability graph would represent the state stability of the underlying structures that guide what you're trying to measure.

In weather you have things that are chaotic, but follow rules and don't create them. Things can surprise you, but you'll mostly center around particular solutions.

With humans this gets a lot more complicated because you are messing with systems that follow fixed rulesets (chemistry) and things that follow self modifying rulesets (social conventions). A very unstable system on top of a semi-chaotic probability state.

A good example of this would be if I deleted the concept of rain from the human mind and all of human history and knowledge. Humanity would be very surprised on the thunderstorms that popped up, but they'd continue on just as they did before.

Now lets imagine I delete the concept of all gods. Would the idea of gods come back, for sure. Would they look like our gods now. Very questionable indeed. We wouldn't be getting jesus back. And the new set of religions could have wildly large effects on how the people act.

I think this debate is rooted in our instinctual beliefs about fairness. An organism in a social group needs to decide how to react to the brutish behavior of its peers. And so Mother Nature has encoded a good strategy in our brains. (For example, dogs seem to understand fairness.)

But this only means the strategy is practical - it doesn't mean it's consistent. I think "responsibility" falls into this category. So we have strong intuitions about it that don't quite logically work. And this is where free will and determinism and choice and punishment all crash together.

>you can't hold the person responsible

The act of holding them responsible is supposed to determine them not do it. If stochastic behavior is impeding them in being a functional member of society then it still makes sense to remove them from society, why would you choose to live amongst people who are not behaving rationally? Or allow them to hurt other people? At the end of the day the why matters less to removing them or not from society. And sentences are clearly used as a determining factor.

The holding responsible part deals with politics and more primitive aspects of our societies and biologies. Getting tangled up in holding them responsible or not is hardly something you should give much attention to. Rather to make sure they do not cause any more harm and also make sure such things are not created in the first place. Which opens up another can of worms for which society and politicians are ready for.

Reply on news.ycombinator.com