Hacker News Reader

Comment by kilgnad | Hacker News Reader

>In defining what it means for a mathematical structure to be of conscious experience, our proposal does not answer the question of what this mathematical structure is.

Nothing is answered in this article. We simply don't know.

This article is up here because nobody is sure whether or not chatGPT is in any way representative of an aspect of our conscious experience.

Is chatGPT a mathematical model of consciousness?

We know it doesn't represent the whole experience because... it has huge flaws... but we aren't sure whether it represents an aspect of what we are.

Disagree or agree, we have something in our hands and it could be it... but at this point we don't know.

View on HN

> Is chatGPT a mathematical model of consciousness?

Unless we reduce models of consciousness to purely stochastic processes, this is trivially false. And no one seriously reduces consciousness to probabilistic pattern recognition.

> We know it doesn't represent the whole experience...

It doesn't represent any experience. ChatGPT doesn't "experience" things as conscious things do. It's not even comparable to the experience of a simple invertebrate, for example.

That's the (hard) problem of qualia. You can't say for sure that ChatGPT doesn't experience anything. And I can't say for sure that you do, either. It's trivial to dismiss it on the basis of "well obviously it's like that"—far more difficult is to say something about why it's not like that.

I think consciousness has to be the product of an algorithmic process. I think this process can be summed up as "prediction". I don't think there's some secret quantum/soul/magic aspect to it. Which is why I think it's plausible that transformer models may capture some aspect of what we refer to as consciousness, though I don't quite buy it at a gut level.

You say that "it's not even comparable to the experience of a simple invertebrate," but that's just you writing down what your gut is telling you. What is the experience of a simple invertebrate like, exactly?

>You can't say for sure that ChatGPT doesn't experience anything. And I can't say for sure that you do, either.

The question rattling around in my brain tonight is - if some, many, people don't develop a theory of mind as young children, how many of the people around me are like this? Is it just the philosophers?

It seems of much more moment than the essence of ChatGPT.

Given how many people in this thread don’t seem to grasp the idea of qualia, and think the hard problem of consciousness to be easy, I’m not entirely unconvinced I’m surrounded by a bunch of imperfect p-zombies.

People without a theory of mind are not p-zombies. They are people who see others as p-zombies.

Given that, I'm not sure how to interpret your comment as it seems to describe an invalid inference.

>Unless we reduce models of consciousness to purely stochastic processes, this is trivially false. And no one seriously reduces consciousness to probabilistic pattern recognition.

Most things in this universe have mathematical models that can describe it. We have models for virtually everything. Thus it is reasonable to say such a model also exists for consciousness. To say what you said definitively without even knowing the meaning of the word consciousness is premature. You don't have enough knowledge to say anything about this.

>It doesn't represent any experience. ChatGPT doesn't "experience" things as conscious things do. It's not even comparable to the experience of a simple invertebrate, for example.

Again you don't know what you're talking about. Nobody can define what consciousness is and nobody fully understands the emergent effects of chatGPT or the human brain. To say what you say with conviction is therefore empty. You don't actually know because nobody knows.

> Again you don't know what you're talking about.

I think that's a bit uncharitable. You're acting like people haven't been grappling with these problems for thousands of years. Searle, for example, makes a pretty good argument why symbol-processing systems will never produce consciousness (you'd have to prove that the Chinese room is conscious, etc.).

It's clear, for example, that a toaster doesn't have any conscious experiences, but a dog does. To say that "nobody fully understands" the human brain is throwing out the baby with the bathwater. ChatGPT is not conscious any more than my (very complex) Swiss wristwatch is.

Philosophers grapple with the problem the same way Greeks used mythology to grapple with the problem of reality. Thunder is not thrown by some god named Zeus. and the impossibility of a mathematical model describing consciousness cannot be explained away by a Chinese room argument that has tons and tons of qualitative holes in it.

If an atom can be described by a mathematical equation that means so can groups of atoms. If groups of atoms can be described math and if a neuron is simply a complex group of atoms then a neuron can be described by an equation. If neurons can be described by an equation then so can groups of neurons. If the brain is a group of neurons then the brain can be described by a mathematical equation. If consciousness is a physical phenomena produced by a mathematical brain then consciousness is a mathematical model.

The only way you can believe that consciousness cannot ever by described by math is if you see some break in the previous chain of logic. Most likely it's in the linking of consciousness entirely with the physical brain. Likely you believe in a soul or something if you don't agree with the logic. Which just means you're using mythology and concepts outside of science to derive your conclusions. Again, lighting is not thrown down from the sky by a god named Zeus.

It's clear a toaster is not conscious. But what chatGPT is, is not clear. Placing it in a category where we aren't sure is reasonable and also it's a category agreed upon by many people. It would be a bit uncharitable to demand that your categorization of chatGPT is more correct then other peoples. Even you must be be aware of the ambiguity associated with chatGPT despite your stubborn stance to the contrary.

> But what chatGPT is, is not clear.

I would posit that it is clear. It's simply a whole bunch of toasters put together (that is, a long list of if-then statements). If you're trying to argue that consciousness is merely a bunch of if-then statements, then yes, ChatGPT might be conscious. But then it's going to be much harder to argue that a toaster isn't conscious, which I thought we both agreed with.

> Likely you believe in a soul or something if you don't agree with the logic.

This seems a bit dismissive, and although I myself am a substance dualist (which is generally seen as a metaphysical position), most materialists aren't mechanistic in their interpretation of consciousness.

>I would posit that it is clear. It's simply a whole bunch of toasters put together (that is, a long list of if-then statements). If you're trying to argue that consciousness is merely a bunch of if-then statements, then yes, ChatGPT might be conscious. But then it's going to be much harder to argue that a toaster isn't conscious, which I thought we both agreed with.

The combination of multitudes of if-else statements forms a macro structure. It is this emergent macro structure that we call consciousness. As stated before the exact interconnect of this macro structure can't be fully articulated yet.

>This seems a bit dismissive, and although I myself am a substance dualist (which is generally seen as a metaphysical position), most materialists aren't mechanistic in their interpretation of consciousness

My arguments are based on likelihood derived from logic and evidence meaning While we don't fully know if a soul fits in this framework, we certainly do not yet have the capability to derive it's existence/likelihood from evidence or first principles. Thus there's really nothing to talk about as it's an argument outside of known logic. What is the point of arguing about topics we don't know about? There is no point. I'm being dismissive because there is literally nothing left to talk about for either of us. Logic and evidence is also the only shared principles that seem consistent across individuals so best keep discussions in that realm fully.

> The combination of multitudes of if-else statements forms a macro structure. It is this emergent macro structure that we call consciousness. As stated before the exact interconnect of this macro structure can't be fully articulated yet.

It can be, and it's called the theory of computation. It's been extensively studied for the past century and a half, and, all-in-all, it's not even particularly strong. For example, most real numbers aren't even computable, Godel showed that consistent theories can't even prove their own consistency, and so on. In my view, our mind does not seem to compute things (though some, like Dennett and Putnam, would disagree).

The claim that our mind computes things is a pretty strong one (and if correct, I think we'd all probably be much better at math). Penrose argues that whatever the mind does, it's (at least in part) a fundamentally unsound operation; this to me sounds way more correct than the idea that our mind operates algorithmically.

>It can be, and it's called the theory of computation.

It's the most likely outcome given what we know of the physical world.

> Godel showed that consistent theories can't even prove their own consistency, and so on.

Godel showed a flaw within logic itself. But it's all we got. If you're operating outside of logic then we don't have a consistent logical framework to use for generating conclusions. Without this, like I said, the conversation is pretty much over. There is nothing left to say.

Can you explain how you know that neither a toaster, your wristwatch nor ChatGPT are conscious? If being conscious means being conscious like a human this would be obvious but maybe the toaster is just conscious like a toaster. Is a bug conscious?

Internally we know we're conscious because we just do. We experience qualia, even though we don't know what "we", "experience", and "qualia" really mean.

Even so. For us, consciousness is just there.

Externally we attribute consciousness to certain behaviours, including (but not limited to) self-reference, goal-seeking, physical and social responsiveness, use of language, memory, and so on.

These two things are not the same. We assume other humans are conscious in the same way we are to the extent that they perform consciousness. We can't be sure, but for human interactions it's a workable proxy.

A toaster, a watch, a rock, and a web server don't perform consciousness in any way at all. They may have some kind of metaphyiscal awareness, or not, but if they do we can't see it, and without a comprehensive theory of consciousness it's parsimonious to assume they're not aware.

ChatGPT performs some elements of consciousness, but not others. It happens to perform the elements AI researchers consider important - specifically use of language. But it doesn't have a memory for previous conversations, it doesn't show any independent goal seeking, and so on.

It uses "I..." and a vast amount of human training to mimic the appearance of consciousness, and plenty of humans seem to want to believe that's enough.

But the essence of the hard problem is the difference between "performs the actions we associate with consciousness" and "is conscious in the same way we know we are."

You could argue that practically there's no difference, and from one point of view that may be true. But that doesn't work if you want to understand what's happening rather than just assuming in a "well obviously..." kind of a way.

If only because historically in science "well obviously..." has been consistently wrong.

They may have some kind of metaphyiscal awareness, or not

Another option is there is some kind of metaphyiscal awareness everywhere, and it's only coincidental that objects of class "human" eventually convince themselves into believing that it's equivalent or highly related to their own thinking process. Unless we exclude this option somehow, AI can't be definitely seen as simply rolex, because it does that too.

Currently AI is discontinuous and passive, but is that even required?

without a comprehensive theory of consciousness it's parsimonious to assume they're not aware

This hypothesis creates more entities (one island of awareness per complex enough object), so why not the other way round?

It's clear, for example, that a toaster doesn't have any conscious experiences

So anthropic. We can agree that it is on the opposite end of a spectrum though. A spectrum that we define out of a single point (“us”, and mostly “I”). We can also say that a toaster is a part of your consciousness instead, like e.g. your arm but less noisy. But then you are a part of a dog’s consciousness as well.

Consciousness is probably best described as just a field in a complexity space, not a discrete set of localized phenomenons. It has a near-zero value in a physical projection of a toaster’s concept, but is that projection even useful?

Consciousness apparently arose from evolution, which is a funny process of optimization under stochastic conditions. (There's also a good amount of pattern recognition involved.)

Why is it that optimizing for survival can lead to consciousness, but not predictive coding? Nothing here is trivial.

While some people believe that consciousness cannot be reduced to probabilistic pattern recognition, I strongly disagree. In my view, the AIXI model is an excellent model of intelligence, as it can mathematically define a notion of Pareto optimal behavior across all possible environments. This model relies on compression at its core, specifically the minimum description length principle.

Moreover, large language models (LLMs), like GPT-3, are designed to learn how to predict, and prediction is the dual of compression. As explained in this StackExchange post (https://cs.stackexchange.com/questions/130655/is-prediction-...), prediction can be used to compress data, and compression can be used to make predictions. Given that LLMs are trained on vast amounts of data and can predict patterns within that data, I believe that they contain all the necessary components to be considered an intelligent system.

Therefore, I firmly believe that consciousness can indeed be reduced to probabilistic pattern recognition, and that LLMs provide a compelling example of this phenomenon in action.

Your viewpoint is one I consistently have to push against.

Computation is not necessarily experience. I can't emphasize enough how fundamental understanding this is. It seems some just can't grasp this concept.

Would you say a program that echoes to the screen the responses of a hidden human is intelligent? Viewed as a black box, it would certainly seem so. But does it have any internal experience and understanding of anything? Of course this is a trivial experiment, but the point of it is to show, in the most absurdly clear terms, that producing plausible outputs !== being intelligent, much less having a conscious experience.

This example also shows that the perceived sophistication of the input-output transformation is immaterial, unless you want to say consciousness requires a certain internal architecture for computation, in which case I would ask: what architecture is that? No matter the sophistication of an algorithm, it's computation lies on a different axis to that of consciousness. You might say human compute because they are conscious, not conscious because they compute.

Another way to understand this: Let's say you touch a very hot object. You skin, in conjunction with your brain, senses and computes the temperature of the object. But, there is no essential reason why that should lead to an experience of pain, rather than a sensation similar to color, which generally does not produce pain.

> This example also shows that the perceived sophistication of the input-output transformation is immaterial

No, it doesn't show that, it merely asserts it. Consciousness could be a logical property of some perceptual algorithm, like ordered outputs is a property of a sorting algorithm.

> But, there is no essential reason why that should lead to an experience of pain, rather than a sensation similar to color, which generally does not produce pain.

Sure there is, because it's approaching the point of causing damage. Pain happens when sensory precepts approach acceptable tolerance. Pain is like clipping in an audio stream.

> No, it doesn’t show that, it merely asserts it.

I’ve given you an example that proves it.

> Sure there is, because it's approaching the point of causing damage.

There are plenty of damaging processes in the body that produce no pain.

But that’s not even the issue. What I’m interested in is how sensations are produced; it’s not just computation

> I’ve given you an example that proves it.

Except you didn't. The black box you described does have feelings because the human inside has feelings. The fact that every component of a sentient system doesn't have feelings isn't particularly interesting.

> There are plenty of damaging processes in the body that produce no pain.

Because they didn't need to for fitness purposes.

> What I’m interested in is how sensations are produced;

https://www.pnas.org/doi/10.1073/pnas.2116933119

> it’s not just computation

You don't know that.

> Because they didn't need to for fitness purposes.

Huh, damaging process are not relevant for fitness purposes? Wow

Your link is full of speculation, and indeed for consciousness, speculation may be all that can be done, since no one can define it, only experience it.

> You don't know that.

It seems self-evident. If a program computes the temperature of the sun, does it feel the heat?

> Huh, damaging process are not relevant for fitness purposes? Wow

Cancer or heart disease in your 50s does not impact your reproductive fitness, for instance. Way to attach a strawman.

> It seems self-evident. If a program computes the temperature of the sun, does it feel the heat?

The sun isn't computing a temperature, it's computing a nuclear fusion reaction. Also, "feeling" isn't a quality of individual particles to the best of our knowledge, but it is a quality of some aggregate of particles we call a human.

So if your question is whether a faithful simulation of a human would feel the heat from a faithful simulation of the sun, then yes, that's very plausible.

> Cancer or heart disease in your 50s does not impact your reproductive fitness, for instance. Way to attach a strawman

But there are plenty of cancers that can and do occur in ones reproductive years, that start without pain.

> The sun isn't computing a temperature, it's computing a nuclear fusion reaction.

I wasn’t comparing the program to the sun. I was comparing it to the human skin.

> Also, "feeling" isn't a quality of individual particles to the best of our knowledge, but it is a quality of some aggregate of particles we call a human.

Hmmm, now you are referring to particles and aggregates. Wonder what the source of that claim is. At least you agree computation (a process) is not the important thing, but particles (matter).

> So if your question is whether a faithful simulation of a human would feel the heat from a faithful simulation of the sun, then yes, that's very plausible.

Wow, now you're making a circular argument. Whats a “faithful” simulation of a human? How would we know it is “faithful”? What form would that even take?

Let’s say ChatGPT becomes “sufficiently” advanced, and is hooked up to a sensor that records the temperature of the sun. Would it feel the heat? The whole idea is preposterous.

> But there are plenty of cancers that can and do occur in ones reproductive years, that start without pain.

So? The incidence of these cancers is insufficient to impact reproductive fitness in our species. At no point did I claim that all damaging processes must produce pain, I only claimed that all pain originates from evolutionary pressures indicating high chance of damage that does impact fitness. That also does not mean that presence of pain is always directly indicative of damage, because biology is messy and precision has a cost to fitness.

> At least you agree computation (a process) is not the important thing, but particles (matter).

Nope, not what I'm saying at all. The particles are a computation in a simulation, so the information content of the computation is what matters.

> Wow, now you're making a circular argument. Whats a “faithful” simulation of a human? How would we know it is “faithful”? What form would that even take?

A faithful simulation is clearly a simulation that contains all of the information content of a human and replicates the mathematical relationships that govern the operation of a human's mind. Since a human is a bounded volume, it contains finite information per the Bekenstein Bound, and is thus amenable to simulation, unless you are asserting some kind of non-physical dualism.

It seems clear that you don't yet grasp the argument being made so maybe you should reserve judgment, like calling the argument "circular", until you do properly grasp it.

> Let’s say ChatGPT becomes “sufficiently” advanced, and is hooked up to a sensor that records the temperature of the sun. Would it feel the heat?

It wouldn't feel heat in the same way that humans feel heat because its sensor and information processing is not mathematically equivalent to human information processing and sensory systems. That does not mean it wouldn't feel something like its own version of heat. A whole lot of vagueness hides behind your "sufficiently advanced".

If ChatGPT v10.0 did reproduce the mathematical relationships underlying human information processing and sensory systems, then yes, it would feel heat in precisely the same way as humans do.

"It seems some just can't grasp this concept.". Your confidence reveals your lack of knowledge. No one has a scooby what consciousness is or how to measure it. It could be that every atom has a modicum of conscious experience. It could be that consciousness is an emergent phenomena (although I think this claim is unlikely as it seems to me to be making a category error). It could be that it's actually Midi-chlorians. I repeat nobody knows. There are as many theories as there are people who have thought deeply about the problem. You coming along all confident with your debunking mindset and your scientism are just showing everyone who has any understanding that you do not know what you are talking about.

To me, it is those claiming that computation equals consciousness who are displaying hubris.

Think about it, if we accept the theory that computation equals consciousness, then then is little to investigate further. Just calculate something, and voila, consciousness exists.

It is this closing down of inquiry that I push against. My assertion is that consciousness is deeper, it cannot be reduced to computation.

How would you even prove otherwise? If it arises from computation, what is conscious? Is it the algorithm? Is it the electric circuits? It is the physical machine?

It is interesting to note my consciousness is bound up what a notion of what my body is. I can identify the limits of my body and make a distinction between it and my environment. What determines the conscious boundary of a entity performing calculations?

On the other hand, there are many strong arguments against consciousness arising from mere computation, some of which I have enumerated.

> And no one seriously reduces consciousness to probabilistic pattern recognition.

All eliminative materialists disagree. To them, consciousness is mechanistic in some way, so why not stochastic pattern matching which can be made Turing complete quite easily. For instance: https://www.pnas.org/doi/10.1073/pnas.2116933119

> It doesn't represent any experience. ChatGPT doesn't "experience" things as conscious things do.

You don't know that because we don't have a mechanistic understanding of "experience".

> We know it doesn't represent the whole experience because... it has huge flaws

To be clear, we can't yet conclude that these huge flaws are due to its structure rather than the extra sensory data that we have and it doesn't. Doesn't really change your overall point, just wanted to clarify what we can and cannot conclude from the evidence, because people are far too quick to jump to conclusions after seeing the flaws.

Completely agreed. There are flaws. But there are times where the output appears that it cannot anything else other then complete understanding of what was given to it as input.

A big lookup table.

A lookup table with compression. Intelligence is a form of compression.

Compression is necessary for intelligence but not sufficient.

The vast majority of information cannot be compressed and the proof of this is trivial by a simple counting argument.

In any case, it is still just a lookup table. Compression, unless lossy, doesn't change the meaning of the information in this lookup table. It just makes access more indirect.

Hutter might think that intelligence is a form of compression, but I think that's just wishful thinking. It might be my ego speaking to say I think I'm the one that's right. But it's not my ego speaking to say that there's no hard proof to say unequivocally that intelligence is equivalent to compression.

> The vast majority of information cannot be compressed and the proof of this is trivial by a simple counting argument

It doesn't have to be lossless compression. Intelligence is also about being discerning about the information that's relevant, although that's basically the same point.

> But it's not my ego speaking to say that there's no hard proof to say unequivocally that intelligence is equivalent to compression.

Actually I think it's kind of trivial to see that intelligence is a necessarily form of compression. Abstractions, concepts, theories, these are all lossy descriptions that discards irrelevant details and capture only relevant details to make problem solving via deduction, induction or abduction tractable. And yet, using those concepts, theories, abstractions you can reproduce/predict observations with very high fidelity. That's compression.

Do you have a counterexample of something that we consider intelligence that doesn't fit this model? I suppose it depends what specifically you mean by "equivalent".

>Nothing is answered in this article. We simply don't know.

No, we haven't clearly defined consciousness. For a given exact definition of consciousness, we can determine whether something fits that definition or not. If we don't clearly define it, then we're just talking about a feeling, a qualia, and it's not possible for people to reach a consensus because they're not even discussing the exact same thing when they say "consciousness".

The consistency arises from the fact that even though we don't know the definition, we are in agreement in what is classified as conscious.

We agree that humans are conscious, we agree that rocks are not conscious. This classification is clear and points to a underlying definition we cannot fully articulate. It is more than a qualia or whatever you call it. We know it has exact parameters simply because we are able to perform classification tasks.

The consensus breaks for things like chatGPT. chatGPT straddles some uncertain border that we can't articulate. But in general for the term consciousness we reach consensus by how we classify things as conscious or unconscious.

> The consistency arises from the fact that even though we don't know the definition, we are in agreement in what is classified as conscious.

Clearly there's not agreement in what is classified as conscious, given this thread/article.

>We agree that humans are conscious, we agree that rocks are not conscious. This classification is clear and points to a underlying definition we cannot fully articulate. It is more than a qualia or whatever you call it.

This is a fallacy. People agree that yellow is not violet and green is not violet, but they don't agree that indigo is not violet. Just because there are some classifications on which people agree about the label this doesn't imply they agree upon all classifications. A more direct example is whether plants are conscious or not.

>Clearly there's not agreement in what is classified as conscious, given this thread/article.

There is general agreement. We agree for most things like rocks and humans..there are a few things where we aren't sure, like chatGPT.

But in general humans are in agreement for most things.

>Just because there are some classifications on which people agree about the label this doesn't imply they agree upon all classifications.

Well just because ambiguities for certain things exist doesn't preclude the existence of things that are utterly clear to be conscious or unconscious.

We know humans are conscious and we know humans have all the required parameters to make them classified as conscious. We just can't deduce the exact thresholds and features that cause a phase shift when changed. But we know these parameters exist.

> Nothing is answered in this article. We simply don't know.

It is unclear what necessitates any closed-form mathematical description of 'consciousness'.

Such a concept seems tantamount to a faith claim.

Reply on news.ycombinator.com