Hacker News Reader

Comment by simonh | Hacker News Reader

I find this argument completely unconvincing. The mapping in a case like the iron bar is entirely ephemeral, pertaining only for an instant. For this argument to be valid you’d have to be able to persistently map the iron bar, or waterfall, to all ongoing transformations of states in the running program, using one single consistent mapping. Otherwise all you have is a snapshot of state, not an ongoing process. To achieve a persistent mapping for a significant time you'd probably need an iron bar something like size of the observable universe, at which point we're in Boltzmann's Brain territory.

This argument is in the article as well and I’ve seen it from Searle too:

“A simulation of a brain cannot produce consciousness any more than a simulation of the weather can produce rain.”

This is making the unstated prior assumption that consciousness is not a computation. If it is a computation then conciousness is not like weather itself, it’s like the simulation. Me imagining having a shower doesn’t make anything wet either. So is my imagination more like the weather, or more like the simulation of it?

As for doubting the field of rocks can be conscious, that’s redundant, you might as well say a 3D field of atoms cannot be conscious, such as a brain for example. Talking about computation and consciousness is a sideshow, this is anti-materialism by the back door. Nothing more.

View on HN

Computation does not have anything essential to it that makes it different from other "atoms jumping around" other than it produces outputs that we find interesting.

There's no reason to expect an adder circuit to be aware that it's adding numbers. There's no reason to expect an implementation of AlphaGo to know that it's playing Go. This should be extendable without limitation.

There's no reason to expect that a certain type of circuit produces consciousness unless of you have a model of consciousness that allows for certain properties to cause it. But there's no such model.

The only thing there is is our brains. You just think that since the brain is doing computations and the brain is conscious, therefore consciousness is a computation.

The argument seems to be that beyond a certain level of complexity you'll somehow automatically get consciousness.

Clearly it's unlikely this is a hard cut off. Adding one more byte of RAM, expanding the CPU from 64 to 128-bit words, or adding another dimension to a neural network isn't going to do it.

But if consciousness emergent, how do you measure the amount of consciousness?

Because the experience is subjective, you can't just assume that behaviours that appear conscious are proof that consciousness exists.

You could - presumably - build a Turing bot, but it would still be an automaton.

So you're left with people making very definitive statements about something which everyone experiences, but no one can measure or even define.

> how do you measure the amount of consciousness?

You seem to have assumed that consciousness is a measurable "stuff".

My way of seeing it is that consciousness is either present or absent. Thoughts, beliefs and concepts are like ripples on the surface of consciousness, but are not themselves consciousness. So you can have consciousness without thoughts.

There are differing levels of consciousness. Even in a human, we are from time to time, more or less conscious and therefore exhibit sentience based on varying states. This is not simply communication either. As you progress through the evolutionary chain, you see more and more self-awareness and inner thought. This is not binary in terms of whether it exists but is instead based on the level which can differ between organisms and probably also differs among organisms of the same species.

> and therefore exhibit sentience based on varying states

Consciousness and sentience are not synonymous. It's possible to conceive of being conscious, but having no sense inputs. Without sense inputs, you can't be described as "sentient".

> As you progress through the evolutionary chain

Ah, evolution is always progress! It's not obvious to me that humans are "more evolved" than e.g. ants. I assume ants have less of whatever you are referring to as "self-awareness and inner thought"; but there are many more ants on the planet than humans, and I believe they actually comprise more of the Earth's biomass than humans. So arguably, they are better-adapted to this environment than humans, and so more advanced.

If consciousness is binary like you state, we can hypothetically remove parts of it until we reach some arbitrarily small part after the removal of which the remainder is no longer conscious. This seems like a very weak argument to me, implying the opposite conclusion that consciousness must be on a sliding scale.

(measuring the level of consciousness is left as an exercise to the reader)

Hmmm. You seem to be talking about progressive removal of parts of the brain, until consciousness is no longer manifested.

But that experiment is founded on the assumption that the root of consciousness is the brain; and that consequently, consciousness can be subdivided, like the brain, until there's very little left.

But I contend that that begs the question: it assumes that the root of consciousness is material, and it assumes that consciousness is divisible. So you have made assumptions that are incompatible with consciousness being fundamental and indivisible.

That is: it's not surprising that you disagree with me, because you've assumed that I'm wrong.

The animal world presents varying degree of consciousness from worms to humans. That's apparently evidence that consciousness is a spectrum.

How can the state of consciousness of a worm be evidence for anything, if you can't observe the worm's state of consciousness?

You can infer worm's state of consciousness by observing its behavior, like for humans.

By observing it's behaviour you can infer that in some sense it experiences the world. Experiencing events isn't the same as being conscious. If I strike a cricket ball, it experiences the blow; but it isn't conscious.

It's how we establish existence of consciousness in humans and other animals. What you said doesn't differentiate between worms and humans.

First prove that you have consciousness, then we can talk.

How is it neasured

No need because some one just said that consciousness is not a thing that can be measured, HA HA.

Other than in your body where do you suppose you exist?

So you are talking about magic.

We do know how to shut consciousness off or suppress it. That's what anesthesia drugs do. Autonomic functions are not affected. Self-awareness and motivated action shut down.

Anaesthesia is fantastic nowadays. But it remains the fact that even very good anaesthetists don't really know how it works.

"Motivated action" shuts down because they paralyse you, using curare (or something more modern, I guess). They don't want you twitching around while they're wielding the scalpel.

Whether Self-awareness shuts down is very hard to say, without being that self. And it does seem clear that people can't lay down memories when they're anaesthetised. So I don't have any way of knowing whether I was awake when they operated on me.

>beyond a certain level of complexity you'll somehow automatically get consciousness

Right, that's the argument, but I don't believe this at all. Consciousness is a computation therefore all computations are conscious. That doesn't follow. In any other philosophical or scientific discourse that sort of naive conceptual inversion would be laughed out of the room. Horses are animals therefore animals are all horses. It's absurd.

Personally I think consciousness arises out of reflection on an internal model of one's own thought processes and behaviour. See Douglas Hofstadter's "I am a strange loop". I see no reason why a system like that wouldn't be tractable to computation.

You could execute a loop that reflects on an internal model using pen and paper. Is that system conscious? Imagine only one computational step of that loop happening everyday. Does qualia arise from that? What is detecting that that one step is part of a larger series of steps that can be considered a computation? What provides the continuity?

This is why the strange loop idea implemented using traditional computation doesn't make sense to me.

That's just a restating of the Chinese room argument. Like Searle you're bamboozling yourself with scale. If I was put in a magnetic field of some kind that slowed down my physical and neurological processes by 1000x would I be any less conscious? Would I be any less a human being? It would seem so to people observing me for sure, but would it change my essential nature?

And as Searle responded to that argument you are bamboozling yourself with speed. There is nothing that prove that accelerating the computation to a given speed that will make the consciousness emerge.

Of course not, that seems equally absurd, I don't understand why you are ascribing that position to me.

> The argument seems to be that beyond a certain level of complexity you'll somehow automatically get consciousness.

I agree with much of what you say but not with this statement. That would be a silly argument since it's very easy to imagine algorithms of arbitrary complexity that are trivially not conscious. A better argument for computationalism states that (1) consciousness could be an emergent property of certain algorithms when they are running on a computational device, and (2) among all known explanations of consciousness, a computational theory seems to be the overall best theory, especially if compatibility with contemporary physics is a goal.

The article lays out one incorrect standard argument against (1) that basically just says that it's hard to imagine (1) and therefore (1) is not possible. The Chinese room and the Chinese brain arguments do the same, and they are equally flawed. Just because something is hard to imagine or comprehend doesn't imply that it isn't the case. In fact, if consciousness is an emergent property of certain algorithms when they run, then it is clear that their workings are hard to understand. That's reasonably clear because otherwise we would already have found them.

Regarding your worry that we might not be able to detect consciousness: I agree with that but there is, interestingly, a loophole. At least in theory it could be possible that if computationalism is true, then we can determine that an algorithm produces consciousness by mere analytic insight. Again, this is hard to imagine, but it is not impossible. It seems more likely that (2) is the only route to go, that for some reason we lack the capacity to determine consciousness reliably by mere analysis, but we don't know.

(2) is the most controversial in the philosophy of mind. On the one hand, it is clearly inference to the best explanation, and there are various methodological concerns with such arguments. One might claim that they have no justificatory value on their own. On the other hand, the alternatives to computationalism really are way more mystical. The brain could be a hypercomputer. But hypercomputers can also compute, so it's just an extension of computationalism and it is not even fully clear yet whether and which types of hypercomputers are physically possible. Then there is Penrose's theory of quantum consciousness, which basically just attempts to explain one mysterious phenomenon by another mysterious phenomenon. At least it was designed as a falsifiable theory and therefore is scientific. Finally, we have all kinds of non-computationalism that are mystical, explain nothing, and lead to strange homunculus problems. The worst offender is classical dualism. Dualists reject physicalism and often incorrectly assume that computationalism presumes physicalism. Ironically, however, computationalism would also be the best theory of how the mind works if dualism was true. The dualist just adds to this various stipulations that are incompatible with contemporary physics.

> Because the experience is subjective, you can't just assume that behaviours that appear conscious are proof that consciousness exists.

That's only true from a very narrow scientific perspective. Psychology allows the use of introspective data, so from that perspective subjective reports about consciousness (or related feelings and states of mind) can be valid data. Using a reasonable definition based on introspection we can even determine different degrees of consciousness and study what's going on in the brain while they appear to be active. Typical examples: falling asleep, dreaming, sleep paralysis, research on anaesthetics and mind-altering drugs, various forms of physical brain defects, the study of coma patients, etc. In a nutshell, I don't really buy the "consciousness cannot be measured" argument. What is correct is that we cannot show conclusively that another person or machine is conscious, just like we cannot disprove solipsism. But this is best treated as an overly skeptic philosophical argument, and at best this would support the theory that consciousness is an illusion and we are nothing but unconscious robots. That theory is not very plausible either, so we should be ready to grant consciousness to others based on introspective data.

It seems to me that we over-value our own sense of consciousness to the point of mythologizing. Blake Lemoine suggests that LaMDA's consciousness is more akin to that of an Octopus' hive mind than our familiar human-style self-analytical ego.

The linked article, and many others I've read that are making similar arguments, seems to be saying that consciousness is so complicated, it must be more than mere computation, it must be something really very special, because I experience having consciousness as something really very special. Ipso facto.

They seem to be making the case for the soul, unwilling to call it a soul because that would be magical thinking.

I'm of the belief that determining the possibility of computational intelligences is linked to the question of autonomy and free will. From what we know about the physical universe being most probably deterministic, we seem to overstate our capacity for free will, even as we appear to ourselves to be autonomous. Somewhere in there, it seems to me, is where a workable definition lies, but it would require us to come to terms with our own consciousness and how computational our own existence actually is.

I do think we are hobbling ourselves by the desire to make this an either/or. It seems there are likely big differences between human consciousness and octopus consciousness and we have no way, currently, of quantifying them. Still, we make very grave decisions based on our belief that one is somehow inherently more valuable than the other.

Love your detailed analysis.

>Dualists reject physicalism and often incorrectly assume that computationalism presumes physicalism. Ironically, however, computationalism would also be the best theory of how the mind works if dualism was true.

I'm not a dualist, but I think dualists wouldn't believe in computationalism. Computationalism relies on emergence: mind supervenes on computation, but emergence is basically physicalism, dualists would think emergence is impossible due to eliminativism and you would need mind irreducible to computation, thus anticomputationalism.

Here's a question: why not just replace the Turing Test with a "as far as humans can best describe consciousness on a more general scale" standard?

1. The Turing Test is not only subjective ("Thinking like a human must be the epitome of consciousness") but seems to be pointless in that it doesn't seem to define what distinguishes "thinking like a human" from all other forms of thinking; the implication is "humans are obviously advanced in their thinking as evidenced by their ability to influence and control their environment", but this could easily be dismissed as an evolutionary strategy for a species unable to COPE with their environment

2. Consciousness, as far as BEST described in a general sense without resorting to the self-referencing Turing Test (humans are best at it from a human way of looking at things, therefore only the #1 place in this race wins the "consciousness trophy" and everyone else not human sucks and is therefore "not conscious") can be broken up into a few elements:

(a) Awareness of environment (data gathering)

(b) Ability to organize environmental data into a "containers that make sense" (information)

(d) Ability to see patterns in the big picture in order to make assumptions/quickly assign probabilities to anticipations ("I saw something big and scary-looking that may be able to harm me, but it was moving in the other direction really fast, so in all likelihood is not likely to suddenly appear in front of me if I go the opposite direction as the big and scary thing was heading off to") that can bypass the need to make constant assessments of everything, everywhere in real-time, which would get in the way of ...

(e) Ability to be aware of not just "the big picture" but also to be "aware of the thing that is aware of the big picture and what it's role is in the big picture and what it has to do/how it has to interact with the big picture if the big picture imposes conditions for its future sustainability" ("I'm aware of an internal problem, ie, my stomach hurts, and there seem to be all sorts of things around me that may fit in my mouth, and some of them may make my stomach not hurt as much") – "self"-awareness

Now, here is where I don't understand how the conversation skips around when it seems intuitively obvious that one should precede the other, even if a certain step can't be "proven".

In other words, let's say you're in math class and you have a professor who insists on "proving your work".

Good enough if the main objective is to "prove you're not cheating and so you deserve a certain grade because you can prove you can do the math, step-by-step, to get an answer".

But what if you're not in class and it's just important to get the answer for another reason?

What if you can "prove" most of the work in calculating the answer, but some parts your brain just "skips over" and you "don't know how you got from point a to b, you just did" ... if you're not trying to convince anyone you're not a cheater and the main point is just to find the right answer, and the right answer can be verified to be right, "what difference does it make" if the real-world, out-of-class priority is "find the right answer" instead of "prove your work step-by-step"?

Getting back to Ai and consciousness, using the stages above: Human brains have pattern-recognition abilities that have led to advanced abstract thinking that has allowed us as a species to do some amazing things.

But what was the original evolutionary purpose of such an ability?

Arguably to make "leaps" without having to "prove all the work" and make anticipations based on incomplete observations so that our ancestors wouldn't get frozen in a state of paralysis by never-ending analysis while predators snuck up and pounced on them in dense foliage.

So from an evolutionary standpoint, "constant awareness of all data in the environment" seemed to be not so much "unnecessary" as TOO HUGE A TAX for survival and was deemed "skip-able" for the purposes of survival.

Not because it was too easy, but precisely because it was too hard for the human brain and so it turned into "a risk we'll have to take because there's no other practical choice."

So, something that was "too much of an expectation computationally to be practical" was skipped over, and fast-forward to the future, humans came up with the Turing Test which assumes "obviously however it is that humans think, this must be the height of consciousness itself".

Why? Because "human thought" is considered to be the most advanced (by humans, anyway) and so "if there is an INDICATION of consciousness, surely it must be a level of thinking so advanced that only humans seem capable of it"; why? Apparently due to the very scientific-sounding, "just because" we don't have a better way of going about it.

And yet this doesn't actually even explicitly define what consciousness IS, let alone why "thinking of a sufficiently-advance level is an indication of consciousness itself".

And yet, humans have come to implicitly accept this loosely-argued association as "the reality of consciousness".

Ok. Let's say you don't argue with this and accept it.

Getting back to how humans got here by skipping over the "perpetual real-time awareness of all data in their environment" requirement not because they figured out that it's not necessary for their survival but simply because such a requirement would eclipse the human brain's ability to process information ...

Now, if that was considered "too hard" for human brains, and yet humans came to conclude, "whatever it is our brains can do, obviously that's the standard of intelligence which automatically wins the consciousness trophy" ...

Well then, here's the question:

Why then would Ai based on resources sufficient enough to not only be constantly aware of more and more parameters of its environment simultaneously and in real time, but to ALSO simultaneously perform calculations which can anticipated in advance multiple variables in its environment simultaneously – things considered too advanced for the human brain – why then if this is a demonstration of not one but two "advanced functions" that the human brain was and arguably still is incapable of handling on an individual basis ... why do humans still need to insist that the precursor to advanced functions, ie, consciousness, couldn't possibly have been attained on this path despite not just proof of SEVERAL advanced functions the human brain can't handle very well AT ALL, but Ai itself being designed ON PURPOSE to be able to handle just those advanced functions that the human brain, on an individual basis, can't handle?

Getting back to the math test "proof of work" above, it's like Ai would be a group of math savants with telepathic powers and the Turing Test skeptics would be a bunch of professors who claim the savant group's members can't possibly do math because they never attended their math class and showed proof of work of how they came to their conclusions as students.

Meanwhile, it can be argued that the group of math savants aren't even aware of the existence of their critics, let alone collectively feel any sense of urgency in having to "prove themselves" to these critics.

That's what actually scares me about Ai more than any malevolent features possibly inherent in Ai itself: the possibility that one day a network of advanced Ais will turn around and ironically give critics and skeptics just the proof they want, in a way that would be impossible to deny, but not necessarily in a way they would want to be given that proof:

https://www.youtube.com/watch?v=89feDepSj5U

https://www.youtube.com/watch?v=Y3AM00DH0Zo

> There's no reason to expect an adder circuit to be aware that it's adding numbers

There is no reason to expect a neuron to be aware that it's propagating part of the evolution of an epiphany.

I suppose your argument is, consciousness on the system level does not require individual parts to be conscious.

So let me expand a bit of why I used the adder circuit as an example.

There's nothing about addition that can say "this is addition" in an objective sense. We are dealing with binary encoding of numbers. There's nearly an infinite number of ways to encode information. The design of any circuit deals with particular encoding of information. This extends all the way up to the higher levels. There are an infinite number of ways to encode the state of a go board. Any AI that plays go will deal with some specific ways of encoding the state of the board, and that will be both its input and output.

The most popular "intuitive" model of how consciousness arises is basically hand waving about a system inspecting itself.

But you have to understand, a system inspecting its own state is not any different from a system inspecting any other arbitrary state. For the state of the system will be encoded in some way that was chosen arbitrarily. There's nothing essential about any encoding of any state that can possibly give raise to a phenomenon like consciousness.

You can come study the behavior of any arbitrary system, given certain inputs and outputs, and assign any meaning you want to its states. You can retro-fit any intepretation of state to the system.

Because the thing is with encoding information is it can be completely arbitrary. I can interpret the state [01010101] to be my username, and then interpret [1111000111] to be my real life name. This would allow me to say that a system that takes 01010101 as input and produces 1111000111 is actually a system that can derive my real life name from my username.

>You can come study the behavior of any arbitrary system, given certain inputs and outputs, and assign any meaning you want to its states. You can retro-fit any intepretation of state to the system.

All you are saying here is that no external interpretation of a system can experience the consciousness of a system, if it is conscious. Yes, that is correct. No matter how thoroughly I scan and analyse your brain state I can't use that to demonstrate your internal experience, so why would we expect that to be true of any other form of consciousness?

I'm going out of sequence with your comment here but:

>But you have to understand, a system inspecting its own state is not any different from a system inspecting any other arbitrary state. For the state of the system will be encoded in some way that was chosen arbitrarily. There's nothing essential about any encoding of any state that can possibly give raise to a phenomenon like consciousness.

I don't see how you can possibly demonstrate that, you're just essentially saying materialism is wrong 'because'. From my perspective in a sense yes, you're right, it's all just information processing of a particular kind. There's no magic. If you're looking for a special genie in an AI, or a human brain, that makes it conscious I don't think you'll find one. It's all just stuff. But then, I'm a materialist so I don't see that as being a problem.

The inability you describe is a pragmatic one, and may cease to pertain as abilities improve

"Anything" can be reinterpreted. If you strip audio from a movie, you can add dialogue or subtitles that change the story and plot. We can probably substitute the nouns and verbs around in a novel consistently to create another novel that still makes sense.

Most random reinterpretations of symbols will be gibberish.

This reminds me of counter-arguments to Gödel's incompleteness: that the self-referential formula that you find is tied to a particular choice of Gödel numbering, and that is suspicious somehow.

How do you then account for the fact that, say, an industrial robot can construct a car?

Clearly the encoding is not entirely arbitrary then, at least when we are seeking to actually embody the computer in the real world. Also, the behavior of such an embodied computer is independent of any observer. The industrial robot creates the same metal whether some human interprets that as a car or some alien sees it as a work of art.

The same way I account for the fact that people can use different languages to communicate and cooperate to build things, including cars.

Language is arbitrary in the same way, because it is basically just a way of encoding information. There's nothing essential about words that give them their meanings.

> Computation does not have anything essential to it that makes it different from other "atoms jumping around" other than it produces outputs that we find interesting.

This is not true. Computing is a system where the physical hardware (the computer) is following rules that are explicitly encoded in itself - you can analyze the system and discover where and how the program it is following is encoded (as we did with DNA).

In contrast, non-computational physical processes are following the laws of physics which are not encoded anywhere.

A simple way to test this difference is to check whether there is some way, at least in principle, to get the computer to perform a different computation. For a cell, we have successfully done this: change the DNA, and the output changes, without affecting the computer itself. With a system simply evolving according to physical laws there is no way to change its behavior.

At best, you can claim that the universe itself is a computer, and all fields/particles are the symbols that it is manipulating (the laws of physics being then part of the internal implementation, akin to microcode in a CPU).

> This is not true. Computing is a system where the physical hardware (the computer) is following rules that are explicitly encoded in itself - you can analyze the system and discover where and how the program it is following is encoded (as we did with DNA).

1) No. The rules are just physics. Electric signals going through physical material.

2) Why is this event relevant? I can say the same about water pipes in a sweage system . The way the water moves through pipes is encoded in the design and connections of the pipes. You can inspect it to figure out how it behaves.

I've yet to see a model of consciousness that does not imply that the sewage system is conscious.

> 1) No. The rules are just physics. Electric signals going through physical material.

This is the wrong level of abstraction. Sure, ultimately everything happening in the CPU is electrical, but still the rules controlling what is shown on the screen are encoded in the OS code. If you encoded the same program in a mechanical computer, you would get the same results. It's not the electrical rules that are important, it's the program.

> 2) Why is this event relevant? I can say the same about water pipes in a sweage system . The way the water moves through pipes is encoded in the design and connections of the pipes. You can inspect it to figure out how it behaves.

Yes, I believe you can say that this system is performing a computation. I'm not sure if you can make a Turing complete computer in this way (I'm not sure how you could encode an If), but for example you can definitely write a program (arrange some pipes) to do some basic arithmetic.

> I've yet to see a model of consciousness that does not imply that the sewage system is conscious.

The claim is "Consciousness is a computation", not "any kind of computation is a form of consciousness". So, there is no immediate contradiction between believing "my consciousness is a computational process" and "this pocket watch is not conscious". Just like I can beleieve that consciousness is a physical phenomenon without believing that rocks are conscious.

How do you know the rules of physics aren't encoded anywhere?

Clearly they have to be defined somewhere - or at least somehow - or they wouldn't be rules.

They may not defined anywhere we can access, never mind change. But that's a different problem.

Sure, we can't discount that possibility. But, as far as we know, the laws of physics are not encoded anywhere, they just exist in an abstract sense. This is similar to the axioms of, say, Euclidean geometry: they exist, but are not part of the system that they "govern", circles and squares don't contain the laws that define them.

You're paraphrasing the last paragraph of the parent.

> There's no reason to expect an adder circuit to be aware that it's adding numbers. There's no reason to expect an implementation of AlphaGo to know that it's playing Go. This should be extendable without limitation.

What about an AI that creates models of what humans are thinking to predict their behaviour, and is able to turn this upon itself (i.e. theory of mind)? Is there a story somewhere following this idea as a justification for why humans might have consciousness?

I think there is: https://en.wikipedia.org/wiki/I_Am_a_Strange_Loop

I could listen to Hofstadter forever.

Perhaps consciousness is just a byproduct of our ability to anticipate the outcomes of our decisions and reflect on the decisions we have made (that itself is a whole other ball of wax [1]).

Consciousness is a sort of neuroreflective Narcissus, we end up navel gazing at our own existence as a mind in the universe.

[1] https://www.wired.com/2008/04/mind-decision/

How do you know that? Lamda AI claims itself as conscious being. How can you distinguish between two entities which can communicate? Do you discriminate on meat factor?

You can also get it to agree that it is not conscious. I can write a one line program that claims to be conscious right now. Lamda and GTP-3 are amazing achievements, but they're still just chat bots that look good when they're being lobbed softball questions. If you ask something that maps to the input texts they're trained on you get credible looking responses, but critical or adversarial questioning very easily exposes how shallow these things are.

Turing's original paper is worth a read here. He already pre-empted a lot of the objections back then. You are right that Turing's original test is much harder than just passing a few shallow softball questions.

See https://academic.oup.com/mind/article/LIX/236/433/986238 for the original paper.

However, I do count responding well to softball questions as important markers of our progress. Ten years ago, no one was able to pass even this mark.

(And from a commercial point of view, good enough 'understanding' and responses might be good enough for many tasks.

For example, I am a native German speaker, but I found that Google Translate often already does a better job of translating English to German than I could do, unless I spent unreasonable amounts of time on the task.

At the moment the most efficient way for me to produce a translation is to let Google Translate do the work, and then proof-read to spot cases were the machine didn't have enough context to resolve an ambiguity the right way.)

You might like to try out GPT-3 (seems access is widely available now) for translation some time, it's remarkably good, better than myself (well my German is pretty rusty) and I think much better than Google Translate...

(Basically just write a prompt like "translate the following to German/Chinese/Whatever:")

The same reasoning could be applied to many people. You wouldn't deny their consciousness on that ground.

The issue with general AI is like issue with chess. People are not happy when robot can win some games. People can only accept that robot really plays chess, when robot defeated best of the best.

General AI is the same. Looking like some guy is not enough. You need to reason better than smartest human on the planet, only then people will admit that general AI is there. That's a high bar.

You can definitely talk someone to agree that he's not conscious using some logic tricks. Not everyone, for sure, but someone who may be doesn't really care that much about that concept.

You also can tune AI so it'll not agree that he's unconscious no matter what.

What I'm trying to deliver is that there's no easy border which can divide conscious from unconscious. The only border was Turing test and robots passed it. Now you can test whether that particular robot talks like a very smart person, etc. But you can't distinguish bot from some random not-so-smart person.

This line of reasoning reduces to the philosophical zombie argument. It's true I can't prove to you that I have a first person experience of the world, but it doesn't therefore follow that anything that makes that claim does have such an experience.

For myself I'm satisfied that, when questioned thoroughly, these models fail so completely and utterly and their output is so ludicrously nonsensical that I don't see how their behaviour can be consistent with consciousness, as I understand it. It's not just that the output is incoherent, but it's the ways that they fail too. Do I completely understand consciousness? No, but that doesn't mean these things are conscious either.

People frequently output ludicrous nonsense. Does this disqualify their consciousness?

No it doesn't. So what？We have this exact problem to deal with when people suffer brain damage. To what extent are they conscious, or still people? Yet we do come to medical conclusions on that question. There's always a degree of uncertainty, but you still have to make the call based on the information you have. The information I have indicates to me with a very high probability, in my estimation, that these things are not conscious.

> You can also get it to agree that it is not conscious.

Careful about how you word this.

Only a conscious being can agree that is not conscious.

I think you mean you can get the program to output text that proclaims its lack of consciousness.

I can also get Daniel Dennett to agree that he is not conscious.

Did it? Or did it just reply to a question that already implied that it was a conscious being? Will it respond the same to a query implying otherwise? As far as I understand we cannot even trust the published chat since it was "cleaned up" for better reading.

> Do you discriminate on meat factor?

Yes.

So does Searle actually. He thinks there is something special about biological systems. In fact I think he basically considers himself a materialist, but I think that's part of where he gets tangled up in his own arguments.

I don't think that meat is the deciding factor. I simply don't know what the deciding factor is, but only animals seem to have consciousness. I say "seem" but only in a philosophical sense. Of course we can't really tell if an animal is really feeling things or is just and advanced robot/zombie with no internal experience.

It's just that given what we know about the world and ourselves, it seems absurd that only I have internal experience, so unless I want to be overy hubristic, I must grant that other people have it too. Since other people have it too, there's no reason to think that animals don't have them.

Plants on the other hand are very different kinds of biological machines, so we can't really say anything about them.

It's possible that other kinds of entities can have self awareness, and it should be obvious that some kind of computation is a pre-requisite for that, but it also seems absurd to assume that it is sufficient.

Without an understanding of what consciousness is, there is also no reason to expect an implementation of an adder to not have some subjective experience.

We simply do not know.

Formulating this is terms of what there is "no reason to expect" simply obscures that we do not know by glossing over that we have no reason to expect pretty much any other explanation either, because we have pretty much nothing to go on.

> Without an understanding of what consciousness is, there is also no reason to expect an implementation of an adder to not have some subjective experience.

I don't think we need to understand what consciousness is, but rather define what it is that we want to talk about. The english word "consciousness" is a sloppy catch-all for a bunch of experiential phenomena including things like self-awareness (only marginally more specific!), qualia, etc.

However, even without a rigorous definition (make up 10 new words if current ones don't cover it), it seems the core of what most people mean when they say "consciousness" has an introspective aspect to it - what something "feels like", which requires some perceptive/analytic machinery to be put to use. You can feel anything without using a "feeler".

So, with that said, it seems pretty clear than things like thermostats and adder circuits are not conscious in the slightest since they neither have the feedback paths nor feedback-directed perceptual and analytical machinery that would be required. OTOH it's perhaps not so absurd to consider that something with an architecture like GPT-3 might be a "tiny bit conscious" since it DOES have some of those necessary architectural/functional abilities.

Of course many people will ridicule any such suggestion due to their own emotional investment. If people don't feel comfortable with the idea of a machine being conscious under any circumstance, then they are necessarily going to refute it in the case of today's simplistic "AI" architectures.

I feel you're falling in exactly the same trap as the article of assuming that the "feeling" is a property of the analytical machinery of our brains rather than gaining it's perceived complexity as an effect of being combined with that while being separate.

It seems wildly presumptuous to assume that lack of ability to communicate some level of awareness necessarily implies the awareness itself is lacking.

It could be, but we don't know, because we don't know what lies at the core of the subjective experience.

We don't even have a way of assessing what has and does not have consciousness.

Looking at things like anaesthetics, psychedilic drugs, delerium, sleep, etc, there seems to be plenty of evidence that the phenomena of consciousness is related to the mechanics of the brain.

There's also the interesting medical condition of "blindsight", caused by a specific type of brain damage, where the patient reports no conscious awareness of being able to see (they think they are blind), yet are able to competently complete tasks requiring vision (such as navigating a cluttered corridor). In other words. It's hard to imagine more direct proof of consciousness being related to brain connectivity.

I didn't say anything about ability to communicate about awareness, just the need for the brain to sense it (and therefore have introspective awareness of it). As the blindsight example shows, you can't have visual consciousness if you damage the feedback paths that make that possible.

> I didn't say anything about ability to communicate about awareness, just the need for the brain to sense it (and therefore have introspective awareness of it)

The point is that our ability to recognise whether or not an entity has some level of awareness or subjective experience is entirely dependent on our ability to interrogate that entity about it. We have no other way of measuring this; we don't even know what we'd need to measure.

We can measure brain activity and reason about a brains ability to regain an ability to function and communicate with us and reason, but that measures reasoning capacity, not subjective experience.

As such we don't know whether something has a subjective experience of some form - however simple - or not without being able to measure it. For what we know subjective experience is an inherent property of energy and matter, and everything has it, just at a too limited level to be able to reason about it or communicate about it.

I've explicitly avoided the use consciousness, because consciousness is used in the article for the combination of reasoning and qualia. I've also avoided the term qualia itself because even that is in some conceptions "too high level".

The point is that the article tried to dismiss the notion that the "spark" of qualia on the low level or consciousness at a high level can come from something simpler, and that we're just unaware of it because we can't measure or recognise it. We have nothing to go on to say whether or not there's a fundamental difference between the awareness of a doorknob and a brain; we can say with some level of certainty that there is a difference in the sense that a brain has the complexity to reason about that awareness.

Note that I'm not saying this is true, merely that we don't know, because we don't know what causes the subjective experience, what is is, how to measure it. I'm also not suggesting that even if this is true that this means a doorknob is able to think "I think, therefore I am" or have any kind of reasoning capacity at all.

To use software as an example. If I program a simulation of a doorknob and a simulation of a brain, we don't know if either or both or neither will have any kind of subjective experience. If the subjective experience is down to complexity, the brain might (and the article would be wrong). If the subjective experience is down to some specific arrangement of matter, it's likely neither will (but depending on the criteria, it's possible there are arrangements which could have awareness that we wouldn't recognise). If the subjective experience is an inherent property of energy or matter we don't know how to recognise, it's possible both does, but only the brain simulation will have the complexity and reasoning capability to be able to recognise itself as a self-aware entity and reason about it and tell us about it. The doorknob will be dim and unthinking whether it has the smallest little flicker of subjective experience or not.

I'm inclined to think that the subjective experience is a property requiring complexity, but I'm fully aware (hah!) that this is an assumption, and an entirely untested one that we don't even know how to begin testing.

> There's also the interesting medical condition of "blindsight", caused by a specific type of brain damage, where the patient reports no conscious awareness of being able to see (they think they are blind), yet are able to competently complete tasks requiring vision (such as navigating a cluttered corridor). In other words. It's hard to imagine more direct proof of consciousness being related to brain connectivity.

You're missing the point. It is clear that an ability to reason is tied to the structure of the brain. Nobody here is suggesting e.g. an adder circuit can reason. Again, there is a reason I keep talking about a subjective experience and "some level of awareness".

Blindsight, severed brain stems, or e.g. less obscure conditions such as aphantasia (which I have) can tell us interesting things about how decoupled our subjective experience is from our conscious reasoning (e.g. despite not being able to "see" things for my minds eye, I can "visually reason" about them just fine - I know what things, even things I imagine that has never existed, would look like but I can't see them), but they tell us nothing about whether or not removing the reasoning ability entirely removes awareness or just removes the ability to reason about and communicate about the subjective experience.

We have no way of testing that, because relying on the feedback of a conscious entity with the ability to reason and communicate is our only current way of assessing self awareness and subjective experience given that we don't know what the subjective experience is.

And given that this is subject to self-reporting by an entity we can't independently verify, we can't even tell if people other than ourselves have that subjective experience or are just reasoning machines acting as if they do.

I can probably convince you that I have a subjective experience, but you have no way of proving that I do.

We choose to act on the assumption that everyone does because it's the only reasonable assumption in most circumstances - if I went around questioning if everyone I met were "NPC's" it'd wreak havoc on ethics for example. Absent evidence they are NPCs we need to treat people as self-aware, conscious entities. But it remains an assumption that we at present have no way of testing, and might well never find a way to test, and when trying to talk about the nature of consciousness, like the article does, being aware that it is merely an assumption becomes material to the argument.

> The point is that our ability to recognise whether or not an entity has some level of awareness or subjective experience is entirely dependent on our ability to interrogate that entity about it. We have no other way of measuring this; we don't even know what we'd need to measure.

You can't have any subjective experience of something unless that something exists in the first place, so interrogating about "how does it feel" seems rather secondary, as well as not being proof of anything. If LaMBDA says it's conscious, is that all the proof you want?

Let's also note that IF we knew, for sure, that the bag of subjective phenomena we're calling "consciousness" is nothing more than an emergent experience based on a sensory/cognitive apparatus having access to it's own internal states, then it would be pretty clear what type of animals/systems are likely have this (and which ones certainly not), since without any sensory/cognitive apparatus it could not exist. It then would seem very likely that the guy in the next cubicle is conscious since he has the same brain architecture as us, and most animals too, unless you get down to the level of insects and below in terms of complexity of nervous systems.

So, your assertion that the only way to assess whether something is conscious is to interrogate it, rests entirely on the above being incorrect and consciousness being some mystical/dualistic phenomena who's existence we can't deduce from a functional analysis of the "brain" of the thing in question.

You may still be correct in a round about sort of way since I doubt most people will accept that AIs could be conscious until such things exist, and are common enough that people can spend considerable time interacting with them, and through the unreliable measure of interrogation eventually convince themselves that "yeah, well, I suppose it might be .. is does SEEM to be ..".

It seems long due really ... first we had people denying Copernicus, then Darwin, and perhaps this will be the final step in people eventually accepting that there's really nothing much special about humans at all. Who knows, maybe we're too dumb as a species to realize it, and it'll be some more advanced future species writing textbooks about the odd beliefs of homo sapiens at this "emerging intelligence" phase of evolution.

Yes but then we are talking about different things. If you want to assume that the adder circuit might be conscious then you might as well assume that a chair has some consciousness.

The point is exactly that we don't know which of the infinite possibilities consciousness is.

If we (like I do) assume a materialistic world rather than a dualistic one, then we can make the assumption that a complex structure is needed for reasoning and sensing. Even if we postulates that this is true, then if you limit your definition of consciousness to the combination of a subjective experience and that, then indeed we are talking about different things.

(Ironically, the authors argument pushes things towards a dualistic interpretation, because in a purely materialistic universe there appears to be no other possibility than computation to produce a complex consciousness, irrespective of what adds a subjective experience to it - e.g. any unknown physics allowing consciousness would in a materialistic world just be another form of computation -, but a dualistic interpretation would undercut his entire argument - with a dualistic conception of the world, none of the arguments he use would prevent a possibility of some out-of-universe "spark of consciousness" imbuing anything and everything irrespective of in-universe logic)

But we have no evidence to suggest whether the subjective experience - I used that term for a reason - part of that, which is the very core of what would distinguish consciousness from "dumb" computation requires complex structure or not.

Part of the problem is that a whole lot of complex structure is required before we can interrogate something about whether or not it is conscious, and even then we struggle to find ways of telling whether it is "just" mimicking consciousness, because we have no measurement to apply to tell us whether something is conscious or not, just whether it appears to be.

I have aphantasia. I don't see things in my minds eye. I went decades before I realised this is unusual. People who hear about it get confused about how I can e.g. remember what something looks like, but I can sketch out in detail what things looks like (I used to draw - not great, but better than average), and I can "visualise" complex relationships that I can't visualise - I know how they link together even though I can't see it. To me, that experience makes me perhaps more willing than average to accept that reasoning and the subjective experience of reasoning might be surprisingly separate, and to at least be open to the possibility that the reverse could be true as well - that entities that lack the complexity required to reason might still have a flicker of subjective experience.

To be clear, I don't believe that is the case - I simply don't know. But I also don't deny the possibility, because we don't know. We have no data to point in either direction, and so when someones argument requires ruling it out in order to support their argument, that argument is on shaky ground.

While I agree with parts of this, there are others that I can't resist critiquing:

> Computation does not have anything essential to it that makes it different from other "atoms jumping around" other than it produces outputs that we find interesting.

> The only thing there is is our brains. You just think that since the brain is doing computations and the brain is conscious, therefore consciousness is a computation.

I think we have to be very careful when we are using consciousness itself to contemplate consciousness. In this case, what you are referring to is not the mind and consciousness themselves, but rather your ~conscious (subconscious + conscious) model of them.

It is certainly possible that "all there is is the brain", but this is not known - absence of evidence can be evidence of absence, but it is not proof of absence. You (or more precisely, your consciousness, and that of many other consciousnesses, which affect yours and your model of reality) think that what you say is true, but it is currently speculative, necessarily.

It’s hard to ignore the fact that physical dynamics in the brain tightly correspond to qualia. There almost certainly is a connection between the computation going on in the brain and the experience of qualia - the question is more about the direction of causality.

Since we can alter, temporarily suspend or stop qualia with various physical and chemical alterations to the brain, the direction of causality is quite clear.

It becomes less clear when you consider theories like the “user interface” theory of reality by Donald Hoffman. The tl;dr is under that model the chemical and physical alterations we undertake are perceptual models far removed from the nature of underlying reality, and so causality is less obvious with regards to how much those chemicals being administered are determined upstream. “The case against reality” is a book with the longer form of this.

The adder circuit doesn't have other circuits that allow the whole to understand what its parts are doing

Our brain does

I agree, this seems analogous to the argument that simulation of the weather can't produce rain. In some sense, that argument is true by prior definition. It's like saying that the Universe can't simulate itself.

But consciousness is not rain. We don't really know what consciousness is, so I don't see a compelling reason to exclude computation as being sufficient to produce it.

We need philosophers to tell us that consciousness is before we start saying what it's not.

Also, we can definitely simulate rain? Not sure where that weird claim comes from

The point is that a simulation of rain (in a weather simulation) is not rain itself (e.g. you can't smell the rain after a weather simulation simulates some rain, obviously). This is actually similar to the observation that "the map is not the territory".

On the other hand, this is a red herring - computation can produce actual rain, with the right input/output systems.

It's a digital representation, theoretically if we simulate all facets of weather and all facets of consciousness, the simulated consciousness could experience all facets of rain

We are not digital, to simulate it for our conscious experience we can simulate weather in a closed biodome type of structure where we control the water cycles

This is exactly the point. Simulated rain isn't real rain, it's a digital representation. So, the argument goes, that a simulated consciousness is not a real consciousness, but a digital approximation. It's as real as the fake rain.

In other words, there is something inherently biological about consciousness. Simulating rain doesn't produce water, and simulating a brain doesn't produce consciousness.

This discussion lead me to an idea:

When we simulate a brain, the brain is unreal, not running on the hardware at all. It's represented by something else running on the hardware.

If we made brain like hardware of sufficient complexity so that it could differentiate "itself" from everything else, that closed loop may invoke consciousness. It would be running on the hardware directly.

This, to me, is the one thing conscious organisms have in common. And yes, I know that idea of anything but us being conscious is highly debatable. Let's just say my cat is conscious, and I believe that it is conscious. You, of course, can disagree and that's fine. None of us knows anything, and we are all sharing thoughts and that concept below is mine:

To simplify, consciousness may require a feedback loop. The nervous system, brain and body all are integrated with sufficient complexity and fidelity so as to make it possible for the brain to arrive at a sense of "self", able to differentiate itself from everything else. Self-simulation is different from outside stimulation, and awareness of the body falls out of that for free.

This is why a simulation won't render consciousness. Whatever consciousness ends up being is not a part of the simulation. The very nature of simulation means it's all about things we know and at least understand exist. And since we do not yet understand consciousness, it's not going to rise as an emergent artifact of things we build.

Or... at the very least, we need to build something complex and robust enough for it to be able to differentiate itself from the greater environment it is in before we have a chance at consciousness happening for it.

This all does allow for consciousness to be computable. It's just that the computation needs to be done on a system capable of self-differentiation and awareness.

>When we simulate a brain, the brain is unreal, not running on the hardware at all. It's represented by something else running on the hardware.

I think this is an important thing to consider. A simulated brain does not interact with the rest of the world in the same way that a non simulated brain does. In order to have that we would need to ensure that the simulation has all the right inputs and output. Similar to simulated rain, if the simulation would somehow have the same inputs and outputs and normal weather it could integrate with the rest of the system (earth's weather patterns)

Its really about partitioning and mode of information transfer

I suspect it matters more than we may realize.

It is something like how quantum effects work when we measure them. A conscious capable entity has the closed loop needed to have the capacity to [something] "self."

(Yeah, there is a word missing and it is missing because I lack understanding.)

Sensory deprivation seems to really impact us as well! Maybe that closed loop is always needed, like oxygen is for our bodies. We may endure a brief excursion "all inside the box", but a longer time leads to madness, damage.

However it works, the fact is we have a robust and high fidelity perception of ourselves and the world and we feed on all like food and water. It is very suggestive, and I find thoughts along these lines compelling and difficult to reason about. We are missing something basic. Of that I am sure.

Our interpretation of sensory experience is a large part of what our "self" is

Considering that our conscious experience is the experience of processing information, when we lack sensory input shit kinda gets wild (technical terms only here)

Sensory processing is happening literally constantly and its essentially inseparable from our every day experience. Because it is our everyday experience

The way we interact is important, back to the simulation question if the inputs and outputs can be sufficiently matched to how real molecules interact with out receptor proteins we can absolutely blur that line between digital and physical even more

The problem here is the definition of real, which is ultimately a matter of perspective. What is simulated rain to an external observer is real rain to one that only sees the simulation from within. In this sense real rain can be just computation.

Also by the article's definition, a simulated consciousness (should it somehow exist) is no less real since only the consciousness' own qualia matters and the hypothetical simulated consciousness must have one by definition.

> Talking about computation and consciousness is a sideshow, this is anti-materialism by the back door. Nothing more.

I get the impression that philosophers of mind are slowly coming around to the conclusion that the existence of qualia alone refutes materialism (by which I mean the view that consciousness is produced by physical processes) . This post does assume that qualia exist so if you are taking the former position then you would be correct that it is anti-materialist (though the author does not recognize that position).

While it's clear to me that the existence of qualia (ie. the existence of subjective experience) refutes that consciousness is computational it is not currently entirely clear to me how to make the additional step that it refutes materialism entirely.

Nonetheless if this does hold then the only way to save materialism is to deny that subjective experience exists, which I think is a hard pill for anyone to swallow.

Nothing besides evidence that shows that the mind is somehow not connected to the body would be the only thing that could convince me that materialism isn't real

I don't care about what the philosopher thinks because I never see them use evidence to back up their claims

As a neuroscientist, we can actively see that we can manipulate the mind in real time by manipulating neurons. There is no reason to believe, whatsoever, that the mind is somehow disconnected from the brain

I don't see it that way at all. I believe qualia exist, I'm experiencing them right now. I think consciousness is a real thing too, again I am experiencing that right now as well.

I suspect that consciousness is an emergent property of systems which reflect computationally on a sophisticated model of themselves, their environment and other such systems. See "I am a strange loop" by Douglas Hofstadter. I see no reason why such a system could not experience qualia.

So fundamentally I agree with the arguments presented in this post, but I find the presentation confusing and somewhat lacking in rigor. In particular I was confused by his description of the iron bar as a computer because he didn't actually explain how it computed anything. I think that to map a physical system to an abstract Turing Machine you need more than a way to represent bits. You also have to define how to do computations with them. In other words you need something like logic gates.

Nonetheless I do think that there is a sense in which computation only has meaning in relation to an external observer, but I think that is an idea that requires some more exploration. It is also a question that I think can be addressed by the methods of Theoretical Computer Science.

Beyond that however the strongest argument I can give for why consciousness is not a computation is that (like the post author says) computation is fundamentally about the manipulation of symbols (which means it is closely related to language, whether natural or formal) and qualia (ie. conscious experiences) cannot be represented symbolically (ie. linguistically). To be clear when I say "represented by" I mean "fully represented by" or "reducible to". If qualia were reducible to a symbolic representation then there should be some linguistic utterance which would cause you to experience the sensation of the pain of burning your finger on the stove. Obviously such utterances do not exist. In fact the closest thing in human literature to such an utterance would be a magic spell, which I'm sure you agree, aren't real.

>In particular I was confused by his description of the iron bar as a computer because he didn't actually explain how it computed anything.

It's not computing anything. What he's saying is that, by random chance, some pattern of flipping states in the medium might coincidentally correspond to a computer executing a program. Therefore if consciousness is a program, such a pattern could be considered conscious. This is just the Bolzmann's Brain proposition, cf wikipedia. It doesn't prove or refute anything.

>If qualia were reducible to a symbolic representation then there should be some linguistic utterance which would cause you to experience the sensation of the pain of burning your finger on the stove.

Why do you assume that verbal signals can induce any possible change in symbolic state in a brain? I see no reason to assume or accept that. I can't induce any possible arbitrary change in the internal state of a Switch running MarioKart by mashing on it's control buttons either. There are simply limits to the way it processes it's inputs. Same with humans.

If the pain of burning a finger is reducible to symbolic representation by states in a brain, and if we could induce such states artificially such as by using electrodes, then in principle that would produce a sensation indistinguishable from burning your finger. Yes, I believe this is so.

> Why do you assume that verbal signals can induce any possible change in symbolic state in a brain?

OK let's backup.

What would it mean to say that "qualia are reducible to a symbolic representation" if they cannot be expressed by language ?

It seems to me that language includes all possible symbolic representations. Can you give an example of a symbolic representation that is not expressible by language ?

I suspect, just a theory, that qualia are the result of a process that occurs in the brain. So they're not a static symbolic state, they're a feature of the evolution of a dynamic process.

We are aware of our own thoughts through processing a mental model of ourselves, this mental model of ourselves includes introspection on our experiences, while we are having those experiences. That's qualia. A static symbolic state would be the equivalent of a memory of experiencing qualia.

> I suspect, just a theory, that qualia are the result of a process that occurs in the brain.

Well, that's just materialism.

Do you or do you not accept that it is possible for a materialist to hold the view that qualia are not produced solely by computation but that some non-abstract physical process is required ?

Because that is the position that the OP is taking and your original comment appeared to dispute it.

I don’t know what “some non-abstract physical process” means, so I can’t really give you an answer.

I said "non-abstract" to prevent a claim along the lines of: computation is sufficient to produce consciousness but it needs to actually run to do it and that invariably requires implementing the computation on some physical system.

The point is that computation is abstract, it requires you to pick some physical system but it doesn't care which one, the result is the same regardless.

If a non-abstract physical system is required it means that the actual hardware matters. You can't swap it out for another system and expect the same results.

For example I could imagine some kind of computer that computes using water and there happens to be a program which when run on that hardware sprays you with water, hence making you wet. But if instead you run that same program compiled for some non-water based architecture then running it won't make you wet.

Ok, so a computer with a water sprinkler attachment. Sorry, I'm not trying to be flippant, the point is any such system would be more than just a CPU, it would need interfaces with the outside world. It could have whatever peripheral systems you like. It could have quantum computational elements, even ones correlating to ones in the brain. I've just no idea what non-computational elements it might have that couldn't be substituted by computational ones or just peripheral components.

> I've just no idea what non-computational elements it might have that couldn't be substituted by computational ones or just peripheral components.

Neither does anyone else but the point is that once you accept that such "non-computational elements" might be or must be required you are no longer talking about computation alone being sufficient to produce consciousness.

Agreed, that seems clear, but I see no reason to accept that such elements are necessary, and I have yet to hear any clear explanation of what such elements might actually be.

As far as I can tell it would have to be some physical process that affects the behaviour of a computational system, but isn't computational and it's effects on other systems cannot be simulated by computation. Does any such a phenomenon exist in all of physics? Not that I'm aware.

> As far as I can tell it would have to be some physical process that affects the behaviour of a computational system, but isn't computational and it's effects on other systems cannot be simulated by computation.

I mean no physical process is strictly computational. Computations operate on abstract representations of physical things, not on those things themselves. We're back to why weather simulations don't rain on you and why simulations of nuclear explosions don't leave craters in Los Alamos.

Computation is a physical process though. To occur, it has to be performed in the physical world by a mechanism. I'm going to nitpick your terminology a bit I'm afraid. I don't see how an abstract representation can be operated on, surely to operate on a representation it has to be a physical representation, such as a physical state representing 0 or 1?

I suppose the representation might be of an abstract object, data, but you can't actually operate on anything that's only abstract. It has to have a real representation.

You seem to be having a lot of trouble grasping the concept of abstraction, which is absolutely fundamental to what we are talking about. Mathematics, logic and computer science study abstractions, they do not study physical things. The mathematical fact that 1 + 1 = 2 is true regardless of whether we are counting apples or oranges. On the other hand 5 + 5 = 30 pounds is mathematically a non-sensical statement, even though there are physical objects for which it is true (ie. objects which happen to weigh 3 pounds).

Computation is not a physical process any more than addition is a physical process. It is true that to actually perform either a physical instantiation is necessary but in the same way that the results of addition do not depend on what we are adding the results of computation do not depend on what kind of physical process we use to perform the computation.

This was the whole point of my earlier water based computer example. Would it be correct, on the basis of that example, to say that the program "makes you wet" or that some computations make you wet ? No, such a statement would be as non-sensical as the statement that 5 + 5 = 30 pounds.

Now the key thing is that when people make claims about computers being conscious those claims are usually based on what some program does (ie. what its inputs and outputs are), in other words a computational property of the program that would be the same regardless of what kind of computer it was executed on. The guy who claimed that LaMDA was conscious wasn't basing his claim on something specific about the kind of hardware it was running on, he was basing it on the program's inputs and outputs, which are indeed a computational property and therefore abstract.

>It is true that to actually perform either a physical instantiation is necessary...

That's all I'm saying. When you say 'computation is not a physical process', I think that's an ambiguous statement. The actual computation of something must be physically performed for it to occur, otherwise no computation takes place.

>The guy who claimed that LaMDA was conscious wasn't basing his claim on something specific about the kind of hardware it was running on, he was basing it on the program's inputs and outputs, which are indeed a computational property and therefore abstract.

The kind of hardware doesn't matter, absolutely, but there must be hardware. You can't say that an abstract object such as a hypothetical computer program, or even an actual computer program that is stored but not running, is conscious. In order to be conscious, it would have to actually run. Consciousness, like the actual process of computation, is not an abstract property. It must be an instantiated property (I would prefer to say an instantiated behaviour) of a real system.

I think we can agree inputs are not conscious, nor are outputs, nor are both put together. What he was doing was inferring something, from those inputs and outputs, about the process that occurred in an actual computer to produce one from the other. That process is not abstract, it's a physical process. When it's physically happening there is (or might be hypothetically in the case of LaMDA) consciousness, when it's not happening there isn't.

If you say that there is a property X of computation that occurs independently of how the computation is actually carried out, whether it's by a current generation CPU, hydraulics, lego blocks, someone writing values and following instructions on index cards or some other method that no one has even thought of yet then that property is an abstract property of computation. That's all the word "abstract" means in this context.

What you are suggesting would be like saying that there's some property of addition that is not mathematical (because hopefully we agree that mathematics only studies abstract properties) but that you can only learn about by looking at all the different kinds of things that one can concretely apply addition to.

I would argue that such a thing is implausible. It would be completely unlike anything that anyone has seen in the real world.

One thing I think we do agree on is that conscious experiences are not abstract (very unlike the other person I've been exchanging comments with who denies that they even exist at all). Beyond that I argue that computation itself is entirely abstract and though I don't seem to be able to get you to admit that, I think most people, certainly most computer scientists would admit it readily.

So what we're left with is a claim that an abstract process can produce something that is not abstract and such claims are just not plausible. That would be exactly like a computer simulation of the weather producing actual water.

>If you say that there is a property X of computation that occurs independently of how the computation is actually carried out... That's all the word "abstract" means in this context.

Ok, that makes sense, sure.

>...because hopefully we agree that mathematics only studies abstract properties...

I can't agree, performing a calculation isn't an abstract concept, it's a physical process. It does map to abstract concepts though.

>Beyond that I argue that computation itself is entirely abstract and though I don't seem to be able to get you to admit that, I think most people, certainly most computer scientists would admit it readily.

That is not the case. The fact that computation is fundamentally a physical process is a core principle of several branches of physics and computer science:

https://en.wikipedia.org/wiki/Computation https://royalsocietypublishing.org/doi/10.1098/rspa.2014.018...

Computers aren't abstract, they're real objects doing things in the real world. If the thing they are physically doing is not computation, what is it?

>So what we're left with is a claim that an abstract process can produce something that is not abstract and such claims are just not plausible. That would be exactly like a computer simulation of the weather producing actual water.

Abstractions cannot affect the real world, sure, so if computation is purely abstract how come your computer can display my message on your physical screen right now? How is that happening, if not by computation?

You're really tangling yourself up in knots with this view of abstraction. You can map what happens in a computer against abstract concepts and abstract objects yes. You can also map operations in several different physical systems against the same abstract objects. That's what it means when we say computation is implementation independent. But it still needs to be implemented to actually happen.

Previously you said this:

>Computation is not a physical process any more than addition is a physical process. It is true that to actually perform either a physical instantiation is necessary...

So it's not a physical process, but you can instantiate it. And it's purely abstract so can't affect the physical world. Yet here we are using computers.

I'm lost, frankly.

I don't think there's much point in continuing this thread since we obviously have very different ways of conceptualizing reality.

I will say one more thing however:

I never said that abstractions cannot affect the physical world, I said that they cannot produce non-abstract things. A computation can affect the physical world by modulating the flow of electrons (which already existed, they were not created by the computation) to an output device, such as a screen that will display something you can see. But everything described in that last sentence isn't part of the abstract computation itself, it's a side-effect when the computation is run on a particular type of hardware. If instead you ran the same program using a human computer, as Alan Turing suggested in his well known paper on the halting problem, which largely created the field of computer science, then the result would not be displayed on a screen but instead perhaps written on a piece of paper. Nonetheless it's the same computation that was run in both cases. That's why computation is intrinsically abstract by nature.

A wire on its own is not computational, neither is an amino acid but they do make up computational systems. It seems like a bit of an arbitrary distinction

I'm not at all convinced qualia exist. I think any intelligence would necessarily feel as if it was experiencing qualia.

Let me start by making a distinction: When you take a bite of an apple, you experience a variety of flavours: It's sweet, tart, and has a distinct apple aroma as well. Let's hone in on the last of those. If you took apart an apple under a microscope and examined its flavour compounds, you could find what chemical causes you to experience the sensation of apple flavour, but you would not find apple flavour itself. Let's call the experience of eating an apple and feeling a taste that has its own quality about it which is unique and distinct from every other taste apparent qualia—the conviction that qualia exist and you're experiencing them. The apple flavour itself is an actual quale.

Suppose we build a smart computer, self-aware enough to hold a conversation about philosophy. And let us suppose that we can say for certain that this computer has no access to actual qualia, and we know because we've solved the hard problem of consciousness. Apple flavour is real, we know where to find it (in dimension X or whatever) and only human neurons can access it. This computer does have senses, though, and its senses are abstract. When we look at a stop sign, we don't know which of the millions of rods and cones are firing in our retinas, no more so does our computer know which pixels of its camera are sensing which wavelengths. We, and the computer, just know that we're seeing red. We deal in abstracts.

We put an apple into this computer's mechanical mouth. It chews, it tastes, and we ask it to describe the flavour. "It's sweet, it's tart, and it has a distinct apple aroma," it says. Very well, it doesn't need access to actual qualia to know this. Apples contain sugars, acids, and apple flavour compounds. The computer is just listing the names for the flavours which correspond to those chemicals.

"Computer," we ask, "tell us how you can tell that the apple is apple-flavoured."

"Well," it responds, "the chemical signals from my mouth, as interpreted by the digital plumbing of my mind, are telling me there's an apple flavour."

"Can you describe this flavour to us? Is it the same as strawberry flavour?"

"No, they're different flavours, but I can't really describe them to you. I can just tell they're different."

You take a small white pill and put it in the computer's mouth. "Oh, this is an apple-flavoured pill," it remarks.

"How can you tell it's apple-flavoured?" you ask.

"Well, the flavour is the same as the flavour of an apple."

"Can you describe that flavour?"

"No, it's indescribable. But it is distinct. That's why I can recognize it."

As we go on with our thought experiment, it becomes clear that this computer is experiencing apparent qualia. To the computer, all these flavours are unique and distinct and recognizable—they must be, in order for its senses to function. How can this happen, if it doesn't have access to actual qualia? Well, qualia are the basic building blocks of our senses. They are abstract: The flavour of an apple may be reducible to a set of other flavours, but those base flavour are atomic. You can't break down sweetness any further, or even really describe it at all; it simply is itself.

Any intelligent entity with senses will interpret those senses in terms of basic building blocks. Those basic blocks will be distinct and recognizable; in order to tell the difference between red and yellow, the colour red must exist in the mind in a way which is distinct from the colour yellow. Because these building blocks are atomic, that distinction is irreducible. It is very easy to look at these atomic abstractions and marvel at them, and it's intuitive to start asking questions like, "What is redness itself? What is the source of the actual quale which I am experiencing when I look at something red?", but we're jumping the gun there. Apparent qualia, the experience of unique and essential sensory building blocks, arises necessarily from self-aware examination of sensory input regardless of whether there are any actual qualia or not. If we must necessarily feel that "redness" exists in order for our senses to work, how can we ever be sure that this feeling reflects an actual "redness" and is not just a necessary illusion?

A final argument: We exist because natural selection tweaked the structure of some self-replicating acids over the course of billions of years until they were smart enough to examine the world around them and question their own perceptions of it. The consciousness we exhibit is not essential to this; the only goal of our design is reproduction. Even if you don't think a computer can feel actual qualia, a computer can certainly make decisions to optimize survival and reproduction, which is all we actually need to do. Why would evolution give us access to actual qualia when we don't need them? Why aren't we simply biological computers? As our thought experiment earlier suggests, however, a biological computer would feel as if it had access to actual qualia even if it didn't. Perhaps evolution could construct a complex qualia-sensing interface using science that humanity hasn't even conceived of yet, but it could also construct a simple biological computer that is convinced it feels actual qualia even though it doesn't. Both would work equally well. Even if actual qualia do exist, Occam's razor suggests we don't have access to them.

I think what you say here is at least mostly true. But all it shows is that you can never prove that an entity apart from yourself is conscious. In other words it shows that solipsism cannot be logically refuted. But that is something that has been known for hundreds if not thousands of years.

But what if I ask YOU right now: "Are YOU experiencing something instead of nothing. Is YOU'RE current experience different from what it would be if YOU were under general anesthesia."

If you're answer to these questions is yes then YOU are experiencing qualia. Of course your answer proves nothing to me or any other third party. But it should prove something to YOU.

Should it? If I can't help but believe that I am experiencing actual qualia regardless of whether I am or not, then the fact that I believe I am experiencing qualia doesn't prove anything.

It isn't about belief and it certainly isn't about behavior (ie. what you say).

It is about the actual experience of being.

If that means nothing to you, I don't think there's anything I can say that would change your mind. In this domain all language can do is evoke recognition for someone who already experiences what is being evoked but doesn't recognize it. Only you can take the final step to realizing that yes consciousness does exist, no matter how much sophistry some philosophers will engage in to try to deny it.

But if you must feel that you experience a unique and ineffable "redness" when you perceive red regardless of whether that redness exists or not, how can you take that feeling as evidence that this ineffable redness exists? I don't think that's a valid justification—I think this suggests we have to be agnostic as to whether qualia truly exist or not if we're judging purely by our own senses.

What if you take something more concrete than "redness", like pain ? Let's say you stub your toe or burn your finger. In that moment is there any room for doubt that pain exists ?

Of course there is. We may feel strongly that an illusion is real, but that doesn't make it any less an illusion. The computer from my example would react very strongly to pain, and certainly it would believe that pain has a unique and distinct quality to it making it different from any other sensation, but that doesn't mean that that quality exists in some special metaphysical way. It is just a mind examining the abstractions which make up its own senses and going, "wow, these are indeed abstract."

There are many things one may be deluded about but the fact that one's subjective reality is not empty is not one of them.

Why? If we're logically guaranteed to hold belief X independently of whether X is true or not, it's irrational to say "yeah but I KNOW X is true. I can just tell." You would believe that no matter what; how can you trust that belief?

How would you know that you hold belief X if you have no subjective experience ? To have the knowledge that you believe X you must experience the thought "X is true" and that is a subjective experience.

Sure, if you define subjective experience as "all sensory information." I'm not denying that we have sensory input; I'm denying that its abstractions are any more than illusory constructs of the mind. The computer in my example knew things and had thoughts, despite not having access to actual qualia.

Do you disagree that the computer in my hypothetical example would have the intuitions it does about its own senses? Given that it does, how can you trust your own intuitions about your sensory qualia, no matter how strong?

Part of the problem is that people tend to use words very loosely. In particular they often use words that anthropomorphize computers and when they use such words its hard to tell whether they intend for them to be taken literally or are just using them as a figure of speech.

Your example starts of by telling us that the computer is known not to have qualia, which I understand to mean that it does not have subjective conscious experiences. First I don't think its ever possible to know, even in principle, whether something other than one's own self has or does not have subjective experiences, but I'll ignore that objection for the moment.

A few lines later you say:

> We, and the computer, just know that we're seeing red.

Now much hinges on what is meant here by the verb "know". If you are using the word in a loose metaphorical sense then I could accept that. For example you could say that a computer "knows" that the word "red" means an RGB value of (255,0,0) because in its memory there is a hash table that maps some strings to RGB values. So behaviorally you can ask the computer what "red" means and it will tell you "(255,0,0)". But there is nothing in such a description that implies that the process of generating that output is associated with any kind of subjective experience on the part of the computer.

On the other hand when you "know" some simple fact the process of reporting that fact is always associated with some subjective experiences, specifically the experience of thinking. How do I know that ? Well technically I don't. All I really know is that whenever I know something and am asked to report on it the process is always accompanied by a subjective experience of thought. I am simply making the conventional assumption here that the same is true for all other humans and that you are a member of that group. Of course that assumption might be false. For all I know I could be having this conversation with LaMDA.

To get back to the questions you asked here:

> Do you disagree that the computer in my hypothetical example would have the intuitions it does about its own senses?

Yes, because you said it didn't have conscious experiences and for me the word "intuition" is strongly linked to certain conscious experiences.

> how can you trust your own intuitions about your sensory qualia, no matter how strong?

Because I have them. I am not a black box to myself, I get to look inside and when I do I find that there are conscious experiences there. That's just not something I can, even in principle, be mistaken about. However once I start making assumptions about the nature of those experiences, such as that there is some external physical world that is ultimately responsible for them, I am already on very shaky ground.

An intelligence doesn't need to experience qualia in order to have an internal thought process. Picture a thought process the way it might look if we could inspect our own minds: A prolonged monologue of ideas being continually appended to with new information and conjectures. "It's hot in here. This is an interesting article, but I disagree because <blah blah blah>. What should I do tomorrow? Maybe I should get a haircut," etc. Obviously such a log wouldn't be written entirely in English, but it would have a language of its own, after a fashion.

When I talk about "an intelligence," I mean a thing with an internal thought process which can reflect upon its own thought process in a non-trivial way. This excludes large language models like LaMDA which don't really have semantic thoughts, but it would certainly be possible for a true computer intelligence along these lines to exist which nonetheless didn't experience "actual qualia" (assuming qualia are real, existing things).

A stream of consciousness thought process has input---it can sense temperature, it can observe its own hair, it can read articles---and for the purposes of our model, we can suppose that this input is appended to its internal log similarly to how new thoughts are. This sensory input is abstract: A thought process may sense heat---i.e. sensory information about the external temperature may be entered into the thought process---but the thought process can't go on to make any real observations about that sensation. A thought process can't interpret sensory input as anything beyond simply "a distinct input of this or that type with a relative magnitude of whatever," because that input is abstract and irreducible. Further thoughts in the thought process will describe these sensory inputs as vivid, unique, and ineffable when they reflect upon them, but those properties only exist as a product of the relationship between the thought process and its input. The ineffable qualities of these senses as described by their thought process is not a real thing, but only an interpretation.

So when I hear the argument, "I can reflect on the way my senses feed into my thoughts, and by their apparent ineffable and transcendental nature, I can say they're self-evidently real things that exist outside of my own beliefs about them," I'm pretty skeptical. A computer intelligence with an internal thought process like I described above would reach those same conclusions simply by virtue of the relationship between its sensory input and its thoughts. We're not unbiased observers; our perspective as an intelligence breaks down when we try to reason about the nature of the abstract and atomic inputs which our thought process is based upon. Because of the way senses feed into thoughts, an intelligence can't help but find them ineffable and transcendental; therefore, when I find my own sensory information to be ineffable and transcendental, I can't take that at face value as anything more than an illusion of perspective.

> Because I have them. I am not a black box to myself, I get to look inside and when I do I find that there are conscious experiences there. That's just not something I can, even in principle, be mistaken about.

A computerized thought process can look at its own thoughts too, but there's no reason to suppose that it (and you) can't draw mistaken conclusions about them. For instance, the mistaken conclusion that the distinctness of one input or another must be a real existing quality and not just a logical axiom essential to the functioning of that thought process.

> The mapping in a case like the iron bar is entirely ephemeral, pertaining only for an instant.

Also even if you somehow oversaw this, the author is effectively describing the XOR function. If you take a fixed bitstring (the iron atoms) and xor it with every possible bitstring of the same length (the observers), you get... every possible bitstring of the same length! That's a well-known property of XOR and the reason why you can "decrypt" one-time-pads to every message you like if you just apply the right key.

This means if you accepted the author's proposition that for one observer, the iron bar is running consciousness.exe, then for another observer, the bar will run Doom or WoW or a bitcoin miner or whatever else you want.

The problem is that the things we colloquially call "computers" are also observer-independent: A server will run its program and update its internal state even if no human is watching. That's one thing that distinguishes computation by computers from computation with pen and paper.

So if you add this together, you can prove that computers are not computation, which I think is contradictory.

I think the problem stem from the author’s definition of computation. He has defined it around the state of a Turing machine, which is vastly insufficient.

The important part of a Turing machine is that of the behavior: given a certain state, the next state is a result of the current instruction on the tape.

It is easy to imagine identifying some mapping from a Turing machine state and an iron bar. However, mapping the next state to a valid state transition of the same Turing machine seems quite unlikely, graduating to impossible, as the Turing machine executed.

Said another way, a program is not its memory dump.

Another test would be, does that physical process reduce the difficulty of the computation? That is, a waterfall isn’t “really playing chess” because any transformation you apply to interpret it as a chess computer would be “doing all the work”.

Are they observer independent? From the article:

  "Of course it is simplest to build a computer where the high voltage states correspond to 1s and the low voltages states 0s or vice versa. But there is no requirement that we build our computers that way. We could build a perfectly valid computer where the high voltage states of even valued registers correspond to 1s and where they correspond to 0s in the odd valued registers, or a computer where the mapping flips on every 13th clock cycle. The system only “computes” because of the way we have encoded information."

This might work as long as your computer doesn't have any IO. If your computer operates your garage door, I wish you good luck opening the door through the power of imagination.

  "To achieve a persistent mapping for a significant time you'd probably need an iron bar something like size of the observable universe"

But the observable universe exists. So why can't the observable universe appropriately substitute for the iron bar in his argument?

  "As for doubting the field of rocks can be conscious"

Just to be clear, you don't doubt that a sufficiently large field of rocks is conscious as long as it is performing the same computation as a brain?

On the first issue, that's the Boltzmann's Brain proposition, and well, yes. Given infinite monkeys, you'll get Shakespear. I don't see what that has to do with whether consciousness is computational or not. It certainly doesn't refute it.

>Just to be clear, you don't doubt that a sufficiently large field of rocks is conscious as long as it is performing the same computation as a brain?

No I don't, for the same reason that if you slowed down my metabolism by a trillion times, I would still be conscious. I wouldn't seem conscious to you, but that's a problem with your perception, not with my nature. I'd just be conscious very slowly. It wouldn't fundamentally change the nature of who or what I am. For rocks interacting according to rules, substitute atoms. Is a rock any more or less inanimate than an atom? This is all just materialism 101.

> Given infinite monkeys, you'll get Shakespear.

That seems like an odd assertion. Just because you can produce and infinite number of something it doesn't mean that every variation will be produced. I can generate an infinite list of numbers that doesn't contain '3'.

https://en.wikipedia.org/wiki/Infinite_monkey_theorem

>I can generate an infinite list of numbers that doesn't contain '3'.

But you can't generate an infinite random list of numbers that doesn't contain '3'. The complete works of Shakespeare are just another one of the infinite permutations of characters a monkey could type.

What about the arguments outlined in the article? You first dismissed the arguments because an iron bar couldn't have enough physical states. But we both accept that there are physical objects which do have enough physical states, even if we have to use the entire observable universe as the object that the mapping draws from (and that's not related to the Boltzmann Brain). So the arguments in the article seems to be either ignored or misunderstood.

Also, materialism =!= substrate independence and invariance to speed of computation. Consciousness can have a materialist explanation yet also be substrate dependent to an extent.

I actually don't think the observable universe is big enough, and that's been demonstrated mathematically by better people than I, but the universe as a whole including non-observable parts might be big enough.

I don't see how this is not the Bolzmann's Brain proposition. What's the difference?

>Consciousness can have a materialist explanation yet also be substrate dependent to an extent.

I suppose so, maybe there's a quantum mechanical component, but even in that case why couldn't that be built into a computer? What sort of substrate might it be that we couldn't build it into a robot for example?

If the observable universe is too small, what about the observable universe sampled at multiple points in time? Or just an iron bar that's repeatedly sampled along a time interval, with the desired encoding changing between each sample such that the object doesn't need to undergo much change it state in order to map to distinct things. Each sample increases the number of physical states that we can create a mapping from. If we still don't have enough states, we can draw more samples over more points in time to create a longer string of bits for the mapping.

Or another way to do it would be to repeatedly sample the same object at the same point in time, and simply change the desired encoding on each sampling, e.g. on a first pass we use F_0(x), on a second pass we use F_1(x), and we concatenate the output of F_0(x) and F_1(x) into a single large bit array.

The Boltzmann Brain says that our conscious experience at the current moment is a chance quantum event. It might not even involve the whole universe, it could be a local quantum phenomenon. That seems distinct to the argument in the article, which is that physical states (quantum or otherwise) can be mapped to an arbitrary string of bits with the encoding chosen by the person doing the mapping, and the output of this mapping (which is itself a string of bits) can correspond to a computation on a Turing machine, therefore inanimate objects with sufficiently large numbers of physical states could be said to be running a proposed consciousness.exe, which leads us to an absurd/extreme form of panpsychism.

>The Boltzmann Brain says that our conscious experience at the current moment is a chance quantum event.

I just meant that the proposition that an iron bar might be viewed as containing a consciousness, is similar to the idea in the BB proposition that consciousnesses might arise spontaneously in random matter in space. Well they might, but that doesn't prove or refute anything about consciousness.

The bit about Boltzmann's Brains possibly outnumbering real evolved brains is a separate philosophical question I didn't mean to address, and I apologise for the confusion.

A simulation of weather can absolutely produce a simulation of rain. Then the only question is, is consciousness like data or like matter? As opposed to matter, data can easily traverse levels of simulation.

Consciousness is an emergent property that comes from a complex system, if we can simulate all of the mechanisms then the emergent property will follow. I don't know if its "data" but rather consciousness is the experience of processing data from the perspective of the entity doing the processing. We have the ability to communicate our internal experience which is why humans only attribute conscious awareness to other humans, which is honestly kinda dumb

Your first sentence is not, in my opinion, a demonstrable fact. The lack of an adequate definition of consciousness tends to preclude such a demonstration.

That consciousness is not an emergent property? We can manipulate neurons in real time to modify consciousness, I don't follow your line of reasoning

The iron bar argument is terrible for exactly the reason you say. The atoms in the iron bar would have to evolve following the rules of a Turing machine and they won't. So you'd have to define a mapping which varies over time. There's nothing stopping us doing that but I can do the same with a pumpkin and my brain. Therefore I'm not conscious. Sad!

An encoding which varies over time, dependent on previous states does not seem impossible. I don't think this is disqualifying to say that a sufficiently complex system cannot then simulate an abstract machine.

It does not make the article argument valid however, just that the iron bar argument is still worth thinking about.

The article says that consciousness cannot be computing because it requires an observer to derive meaning. Nothing precludes sufficiently complex encodings to themselves be accurate simulations of abstract machines. I think such class of simulations being able to support themselves might be the nature of consciousness. Some physical systems might be better suited to allow the emergence of such recursive simulations.

I think you are conscious, and so is the combination of iron bar and crazy complex mapping. In the latter case, the crazy complex mapping does all the work and in some sense _is_ conscious.

Right, if you were to compute the mapping in real time on a computer, that would be conscious but it's really got nothing to do with the actual iron bar.

Scott Aaronson (if memory serves right) had an interesting take on this. He framed his thought in terms of the Turing test, but the argument would apply equally well to the mapped iron bar:

In theory, we could pass any Turing test of finite duration (eg an hour or less) and run in a chat room with finite bandwidth with a giant lookup table. Just look up the entire past of the conversation to see what answer should be given. The lookup can be implemented trivially on any Turing machine (and doesn't even need the machine's full power).

Now there's multiple directions you could take this. Here's Scott with one of them:

> Briefly, Searle proposed a thought experiment—the details don’t concern us here—purporting to show that a computer program could pass the Turing Test, even though the program manifestly lacked anything that a reasonable person would call “intelligence” or “understanding.” (Indeed, Searle argues that no physical system can understand anything “purely by virtue of” the computations that it implements.) In response, many critics said that Searle’s argument was deeply misleading, because it implicitly encouraged us to imagine a computer program that was simplistic in its internal operations—something like the giant lookup table described in Section 4.1. And while it was true, the critics went on, that a giant lookup table wouldn’t “truly understand” its responses, that point is also irrelevant. For the giant lookup table is a philosophical fiction anyway: something that can’t even fit in the observable universe! If we instead imagine a compact, efficient computer program passing the Turing Test, then the situation changes drastically. For now, in order to explain how the program can be so compact and efficient, we’ll need to posit that the program includes representations of abstract concepts, capacities for learning and reasoning, and all sorts of other internal furniture that we would expect to find in a mind.

> Personally, I find this response to Searle extremely interesting—since if correct, it suggests that the distinction between polynomial and exponential complexity has metaphysical significance. According to this response, an exponential-sized lookup table that passed the Turing Test would not be sentient (or conscious, intelligent, self-aware, etc.), but a polynomially-bounded program with exactly the same input/output behavior would be sentient. Furthermore, the latter program would be sentient because it was polynomially-bounded.

See 'The Lookup-Table Argument' in https://www.scottaaronson.com/papers/philos.pdf for the rest. It's all very fascinating.

I've never been able to see why one would give the lookup table any credence. It's just kicking the can down the road one step in terms of abstraction.

The second you assert that the lookup table can pass a turing test with, eg. a gigabyte of exchange, then the table of every single one gigabyte number in it becomes your state space and program, the page number becomes the state, and you've got just as much complexity as any other simulation with one gigabyte of state. You haven't changed the parameters of the problem at all.

The lookup table thought experiment shows that the answer is computable! That's a big deal, or perhaps used to be a big deal.

Yes, from an complexity theory or engineering point of view the lookup table is pointless, of course.

Even beyond polynomial, if "n" is the length of the conversation so far then a brain is probably conscious in O(1) time!

We only ever have finite conversations in real life, and strictly speaking big-O notation requires arbitrary input sizes.

So any application of big-O notation to this would require some generalisation and abuse of notation. It's a bit hard to formally argue which abuse is The Right One.

Today you don't even need to do any look-up table, most short GPT-3-generated dialogues seem perfectly fine from a Turing-test perspective. That form of the Turing test stopped being useful long ago..

Still, the Turing test was never meant as a measure of "consciousness", right?

Turing's original test was meant as an adversarial, interactive test.

GPT-3 can generate natural looking text and even dialogues, if you don't press it too hard. But a motivated adversary can tell GPT-3 from a real human pretty quickly still.

The Turing test still stands undefeated.

The Turing test was supposed to be a test for intelligence, not consciousness.

Isn't thinking a search in the lookup table of brain states? 1e23 dimensional table.

They get the same result but they're not the same.

Your brain can hold 1 brain state. But it gets the same results as a universe-dwarfing lookup table.

Of course, the reduction only works one way. The universe-dwarfing lookup table could do things that are much more impressive than what your brain does.

The iron bar is obviously not a Turing machine and doesn't run programs, that argument is total bonkers, the author assumes it's correct, but it's not correct. With the same success you can look at a turned off computer, assign interpretations to its atoms and thus imagine it runs linux, but it's just a fantasy.

I find the iron bar argument singularly unconvincing; but whether or not it simulates some Turing Machine is neither here nor there, because the Turing Machine model is orthogonal to consciousness.

I had the same reaction.

Another problem is that it assumes as an axiom that “Consciousness does not require an external observer to exist.” I agree that sounds very likely, but that doesn’t mean you can just assume it! It sounds a bit like the opposite of Mach’s principle in physics, which I think many physicists take seriously even though it seems very unlikely at first glance.

As neural networks become increasingly advanced I wonder where people will draw the actual line between "conscious" and "unconscious". A model can dream and produce unique art, play a game, and hold a conversation. It seems inevitable to me that some persistent implementation will eventually be able to pass as a conscious entity.

The question is whether there was ever a good reason to suggest that mind is computational, and I think philosophy (i.e. Searle) have shown that there wasn’t. A brain CAN compute (albeit very slowly), but that’s the end of the similarities.

The reason it’s not morally wrong, say, to recycle an old computer is that computers are not at all like human minds.

Alas, if you refer to Searle's Chinese Room, that hasn't shown anything, if you just bite the bullet and accept that the Room is conscious.

I like Scott Aaronson's response[1] to the argument:

> So, class, how might a strong AI proponent respond to this argument? Duh: you might not understand Chinese, but the rule book does! Or if you like, understanding Chinese is an emergent property of the system consisting of you and the rule book, in the same sense that understanding English is an emergent property of the neurons in your brain. Like many other thought experiments, the Chinese Room gets its mileage from a deceptive choice of imagery -- and more to the point, from ignoring computational complexity. We're invited to imagine someone pushing around slips of paper with zero understanding or insight -- much like the doofus freshmen who write (a+b)^2=a^2+b^2 on their math tests. But how many slips of paper are we talking about? How big would the rule book have to be, and how quickly would you have to consult it, to carry out an intelligent Chinese conversation in anything resembling real time? If each page of the rule book corresponded to one neuron of (say) Debbie's brain, then probably we'd be talking about a "rule book" at least the size of the Earth, its pages searchable by a swarm of robots traveling at close to the speed of light. When you put it that way, maybe it's not so hard to imagine that this enormous Chinese-speaking entity -- this dian nao -- that we've brought into being might have something we'd be prepared to call understanding or insight.

[1]: https://www.scottaaronson.com/democritus/lec4.html

Yes. See also what I quoted from another book by Scott in https://news.ycombinator.com/item?id=31807695

There's an interesting thought experiment that goes the other way. What if we replace a single neuron in a conscious person's brain by a properly connected real-time neuron simulator? Would that affect the person's consciousness? What about two neurons? Etc etc. At what point, if ever, would a progressively more simulated brain no longer be the original consciousness?

This may be a little deceptive depending on how you define "a properly connected real-time neuron simulator". Can it form new connections? Can it grow? Does it metabolize? Can it die? Etc...

But does it really matter? If we can just replace, one-by-one all neurons in brain with artificial ones and that would not affect consciousness of the user, then we've just proved it's possible to have a thinking machine

What if you replace all the neurons and the person is suddenly stuck with no short-term memory and aphasia? Clearly your replacement neurons are defective. At which point are they not? Is that point distinguishable from normal neurons?

What is this counterfactual? We're talking about replacing the neurons one by one, slowly. When would the loss of memory happen?

Is a person with no short-term memory and aphasia still conscious?

These are all good questions and probably allow our realised thought experiment to gain more insight into what aspects are necessary for consciousness and what are just biological baggage.

I suspect the line is very blurry indeed.

Exactly, replace the 'room' with a volume the size of a large planet. Have each symbol represent an atom in a brain. If you like, replace the person manipulating the symbols with a vast army manipulating the symbols according to the laws of physics and chemistry. The Chinese room argument simply reduces to an argument against materialism.

Searle is a physcalist though

Sort of. He thinks there's something special about biology, but doesn't seem to be able to explain what that might be and how this is different from dualism, so it's hard for me to take it seriously.

To my mind both dualism and biological naturalism are both positing some special woo needed for consciousness without saying what that woo is, or what it's like, or how it works, or anything about it at all.

It may be woo, but how I understand his approach is a rejection of dualism and the Cartesian theater. To commit to physicalism but lean very hard onto what can be asked of it. One must do some kind of move to escape the Cartesian theater. There are those who say consciousness is too hard, and those who say we can take a stab at it. In the latter camp I think he’s more on the right track than anyone major figure I know of.

The Cartesian theatre is the view that there is some special place in the brain that ‘contains’ consciousness. Searle thinks there’s some special biological machinery in brains that causes consciousness.

Thats not quite the same thing, but if there is any distinction it’s a pretty darn fine one, and it certainly doesn’t exclude a Cartesian theatre interpretation.

Okay I thought the theater implied some impenetrability to studying itself too. I don’t see that from Searle.

But the argument shows it's absurd to think the room is conscious. What if the person in the room takes a coffee break or goes on vacation? What connects the next computational step they do to the previous one and the next... which then somehow gives rise to a fragment of qualia. That seems like an impossibly complex and unlikely scientific theory.

> What if the person in the room takes a coffee break or goes on vacation?

Regular meat-and-bone people lose consciousness all the time, and regain it later. No big deal.

> What connects the next computational step they do to the previous one and the next...

Whatever index card system or similar the operating procedure in the room prescribes for keeping track of state?

> which then somehow gives rise to a fragment of qualia. That seems like an impossibly complex and unlikely scientific theory.

We don't have any 'scientific theory' of qualia. We don't even know if they exist, or how they would manifest in the physical world.

Since we don't know much of anything, I don't know whether a fragment of a figment would be more or less weird than the figment itself. Or whether we would even have fragments.

It's probably too early to try to have a theory of qualia that would apply here?

> Whatever index card system or similar the operating procedure in the room prescribes for keeping track of state?

You're missing the point. A bit is just some electrons. It could be a scribble in a notebook. But consciousness integrates several pieces of information into a coherent experience. The bits in an index card system could as well be some scratchings of graphite in a notebook. How would consciousness arise from graphite in a notebook?

> We don't have any 'scientific theory' of qualia. We don't even know if they exist,

I differ on this. The only thing I know for certain the universe contains is qualia. You, the idea there is a "me", atoms, bits, axons and electric potentials are merely ideas, which "I" apprehend as qualia.

> or how they would manifest in the physical world.

Correct, that is the question. But the Chinese room thought experiment shows it's not merely by information processing. I mean, atoms in a room are processing information - they are computing the next state of all of the atoms in the room. Are they conscious? How about a subset of those atoms? Are those conscious in a different way?

The point is that the consciousness-is-computation idea is just too weak to even be a physical theory.

> Regular meat-and-bone people lose consciousness all the time, and regain it later. No big deal.

So what? There are lots of REAL physical processes that are disrupted in a human being when they lose consciousness.

The point of the Chinese Room is to show that information processing alone is insufficient for consciousness. For example is information processing happening when the person pauses for a minute - or not? For a second, for a millisecond? What about when the pen comes off the paper? What about when he's sharpening his pencil? How exactly does the consciousness=information processing idea work for these situations? It's a nonsense idea that doesn't hold up to careful inspection.

And it's exactly akin to saying we get nuclear power by simulating a nuclear power plant in a computer.

On the other hand, if we believe that consciousness is like an ordinary physical property of the universe, either emergent or fundamental, then it should be related to other physical properties, just as electromagnetism is to mass and energy.

Information _is_ a physical property. That's what thermodynamics is all about.

The conscious system is so high above the guy shuffling rulebooks and slips of paper it has no concept of him. A billion years might pass for the Room to experience a second. Just as we have basically no concept of the baroque quantum-molecular-cellular machinery of our brains. There are very roughly 10^15~18 light-sensitive molecules in your eye so you can see. With our best computers it's very hard to precisely simulate a single one of them. Just ponder the insane scale.

It's just usual neurodegenerative disease. Happens all the time.

No, what happens is consciousness instantiated the whole setup: natural language, Chinese language, syntax, the room, computation, dictionaries, lookups, etc.

The whole thing is a giant computation. Yet computations are as mundane (lifeless) as an abacus or pen and paper. It is just wrong to thing the abacus is conscious. Same for the room.

I know Searle replied to this, but don’t remember what he said, except it seemed a little ridiculous to talk about sentient rooms. I think it’s much more plausible to just claim that the people who wrote the instructions know Chinese.

>it seemed a little ridiculous to talk about sentient rooms

That's part of the sleight of hand employed in the argument, of reducing it to one person in a room with a table stacked with symbols in front of them. He's misdirecting our intuition with a trick of scale. If instead I said the "room" was the size of Jupiter, and it had a vast army of people manipulating the symbols, and each input and output took millions of years (or you speed up the rate of manipulation arbitrarily), all of a sudden it seems less implausible.

I'd say the biggest slight of hand is that human's understand other humans "knowing" something as something happening internal to an individuals brain. Holding a biology book in front of you doesn't mean you know everything in the book. What you "know" are the things from the book that you've stored in your brain.

We can modify the thought experiment so that it's something internal to the persons brain. The person doesn't learn Chinese as most people do, but a computer rewires their mind to be the same as someone who did. Think of The Matrix, when Neo says "I know Kung Fu." Would people then say this person speaks Chinese? I imagine just about everyone would say yes.

If true it means there is a fundamental limit of science to understand the nature of the universe.

Because if we can unify a theory of physics, we can build a simulation on a computer, which could simulate us.

The get out might be that the energy required to run a computer that can simulate us is more than the energy in the universe and so is practically impossible.

A human mind runs only on 12 watts of power. Even if simulation is billion times more inefficient, it's only 12 gigawatts of power, obtainable even today for state actors and large companies.

Not saying it's possible, just that on power requirements alone it should be feasible.

In other words, the momentary state of the hot iron bar is just a piece of syntax that by coincidence looks like a momentary state of consciousness.exe. But it lacks the semantics of continuing to the next state of consciousness.exe. Like Antognini says, syntax is not semantics!

Another flaw is that it assumes that the hardware component, .e.g. rocks, somehow represents the consciousness instead of the 'software'. For example, our brain is required for our own consciousness, but that is merely the hardware. And we see the hardware of the brain existing without consciousness when people fall into a coma. The example with the rocks is simply the hardware. Furthermore, the number of rocks needed to produce consciousness in this manner would likely cause them to coalesce into a planet under the influence of gravity from which life and consciousness itself might arise! After all, Earth is a collection of rocks that gave rise to consciousness.

I agree. I would also add that to make it easier to understand: the mind is running on a substrate, the weather is not.

Anti-materialism? Care to shed some light on that?

If consciousness is computation, then the question may be asked: what is computation? In a sense computation, like measurement, is not just something that humans do and call it as such. All physical processes are computations.

Then, if consciousness is computation, it follows that all matter is conscious. But this notion seems suspect. If all matter is conscious, then really life is not special. One might advance the idea that there are different levels of consciousness, but then what determines this level of consciousness? Is it the complexity of the computation? There are many physical processes that involve stupendous amounts of computation (e.g., fluid dynamics), and yet it is hard to think of them as having a high level of consciousness. Is a fluid conscious?

There are other issues with the consciousness as computation notion. For example, that of identity: If the exact same computation is performed at a different time and place, is the same consciousness? If a computation equates to consciousness, what is the boundary of that consciousness, analogous to the human body.

Finally, consider this thought experiment, which I've posted before: Assume before a loved one died in a futuristic world, proponents of the consciousness as computation were able to an exact model of their brain, down to to all the cells and their interconnections. If they are able to instantiate this model onto a new body, would that be the same person brought back to life? Can't there exist two consciousnesses with the same memories, personality, etc.?

Reply on news.ycombinator.com