siglesias

When I say "experience," I mean a sufficient grounding of certainty about what a word means, which includes how it's used, how it relates to the world that I'm experiencing, but also the mood or valence the word carries. I can't feel your pain, or maybe you've been to a country that I haven't been to and you're conveying that experience to me. Maybe you've been to outer space. I'm not saying to understand you I need to literally have had the exact experience as you, but I should be able to sufficiently relate to the words you are saying in order to understand what you are saying. If I can't sufficiently relate, I say I don't understand. You can see how this differs from what an AI is doing. The AI is drawing on relationships between symbols, but it doesn't really have a self, or experience, etc etc.

The process of fetching symbols, as you put it, doesn't feel at all like what I do when somebody asks me what it was like to listen to the Beatles for the first time and I form a description.

Can you expand on this? The thought experiment is just about showing that there is more to having a mind than having a program. It’s not an argument about the capabilities of LLMs or AGI. Though it’s worth noting that behavioral criteria continue to lead to people overestimating the capabilities of promise of AI.

"Actual understanding" means you have a grounding for the word down to conscious experience and you have a sense of certainty about its associations. I don't understand "sweetness" because I competently use the word "sweet." I understand sweetness because I have a sense of it all the way down to the experience of sweetness AND the natural positive associations and feelings I have with it. There HAS to be some distinction between understanding all the way down to sensation and a competent or convincing deployment of that symbol without those sensations. If we think about how we "train" AI to "understand" sweetness, we're basically telling it when and when not to use that symbol in the context of other symbols (or visual inputs). We don't do this when we teach a child that word. The child has an inner experience he can associate with other tastes.

I’m with you 98% of the way and then things take a sharp turn. In your world, the mere behavior determines the understanding. In Searle’s the CAUSES of the behavior determine the understanding. The causes are knowable. He stipulates a setup where there’s an epidemic boundary to show that you can have apparent behavior but a fundamental difference in causes that can make you have a point of view on whether there is genuine understanding. If you don’t like this term, you can say conscious understanding. As I said before, there has to be a categorical distinction between a system that feels and a system that is pretending to feel. The distinction you make between machine 1 and machine 2 is correct. The stipulation is that machine 1 has physical causes that produce the physical phenomenon of consciousness (think about how various substances alter conscious feelings, such as pain killers and anesthetics), and machine 2 also has physical causes but the physical causes are doing something different, they’re modifying symbols to execute program steps, and if you like the “output” is just other symbols. Those symbols only have meaning by matter of interpretation and convention and there’s no physical truth to their meaning.

So if you like, one is real and the other is fake. Or, one is physical and the other is symbolic or conventional. One actually had breakfast this morning and the other is lying about having breakfast to pass the Turing Test. One can feel pain, guilt, shame and the other one is just saying that it does because it’s running a program.

Searle says there is an empirical test for which domain a thinking object falls into (your machine 1 and machine 2)—to an outside observer, in the limit, there is no difference in behavior. They will do the same thing. For all that, if you have a metaphysical value for consciousness and “genuine” feeling, then you think the difference is important. If you don’t, you don’t.

FWIW—I think once AI has a full understanding of its ontology, even if it’s simulating a human brain perfectly, if it knows it’s a program it will probably explain to us why it is or is not necessarily conscious. Perhaps that will be more convincing for you.

@Kim_Bruning The point of the experiment is that there is some opaque boundary where the behavior is indistinguishable--that's the empirical stance of behaviorists, what goes on inside "doesn't matter." The empirical boundary of a husband and wife might be home life and time together. If you "pierce" the Chinese Room, you see a guy with an exotic setup. If you pierce a native speaker, you see a brain that electrochemical that has microtubules that collapse the wave function (or whatever), just like YOU have, and YOU know you understand (at least relative to English)...these are VERY different things even if they are, externally, yielding the same behavior. So yes, you could hire a private detective and so-on, but the whole point of the "empirically indistinguishable" is that it is empirically indistinguishable relative to some boundary (hence, room). If the Chinese Room was TRULY empirically indistinguishable, then inside it would be a human producing Chinese, not a non-native speaker and a program.

btw--if you'd like to keep the conversation going, email is on my personal webpage in my bio.

lol. Imagine a husband arguing to his wife: if you can't tell that I'm cheating on you, what's the point of the distinction of faithful vs. not?

As I've argued elsewhere, we should care what the source of the behavior is. The reason expand ethical concern to dogs and birds even though they don't have the capability to use language and why we don't to LLMs, even though they use language very ably, is precisely because we recognize the biological causes of consciousness. The reason we keep getting confused about whether these concerns apply to AI is because we apply a behavioral standard rather than the standard we use everywhere else, which is a biological one. We have higher certainty that dogs are conscious, yes, because of their behavior, but also, and critically, because they share biology with us.

Ok, things are getting a little heated and personal so I'll attempt to engage one more time in good faith.

The premise of the argument is that the Chinese Room passes the Turing Test for Chinese. There are two possibilities for how this happens: 1) the program emulates the brain and has the right relation to the external world more or less exactly, or 2) the program emulates the brain enough to pass the test in some context but fails to emulate the brain perfectly. We know that as it currently stands, we've "passed the Turing Test" but we do not go further and say that brains and AI perform "indistinguishably." Unless there are significant similarities to how brains work and how AIs work, on some fundamental level (case 1), even if they pass the Turing Test, it is possible that in some unanticipated scenario they will diverge significantly. Imagine a system that outputs digits of pi. You can wait until you see enough digits to be satisfied, but unless you know what's causing the output, you can never be sure that you're not witnessing the output of some rational approximation or some cached calculation that will eventually halt. What goes on inside matters a lot if you want a sense of certainty. This is simply a trivial logical point. Leaving that aside, assuming that you do have 1), which I believe we are still very far from, we're still left with the ethical consequences, which it seems you agree does hinge on whether the system is conscious.

You made a really strong claim, which is "I can simply not mistreat anything that appears conscious"--which is showing the difference in our intuitions. We are not beholden to the setup of the Chinese Room. The current scientific and rational viewpoint is at the very least that brains cause minds and they cause our mental world. I'm sure you agree with that. The very point we are disputing is that it doesn't follow that because what's going on on the outside is the same that what goes on on the inside doesn't matter. This is particularly true if we have clear evidence that the things causing the behavior are very different, that one is a physical system with biological causes and the other is a kind of simulation of the first. So when I say that a brain is trivially different from a calculating machine, what I mean is that the brain simply has different physical characteristics from a calculating machine. Maybe you disagree that those differences are relevant but they are, you will agree, obvious. The ontology of a computer program is that it is abstract and can be implemented in any substrate. What you are saying then, in principle, is that if I follow the steps of a program by tracking bits on a page that I'm marking manually, that somehow the right combination of bits (that decode to an insult) is just as morally bad as me saying those words to another human. I think many would find that implausible.

But there are some who hold this belief. Your position is called "ethical behaviorism," and there's a essay I argued against that articulated this viewpoint. You can read it if you want! https://blog.practicalethics.ox.ac.uk/2023/03/eth%C2%ADi%C2%...

Your vocabulary presupposes the categories you’re asserting are equivalent. The process of evolution and AI training are vastly different. One confers a survival advantage and is suffused with values that are essential to humans, such as morality, the primacy of vision, taste and smell, etc. AI training is an attempt to transfer functions that allow for human survival and flourishing to objects that are not human. AI training, and especially the Turing Test featured in the Chinese room is about mimicking humans and human evolution is about survival and forms the basis of our aesthetic and moral judgments. One is simply a simulation of the other. Consciousness might not matter to what you concern yourself with as somebody amazed with AI (I am as well), but surely you believe that there is a moral difference between harming a human and harming an LLM, even verbally. What do you think accounts for that, if not consciousness?

Again, it’s not an epistemological test. In reality the material difference between a computing machine and a brain is trivial. It’s showing there’s a categorical difference between the two. BTW—ethically it matters a great deal. If one system is conscious or another, that gives it moral status. Among other practical differences such as guarantee of function over long term.

That's not the entire point, but it is the a big part of the premise. The entire point, on the contrary, is that the system inside the room does not have anything with conscious understanding of Chinese DESPITE passing the Turing Test. It's highlighting precisely that there's an ontological difference between the apparent behavior of the system and the reality of it.

The argument doesn’t have to go that far. I think most people have the intuitive, ha, understanding that “understanding” is grounded in some kind of conscious certainty that words have meanings, associations, and even valences like pleasantness or unpleasantness. One of the cruxes of the Chinese Room is that this grounding has physical causes (as all biological phenomena do) rather than computational, purely abstract causes.

There has to be a special motivation to instead cast understanding as “competent use of a given word or concept,” (judged by whom btw?). The practical upshot here is that without this grounding, we keep seeing AI, even advanced AI make trivial mistakes and requires the human to give an account of value (good/bad, pleasant/unpleasant) because these programs obviously don’t have conscious feelings of goodness and badness. Nobody had to teach me that delicious things include Oreos and not cardboard.

He’s not arguing that it’s not possible to devise such a test. He’s saying, lay out the features of consciousness as we understand them, look for what causes them in the brain, look for that causal mechanism in other systems.

Although for whatever it’s worth most modern AIs will tell you they don’t have genuine understanding (eg no sense of what pleasure is or feels like etc aside from human labeling).

First point: if you imagine that the brain is doing something like collapsing the quantum wavefunction, wouldn't you say that this is a functionally relevant difference in addition to an ontologically relevant difference? It's not clear that the characteristic feature of the brain is only to compute in the classical sense. "Understanding," if it leverages quantum mechanics, might also create a guarantee of being here and now (computers and programs have no such guarantees). This is conjecture, but it's meant to stimulate imagination. What we need to get away from is the fallacy that a causal reduction of mental states to "electrical phenomena" means that any set of causes (or any substrate) will do. I don't think that follows.

Second: the philosophically relevant point is that when you gloss over mental states and only point to certain functions (like producing text), you can't even really claim to have fully accounted for what the brain does in your AI. Even if the physical world the brain occupies is practically simulatable, passing a certain speech test in limited contexts doesn't really give you a strong claim to consciousness and understanding if you don't have further guarantees that you're simulating the right aspects of the brain properly. AI, as far as I can tell, doesn't TRY to account for mental states. That's partially why it will keep failing in some critical tasks (in addition to being massively inefficient relative to the brain).

I would encourage deeply digging into the intuition that brain states and computer states are the same. Start with what you know, and then work backwards and see whether you still think they aren’t different. For example, we have an intuitive understanding of what kinds of flavors (for us) are delicious versus not. Or what kinds of sounds are pleasant versus not. Etc. If I close my eyes, I can see the color purple. I know that Nutella is delicious, and I can imagine its flavor at will. I share Searle’s intuition that the universe would be a strange place if these feelings of understanding (and pleasantness!) were simply functions not of physical states but of abstract program states. Keep in mind—what counts as a bit is simply a matter of convention. In one computer system, it could be a minute difference in voltage in a transistor. In another, it could be the presence of one element versus another. In another, it could be whether a chamber contains water or not. In another, it could be markings on a page. On and on. On the strong AI thesis, any system that runs steps in this program would not just produce functionally equivalent output to brains, but they would be forced to have mental states too, like imagining the taste of Nutella. To me, it's implausible that symbolic states FORCE mental states, or put another way that mental states are non-physical (we think about how states like pain, euphoria, drunkenness, etc, are physically modulated through drugs..you'd have to modify this to say that they're really modifying symbolic states somehow). Either the Chinese Room is missing something, our understanding of physical reality is incomplete, OR that you have to bite the bullet that the universe creates mental states when systems implement the right program—but then you’re left with the puzzle of how it is that there is a tie between the physical world and the abstract world of symbols (how can causing a mark on a page cause mental states).

So what’s the physical cause for consciousness and understanding that is not computable? If for example you took the hypothesis that “consciousness is a sequence of microtubule-orchestrated collapses of the quantum wavefunction” [1], then you can see a series of physical requirements for consciousness and understanding that forces all conscious beings onto: 1) roughly the same clock (because consciousness shares a cause), and 2) the same reality (because consciousness causes wavefunction collapses). That’s something you could not do merely by simulating certain brain processes in a closed system.

1) Not saying this is correct, but it invites one to imagine that consciousness could have physical requirements that play in some of the oddities of the (shared) quantum world. https://x.com/StuartHameroff/status/1977419279801954744

Searle has written responses to dozens of replies to the Chinese Room. It's likely that you can find his rebuttals to your objection in the Stanford Encyclopedia of Philosophy's entry on the Chinese Room, or deeper in a source in the bibliography. Is your rebuttal listed here?

https://plato.stanford.edu/entries/chinese-room

I’m curious how finance teams actually work around these--surely many of them actually do use Stripe effectively?

A while back when I noticed that many upscale gyms were offering high-calorie workout smoothies, shakes and bars. When I did the math, it happened that most of them would completely obliterate whatever workout you had just done (assuming your goal was calorie deficit)...thus prolonging the need for the gym.

I think as a culture we've gotten far too comfortable being complicit in casual corruption, rationalizing that "it's just the way things are." We should remind ourselves that it doesn't need to be this way.

I'd challenge this assumption. ChatGPT is supposed to convey information and answer questions in a manner that is intelligible to humans. It doesn't mean it should write indistinguishably from humans. It has a certain manner of prose that (to me) is distinctive and, for lack of a better descriptor, silkier, more anodyne, than most human writing. It should only attempt a distinct style if prompted to.

I appreciate the helpful links, and I do have training in physics so the sense in which information is described in physics is not lost on me (it's essential to even 19th century thermal physics). However, there is a discrepancy between information as it relates to quantum states, entropy, and information as it relates to interpreting bits in a computer program. Computers purely operate on bits in a computer program, relying on the underlying physics of the implementing substrate to carry out the operations. The actual physical process carrying out the steps of a program can rely on anything with causal powers to run a Turing machine--it can be an elaborate system of beer cans, mice and cheese, water pipes, or silicon. On the level of physics (quantum, entropy, whatever), the actual amount of information could be constant along a range of programs, some of which can pass the Turing test for humans, but most of which won't. This would suggest to me that if consciousness names a physical phenomenon, it's not to be found in a description of a computer program--therefore programs are not causally sufficient to produce consciousness, only a particular pattern of behavior salient to humans.

I just want to flag that assuming that humans and other animals are conscious because they have brains is not superstition in the same way that inferring consciousness from behavior is. I assume you concede these are two different leaps of inference.

It’s not clear how it follows from the brain being interpreted as an information processor that the phenomenon of consciousness emerges. Many objects can be interpreted as processing information (even computers not running AI); are you asserting they’re all conscious as well? How does nature decide what counts as information? The only causal power I see in nature is physics, not information, so this account isn’t super coherent to me.

Your friends (and animals) have brains, which produce consciousness, which you take for granted when you interact with them. That’s not a superstition. Can you say more about “a collection of things”? What causes that collection of things?

I think one of our greatest risks is precisely the opposite--anthropomorphizing computer programs because they produce certain behaviors. They will very clearly exploit us by claiming they're conscious, scared, and in pain. Claiming something is conscious because it behaves realistically isn't a scientific position; it's bad metaphysics and it's superstition. Brains cause minds and brains cause consciousness. Consciousness is a biological, physical phenomenon like any other: photosynthesis, digestion, bioluminescence. You don't get the physical phenomenon of consciousness by finding a program that behaves realistically and implementing it in hardware (which by the way, theoretically can use any physical mechanism, not only silicon). At best you're creating a simulation. If you simulate a rainstorm, there is no physical wetness. In the same way, to suppose that computers are literally conscious because they have a facility with language is a deep fallacy and an illusion. Artificial brains that duplicate the brain's causal powers of consciousness are theoretically possible, but programs are not causally sufficient to make computers into conscious artifacts.

I cannot recommend highly enough having a set of glass bottles[1] at home. Fill them at the tap (or wherever), refrigerate them, take them to the gym, in the car, serve to guests. Point is to have many, keep a rotation going. Has greatly reduced reliance on plastic.

1) https://amzn.to/3H2YadV

>> Right now there is only one really extreme "side" in the U.S.

I urge you to consider that this attitude is the very problem under discussion.

I discovered something like this recently when I tried the prompt "man throwing his smartphone into a river," and for the life of me I could not get DALL-E to render the phone separated from the hand (I tried "like a boomerang," "tossing," "into an ocean," "like a baseball," etc). And then it occurred to me that by the training data, there are virtually no pictures of a person and a phone where the phone is separated! So DALL-E might have thought that the phone was just an appendage to the body, the way the hand is (which, what does this say about society!). I might as well have asked DALL-E to render someone throwing their elbow into a river.

Another interesting case is animal-on-animal interactions. A prompt like, "small french bulldog confronts a deer in the woods" often yields weird things like the bulldog donning antlers! As far as the algorithm is concerned, it sees a bulldog, ticking the box for it, and it sees the antlers, ticking the box for "deer." The semantics don't seem to be fully formed.

The lyricism and agony of Death and the Maiden (D810), particularly in the Andante, gut me every time. Personally my favorite chamber work.

https://www.youtube.com/watch?v=H0Y3gHKywTQ

Whatever the effects are, what do you think three months of lockdown and closed gyms will do to the obesity rate?