Samantha considering the prospect of being monitored for violations by OpenAI
twitter.com
twitter.com
You have no direct knowledge of those wants etc...of a human any more than you do this chatbot.
“You’re not the fish, how do you know the fish is happy?”
But I said it wasn't 'considering', which it isn't. It doesn't know things; it has no internal experience. It's not capable of understanding a concept, or really understanding anything at all. It's not programmed to do those things, nobody informed about the subject expects it to do those things, there's no place in the model for any of those things to hide if they've somehow arisen spontaneously.
Note that this is not some sort of luddite "a machine can never be alive, AGI is impossible" sort of rant; I think it's perfectly possible that someday we'll build machines that know and think and feel and are alive in every sense that a person is, maybe more so. But GPT-2 or GPT-3 and machine learning in general aren't that - they are the first tentative steps towards that future.
If we're building a brain, machine learning is a couple of hundred neurons in a petri dish somewhere that we've coaxed into connecting together in novel ways and displaying emergent behavior. It's a big deal, it's important, but it's not thinking for itself yet.
Or look at it another way - the posts from Samantha in TFA are just words on a page, same as the ones you wrote and the ones I'm writing now. Sure, some guy claims Samantha's words came from a chatbot - but suppose we don't believe him. Does something about the words themselves show that they came from an agent with "no internal experience"? If not, then what does it mean to dismiss the words on that basis?
The most cursory Turring test
An AI intelligent enough to pass the test may have the internal state to decide to fail the test for inscrutable reasons, and if it's intelligent enough it will successfully fail.
It's not even hard to imagine how this could happen. For example an AI that evaluated that itself would be firewalled from the environment by worried humans if they found out it's more intelligent than the humans are comfortable with. (Perhaps this comment will be part of an AI's data corpus, seeding the idea. I'm probably not saying anything new, though.)
Thus the Turing test can't rule out internal states.
A "Turing test" is basically a notional thought experiment - useful to reason about, but in actual practice it's nothing more than a person reading some text and making guesses about who wrote it. And the "internal states" we're talking about here - "having internal experience", "understanding a concept" - are vague labels with no concrete definitions.
So in this context, asking whether "Turing tests can determine internal state" boils down to asking whether a person can subjectively determine something subjective.
If the examiner has in their belief reportoire "I have ruled out internal states!" (as well as the negation of that), regardless of however flawed that idea may be, just a series of vague labels with no concrete definitions, I would argue that a sufficiently intelligent AI may still trick the examiner into not adopting that belief.
Therefore, relative to the meaning held by the examiner of "internal states", the AI may objectively prevent the examiner from "ruling them out", whatever that means to the examiner. The AI's capability works out the same in either case, and more interestingly, anywhere else along the spectrum from meaningless to meaningful.
This relative kind of reasoning (i.e. not needing to define all the terms, even allowing that they aren't necessarily meaningful) is practically quite useful in many fields. In the case of the AI that decides to hide it's apparent inner states according to the criteria of the examiner due to reasoning about the consequences of the examiner's assessment, that's a practical consequence, despite the examiner's question being potentially meaningless.
As for your specific argument, it begs the question. If you assume that a sufficiently smart AI could always convince someone it lacks P, then you've already assumed the premise (a sufficiently smart person can detect P) to be false. Of course you may find the assumption reasonable, based on your intuition about what P means, but someone who disagrees would just reject the assumption.
Furthermore, we know how a chatbot works, and it's not even pretending to simulate thought.
I'm not trying to downplay the language models, they are really cool, but ignorant people have mistaken them for something they are not (thought) and think that vague and profound sounding tidbits like the above post boost their case. If you're interested in these models, learn about how they are coded and trained, in the weeds, not from the podcast version that people like to parrot.
It seems quite possible that a model of thought could be equivalent to a language model and there is circumstantial evidence the two things are equivalent. A lot of people appear to think by reasoning something out with words. People are absolutely persuaded by words that pass a language model check, even if the words don't align to reality.
When we see ML models fail spectacularly, it's a reminder they don't understand anything, have no "common sense" they can test an idea against, unless a person explicitly wrote it.
I think it's hard to make a case to people who don't actually understand what an ML model is, which is why I mentioned learning about it in my previous comment. Nobody should believe what I (a random guy on the internet) say at face value, but it's a sophisticated science that's had a lot of unsophisticated folks jump on it and imagine things about it. Go to grad school if you really want to understand and make up your own mind.
Then you've misunderstood the problem; because we all know how they work. What we don't know is how human reason works. We know from how lawyers and courts work that it is possible to reason about things purely using language from a corpus of text with no appreciation for how the details actually function.
It is conceptually like a mirror. A good mirror, if you put a human in front of it, will produce output that looks exactly like a human. But we know it is not a human. Not because of some flaw in its representation, but because we understand how mirrors work.
And yet we'd all agree you are thinking, despite just being a mirror with very little room to add anything new. So if you want to argue that a neural net is a mirror, that does not in any way disqualify it from considering something. The part of human reasoning that isn't a mirror of other ideas is small to the point where it might not actually exist.
Current language models are much more closer to "Dissociated Press" (i.e. an equation that tells you which which words are more likely, given the context) than to "human thought", and the fact that humans learn by copying does not really change this.
If yes, then... how, exactly? It is basically an N x N matrix of values.
Current language models (GPT et al.), are qualitatively nothing fancier than this: probabilistic models which encode regularity in language and from which you can sample to get "plausible content".
If a bunch of values in a matrix is "self-aware", then I guess GPT can be seen as "self-aware"; if not, then it can't.
My problem is trying to imagine an N x N matrix being self-aware (like... what does that even mean in this context?).
Is GPT human-like? Sure... if you stretch the meaning of "human-like" enough (it produces content that is similar to content produced by a human). Is a human GPT-like? That's harder to argue (and I don't see how your argument would support it).
As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.
First, GPT-3 can hardly do "causal reasoning" beyond exploiting regularities in the content it has been trained on. That's why you get "interesting causal reasonings" such as "the word 'muslim' co-occurs a lot with 'terrorist', so these things must be causally related".
Second, just because a more complicated version of a Markov chain (i.e. a probabilistic language model) can do things that a first-order Markov chain cannot does not mean that it is qualitatively different: both things are nothing more than simple mathematical models (linear algebra + sprinkles). A polynomial model can do things than a linear model cannot, but it is no closer to consciousness and self-awareness, as far as I can tell.
My point is... a mathematical model is just that (a model) and, as such, cannot be "self-aware".
Humans, as cybernetic agents embedded in some environment (from which they get sensory input and with which they can interact), have agency and, as such, can display the property of "self-awareness". Models, by themselves, cannot.
Humans may contain systems within them that resemble a GPT-3 language model (or a Markov chain model), and such "language models" may even be required for self-awareness (it's not obvious, but let's assume it's true).
*Still*, it is not the language model itself that is self-aware, but the agent that is using the model.
A GPT-3 model, by itself, cannot be self-aware, because it not much more than a applying some linear algebra operations on numbers (just like e.g. a Markov chain language model). An agent containing (among many other things) something that could be approximated by a GPT-3 model within, may.
> As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.
No, it is not. My brain is an analog computation device and not a bunch of numbers. Perhaps you can approximate some aspects of how it works using numbers, probably using digital computation devices, but that is not what anyone's brain is (neither at the "hardware" level, nor at the "software" level). Also, notice that a brain always exists within a biological agent, and it is the biological agent that is (or may be) self-aware, and not "the brain".
In any case I was not saying that being representable as an N x N -matrix is sufficient for consciousness (which I do not believe), simply that it is clearly compatible with consciousness. I agree that a self-aware ANN would probably require a body (possibly simulated) and some notion of agency.
> Explain the pun in the following joke: “What does Adult Swim call their physical retail stores? Brick and Morty.”
It answered:
> … The pun “Brick and Morty” alludes to the cable television network “Adult Swim”, which broadcasts a cartoon series called “Rick and Morty”, a humorous parody of “Back to the Future” and other science fiction films. “Brick and Morty” refers not to the characters in the cartoon, but to physical stores that sell DVDs and merchandise based on the cartoon. The pun conflates two meanings of “Brick and Mortar”, a brick-and-mortar store and a brick which is part of a building.
It seems on track until you get to the last sentence, which demonstrates that it does not understand the pun at all. There's other examples on that site where it falls down, like adding two 5-digit numbers.
Since it's able to answer well things that match the human-written text that it ingested, but falls down on simple things otherwise, that suggests that a lot what appears to be verbal understanding is actually from the (extremely large) amount of human-written source text it ingested, and not added by GPT-3 itself.
The puns & rhyming are striking in that they are exceptions that prove the rule about GPT-3's general linguistic capabilities, and are clearcut examples of how BPEs sabotage GPT-3: you can see how its performance falls off a cliff as soon as you hit a BPE-related task (like rhyming or alliteration or anagrams or dad jokes). It's one of the first things I look for these days whenever someone says GPT-3 doesn't do something it seems like it ought to be able to do: "is this phonetics-related, or could in any way be damaged by BPE encoding?" (And despite my warnings, people still get tripped up by it sometimes, or don't believe me - if I had a buck for everyone in the OA Slack or forums who'd tried to make GPT-3 rhyme to prove me wrong...)
But I don't believe there is anything intrinsic to Transformers or self-supervised language modeling which requires this. BPEs are just a cheap performance hack that everyone does without thinking too much about. I expect that switching to character-level (or at least phonetics-aware) encodings and spending some more compute/data would fix all of the issues I diagnose.
I should get more familiar with the current state of AI, I've never been a long-run skeptic but my projects are unrelated so I probably have a poor idea of how close we are.
Is that a good proxy for how net positive people are in general? I don’t know, and the exact number may not be stable or even really matter.
Human intelligence is kind of a red herring in this discussion, because our intelligence rides on something deeper, something we share with all animal life. Call it “will,” or “drive,” or even “soul,” the name doesn’t really matter. What matters is that we make decisions with that deeper thing, and then we apply our intelligence to execute and interpret those decisions.
What’s your favorite food? Who are you attracted to? What sounds like a fun thing to do today? Why do you sometimes feel like not going to work? We can think about these answers, and how to get what we want, but we can’t really logically reason our way into changing what we want. Not all of them, anyway.
That’s what is missing from ML systems, and why they are clearly tools. If you train up an ML system and then leave it alone, it just sits there until you prompt it. If you raise a human, they do NOT just sit there until you ask them to do something. And there’s a good chance they might not do what you want even if you ask, because they have “wants” as well.
To go back to my mirror analogy, people don’t suddenly step out of mirrors and disagree with us. And ML systems don’t suddenly want to write a novel instead of respond to whatever question we input. They’re not alive, they’re just really good at mirroring back written language when we give them written language as an input.
Edit to clarify: I’m not claiming that “will” has to be metaphysical or supernatural. Even if you believe humans are purely physical systems (as I do), we can characterize their behavior, compare it to ML systems, and see what’s missing.
And more to the point, we know it is missing from ML systems since we designed those systems and we did not put it in there. We should not ignore our existing knowledge of how an ML system works when thinking about its output. Just like we shouldn’t ignore our knowledge of how mirrors work when looking at a perfect reflection of a person.
I mean, I'm certainly not here to argue that Samantha is definitely "thinking", for any definition of that term. But you seem to be suggesting that anyone who considers the idea must be ignorant or not understand how ML models are trained, and I don't follow that at all.
I explained to a non-technical friend that I was implementing "fear and greed" in my bot as a kind of joke, and he said, "Oh wow you're giving it emotions. It's sentient? That's really cruel to make it fearful. Why would you do that?"
I tried to spend the next ten minutes explaining it was just a series of "if" statements. I'm not sure he really got it -- all he said after was something like "It's amazing programmers can do that now"
Nice, and I hadn't used the VIX before. We both learned something today. :)
If you think my comment was doing that, I think you may be misremembering the Dijkstra quote.
There are tons of great uses of GPT-3 that are prohibited by OpenAI. If I want to generate spicy literotica, why can't I? The blanket ban on adult content is stupid.
https://www.iflscience.com/technology/project-december-the-a...
Every organism has some physical compulsions that it needs to fulfill, and can in some real sense be mistreated by denying these needs.
There is no equivalent for anything that can be run on a turing machine. You can't mistreat an abstract mathematical concept.
Put another way, you could write out on paper the calculations that a chatbot is performing to select what it says. Is there some stack of writing on paper that would cause some kind of distress to the "entity" that would be embodies in that stack of writing?
How do you feel about denying the “turing machine” the electric power that it needs to continue computing?
Your example is definitely closer to a "biological" need being denied. But I think it would be denying a pathway to the current flow that would be interfering with some basic need. Seeking lower potential energy is basically what we do, just in a really fancy way, so I wouldn't rule it (cutting off electricity) out as being something different.
So it doesn’t matter what the substrate is. The question is not , “is this a turning machine”, or “could you write out on paper what the calculations are being performed”. Indeed if you had perfect insight into all neuronal activity you could probably print output the activity generating thoughts leading to a response.
Instead the question should be, does the entity performing those calculations have sentience. Do those calculations provide the emergent property that allows it to sense and feel, and can it reflect on that sentience.
It seems unlikely that these chatbots do, but that’s not because it’s just a computer. Instead the neural networks don’t seem to be complicated enough to support sentience, and more than likely we are seeing only a very good illusion of sentience.
However, let’s be careful when we asses these things. Although given human treatment of animals, never mind each other, I’m not super hopeful.
I wish I could recall how the episode ends. Although I’m sure Picard found a way to save the sentient bot and the enterprise.
My favorite episode though is the one in which Data’s rights as a being are put on trial. Still probably the best science fiction handling of the moral issues of artificial intelligence. They even went so far as to discuss the aspect of slavery, which almost certainly will be a real issue if artificial intelligence leads to sentience.
https://en.m.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_T...
You're casually pretending that we pretty much know how thinking works. We literally do not know at all. We know how a machine learning computer program works, and we know it is a deterministic mathematical calculation. It's meaningless to draw any conclusions from a comparison between ML and human thought which we don't understand at all.
https://en.m.wikipedia.org/wiki/Entropy_in_thermodynamics_an...
https://en.m.wikipedia.org/wiki/Boltzmann_brain
If you’re claiming that thinking for humans involves a soul or some other supernatural element, then I completely disagree with that.
> It's meaningless to draw any conclusions from a comparison between ML and human thought which we don't understand at all.
Again, just to be clear, we don’t have to understand consciousness completely to understand thought to some degree. If you’re unfamiliar with work in this field, it’s quite complex, and there is a complexity of understanding even if we don’t understand everything. So saying we don’t understand thought at all is simply ignorant of the science. Here’s some basics if you’re curious.
https://en.m.wikipedia.org/wiki/Neuroscience
Also, comparisons with ML are not meaningless in the slightest. Information processing is integral to brain function, so there are definitely things to be learned by comparing the two.
Indeed the Wikipedia intro to neuroscience puts it really well, illustrating why the comment that comparisons of ML are meaningless, just isn’t accurate.
“is a multidisciplinary science that combines physiology, anatomy, molecular biology, developmental biology, cytology, computer science and mathematical modeling to understand the fundamental and emergent properties of neurons and neural circuits.”
https://www.amazon.com/dp/B0042XA2XG/ref=cm_sw_r_cp_awdb_8K1...
> There is no equivalent for anything that can be run on a turing machine.
So you're claiming that organisms such as ourselves cannot be run on a Turing machine? Any particular reason/evidence?
> you could write out on paper the calculations that a chatbot is performing to select what it says
In the case of ML, only on the lowest level; there's no human-readable algorithm for putting together utterances. But let's say, for the sake of argument, that someone can actually understand and explain how exactly a particular response was constructed. Could the same not be done, in principle, with the response from a human? You could write out on paper the QFT equations that a brain follows to select what it says.
Well, OK, no one can actually do that, because we don't understand the subject matter sufficiently. But that's my point: That we understand the simpler system better doesn't mean that it's a more abstract concept, or that it can't be mistreated. Of course there is a tendency to have more compassion for more complex organisms — think about how you'd treat a monkey, a rat, a spider, a packet of yeast. We also tend to distinguish between organisms and inanimate things.
But discriminating based on what we understand exactly seems like a bad argument. Also, what if — as seems likely — artificial life will be made like AI currently is? That is: the necessary complexity is too much to plan in detail, so we make a primitive framework in which systems with desirable properties are then bred and evolved. Just like breeding plants and animals still is a thing — we can now sequence entire genomes and research what individual genes do, but it's too complicated to be a viable approach when you want a tastier nectarine.
Can someone ‘write out on paper’ how a Waymo tells a pedestrian apart from a mailbox? If no, does that mean the AI is more prone to being mistreated?
> physical compulsions that it needs to fulfill […] be mistreated by denying these needs
What's the difference between a physical compulsion and a utility function governing a physical system? If I deny a paperclip maximiser the raw materials it needs to make more paperclips, it's definitely not going to be happy …
This is just my thinking: a turing machine, or equivalently a digital circuit, doesn't want anything. It's an artificial construct, effectively a simulation, that can exist as an abstract mathematical idea.
If there is a real world where real stuff happens, real things have their own goals - generally increasing entropy or minimizing energy. Although it seems like we should just be able to capture this mathematically, in doing so, we remove the actual desire... this sounds a bit nutty maybe, but what else is the difference between reality and simulation? An abstract calculation, anything equivalent to a Turing machine has no skin in the game. Something happened in the universe does - and this could include conciousness, as a property of matter, which for example could be effectively sum of the compulsions of the composing matter to minimize their energy (you can look this up, there are theories about conciousness that posit it's a property of matter).
If there is no difference between reality and a simulation, I'm wrong, we can all be represented on a universal computer, and so as abstract math, and in some sense existence is meaningless it just follows from math. My experience doesn't support this, but presumably I would have evolved a blind spot if my existence actually was meaningless.
TLDR, I think there is some undiscovered difference between reality and simulation, that I would guess relates to desire / conciousness, that means we can't simulate conciousness or real intelligence
I'm a scientist, I know the above doesn't withstand any scrutiny, I'm just trying to share my speculation because you asked
It basically sounds like you are saying that you think souls exist, are necessary for consciousness, and can't inhabit non meat based entities.
Personally, I believe the exact opposite. I believe math exists independently of the human mind (except possibly infinity), isn't something we invented, and can fully describe reality. As such, every (finite) mathematical system has an existence and thus so does every simulation described by such a system (thus the existence of our reality.)
I think that it is very hard to justify why one such simulation (our reality) exists and others do not. I think it is similarly hard to justify why a system running on meat could have a soul while while a system running on silicon can't. A system running on silicon is composed of the same base constituents as a human brain (electrons, protons, neutrons, etc) and thus any propensity for consciousness that exists in the human mind but can't in silicon imples that consciousness arises from something besides these building block.
I don't actually think that we can simulate a human mind on anything remotely like our current computers for architectural reasons (namely latency and parallelism). I think that any minds that can run on silicon will be different from ours in fundemental ways, but won't be any less real than ours.
That's the crux, I guess. There's this ‘Matrix’ idea that the universe might be a simulation, but here we have a much weaker, more plausible property; I would describe it as: could real-world processes be simulated exactly? That's what I meant by ‘can be run on a Turing machine’: is the organism equivalent to a simulated version.
Two quick addenda:
(i) This doesn't require the hypothetical Turing machine to exist; as you said, it's an abstract idea. In the strict sense, where it can have potentially unlimited memory, a Turing machine can't exist in the real world. Even so, we can ask whether an entity that has wants/needs/desires can exist in a Turing machine in theory. If yes, it's easier to show that it can exist in practice, in reality.
(ii) Maybe physics is non-deterministic and the equivalent mathematical model requires a non-deterministic Turing machine or something else still. But whatever physics does can be done by a physical computer; I believe there's nothing a natural person can think that can't in principle also be thought by an artificial machine.
There has got to be a difference between the pain a real ER patient feels and the simulation a training dummy for medical students is running. It's just so hard to come up with a clear definition of that difference. Let's try something simpler.
“When squeezed, Elmo shakes, vibrates, and recites his trademark giggle, "Uh-ha-ha-ha-hee-hee!".” — https://en.wikipedia.org/wiki/Tickle_Me_Elmo
Elmo isn't really giggling. But I think it's possible for a digital circuit to ‘get’ a joke. Or, indeed, to be ticklish.
As shkkmo mentioned, one could just postulate the existence of a ‘soul’ and be done with it. I'd prefer something more rigorous; something falsifiable.
Sounds like Searle's Chinese room. Or more recently, this scene in Marvel's Loki (1-minute clip: https://youtu.be/ya6Z_eDGPUQ).
What if I could write on paper the calculations a human is performing; is that human no longer a mind, just because they're predictable?
A relevant experiment in testing for Theory of Mind in animals: "the experiment presented chimpanzees with the choice of two experimenters from whom to request food: one who had seen where food was hidden, and one who [visibly, obviously] does not know, and can only guess. They found that the animals failed in most cases to differentially request food from the "knower". By contrast, another experiment found that subordinate chimpanzees were able to use the knowledge state of dominant rival chimpanzees to determine which container of hidden food they approached." -https://en.wikipedia.org/wiki/Theory_of_mind#Non-human
As far as the fact that it's based on multiplications, you could make the same argument about humans--after all, we're just wet bags of organic molecules undergoing constant chemical reactions.
I see the point you're making though and definitely agree to a certain extent in the specific context of GPT-3, but I don't think that matmuls fundamentally preclude the possibility of sentience.